Add callback for oauth #960

stevenylai · 2020-09-25T03:04:55Z

We have requirements to make use of the OAuth server in our company for Kafka. Our Kafka server-side implementation is based on: https://github.com/kafka-security/oauth

This change expose the necessary callback for Python client

This closes #839

ghost · 2020-09-25T03:04:57Z

It looks like @stevenylai hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

stevenylai · 2020-09-25T03:09:28Z

[clabot:check]

ghost · 2020-09-25T03:09:29Z

It looks like @stevenylai hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

stevenylai · 2020-09-29T08:12:34Z

It looks like @stevenylai hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

[clabot:check]

ghost · 2020-09-29T08:12:35Z

It looks like @stevenylai hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

stevenylai · 2020-09-29T08:16:29Z

[clabot:check]

ghost · 2020-09-29T08:16:30Z

@confluentinc It looks like @stevenylai just signed our Contributor License Agreement. 👍

Always at your service,

clabot

robooo · 2021-01-27T18:48:29Z

any updates @stevenylai ? in which release this could be merged?

stevenylai · 2021-01-31T03:46:01Z

any updates @stevenylai ? in which release this could be merged?

I have updated my branch. Let's see if the developers from confluentinc has time to review my change.

edenhill

Great stuff!

edenhill · 2021-03-09T07:31:27Z

src/confluent_kafka/src/confluent_kafka.c

+	        Py_DECREF(result);
+	        PyErr_Format(PyExc_TypeError,
+	                     "expect returned value from oauth "
+	                     "to be (token_str, expiry_int_ms) tuple");


Durations (in various forms) are usually expressed as floating point seconds in Python, we might want to do the same here for expiry_int_ms

edenhill · 2021-03-09T07:31:54Z

src/confluent_kafka/src/confluent_kafka.c

+
+	if (!result) {
+	    goto err;
+	} else {


This else block is not needed since the if above has a goto. Reduce indent.

edenhill · 2021-03-09T07:33:00Z

src/confluent_kafka/src/confluent_kafka.c

+	PyObject *eo = NULL, *result = NULL;
+	CallState *cs = NULL;
+	const char *token = NULL;
+	long long expiry = 0;


use int64_t

edenhill · 2021-03-09T07:33:27Z

src/confluent_kafka/src/confluent_kafka.c

@@ -1521,6 +1521,48 @@ static void log_cb (const rd_kafka_t *rk, int level,
        CallState_resume(cs);
 }

+static void oauth_cb (rd_kafka_t *rk, const char *oauthbearer_config, void *opaque) {
+	Handle *h = opaque;
+	PyObject *eo = NULL, *result = NULL;


No need for initialization of eo, result, cs, token and expirty, err_code, they're always set below before being used.

edenhill · 2021-03-09T07:36:16Z

src/confluent_kafka/src/confluent_kafka.c

+	Py_DECREF(result);
+	if (err_code) {
+	    PyErr_Format(PyExc_ValueError,
+	                 "invalid token: %s", err_msg);


I think you should remove "invalid token: " and just keep the "%s" as the error message from librdkafka should be complete as-is.

edenhill · 2021-03-09T07:37:35Z

src/confluent_kafka/src/confluent_kafka.c

+	Py_DECREF(eo);
+
+	if (!result) {
+	    goto err;


use 8 whitespace indent in the C code (throughout)

edenhill · 2021-03-09T07:39:39Z

src/confluent_kafka/src/confluent_kafka.c

@@ -1949,6 +1991,19 @@ rd_kafka_conf_t *common_conf_setup (rd_kafka_type_t ktype,
                        Py_XDECREF(ks8);
                        Py_DECREF(ks);
                        continue;
+                } else if (!strcmp(k, "oauth")) {


Think this should be called oauth_cb to be consistent with other callback names

edenhill · 2021-03-09T07:39:54Z

src/confluent_kafka/src/confluent_kafka.h

@@ -236,6 +236,7 @@ typedef struct {
        rd_kafka_type_t type; /* Producer or consumer */

        PyObject *logger;
+        PyObject *oauth;


edenhill · 2021-03-09T07:40:02Z

src/confluent_kafka/src/confluent_kafka.c

+	    if (!PyArg_ParseTuple(result, "sl", &token, &expiry)) {
+	        Py_DECREF(result);
+	        PyErr_Format(PyExc_TypeError,
+	                     "expect returned value from oauth "


edenhill · 2021-03-09T13:02:58Z

src/confluent_kafka/src/confluent_kafka.c

@@ -1949,6 +1990,19 @@ rd_kafka_conf_t *common_conf_setup (rd_kafka_type_t ktype,
                        Py_XDECREF(ks8);
                        Py_DECREF(ks);
                        continue;
+                } else if (!strcmp(k, "oauth_cb")) {
+                        if (h->oauth_cb) {


add a PyCallable_Check here, see stats_cb above.

Done. I have also updated my branch to the current master

edenhill

Looking good!

Please add a dry-run test-case to test_misc.py that simply instantiates a client (with no bootstraps) with a proper callable oauth_cb, just so we have verification that part works.

edenhill · 2021-03-11T08:13:59Z

docs/index.rst

@@ -459,6 +459,9 @@ The Python bindings also provide some additional configuration properties:
  This callback is served upon calling ``client.poll()`` or ``producer.flush()``. See
  https://github.com/edenhill/librdkafka/wiki/Statistics" for more information.

+* ``oauth_cb(oauthbearer_config_str)``: Callback for retrieving Oauth token.


Add a line saying that oauthbearer_config_str is the value of the sasl.oauthbearer.config configuration property.

edenhill · 2021-03-11T08:14:35Z

docs/index.rst

@@ -459,6 +459,9 @@ The Python bindings also provide some additional configuration properties:
  This callback is served upon calling ``client.poll()`` or ``producer.flush()``. See
  https://github.com/edenhill/librdkafka/wiki/Statistics" for more information.

+* ``oauth_cb(oauthbearer_config_str)``: Callback for retrieving Oauth token.
+  Return value of this callback is expected to be ``(token_str, expiry_time)`` tuple.


Need to indicate that expiry_time is float seconds absolute (or is it relative?) time.

edenhill · 2021-03-11T08:15:23Z

examples/oauth_producer.py

+    resp = requests.post(args.token_url,
+                         auth=(args.client_id, args.client_secret), data=payload)
+    token = resp.json()
+    return token['access_token'], time.time() + token['expires_in']


Perhaps cast token[exp..] to float?

Personally, I think time.time() + float(token['expires_in']) is a bit redundant but I see there are already codes which does float() conversion so I adapted mine anyways to be consistent.

edenhill · 2021-03-11T08:15:53Z

examples/oauth_producer.py

+        'value.serializer': StringSerializer('utf_8'),
+        'security.protocol': 'sasl_plaintext',
+        'sasl.mechanisms': 'OAUTHBEARER',
+        'sasl.oauthbearer.config': "not-important",


micro-nit: mix of ' and "

edenhill · 2021-03-11T08:17:16Z

src/confluent_kafka/src/confluent_kafka.c

+        const char *token;
+        double expiry;
+        char err_msg[2048];
+        rd_kafka_resp_err_t err_code = RD_KAFKA_RESP_ERR_NO_ERROR;


nit: the initialization is not neededd, it is never used before assignment.

My bad, I missed this one from my previous change.

edenhill · 2021-03-11T08:17:30Z

src/confluent_kafka/src/confluent_kafka.c

+                             "to be (token_str, expiry_time) tuple");
+                goto err;
+        }
+        err_code = rd_kafka_oauthbearer_set_token(h->rk, token, (int64_t)(expiry * 1000), "",


Try to keep below 80 columns width.

edenhill · 2021-03-11T08:21:54Z

I think this part of from the rdkafka.h documentation needs to go into the python docs as well:

 * Note that before any SASL/OAUTHBEARER broker connection can succeed the
 * application must call rd_kafka_oauthbearer_set_token() once -- either
 * directly or, more typically, by invoking either rd_kafka_poll() or
 * rd_kafka_queue_poll() -- in order to cause retrieval of an initial token to
 * occur.

(rewritten to make sense for python)

update docstr Addressing review comments No need for 1k multiplication now that we are in secs Add doc for oauth_cb Check callable

edenhill · 2021-03-15T12:54:51Z

examples/oauth_producer.py

+        'value.serializer': StringSerializer('utf_8'),
+        'security.protocol': 'sasl_plaintext',
+        'sasl.mechanisms': 'OAUTHBEARER',
+        'sasl.oauthbearer.config': 'not-important',


nit: this make it seem like sasl.oauthbearer.config is not generally important, but required.
Perhaps better to make it more explicit by commenting out the line with a comment saying that no extra configuration is required for this example, but that the property is used to pass configuration to the oauth_cb.

edenhill · 2021-03-15T13:10:42Z

examples/oauth_producer.py

@@ -55,7 +55,10 @@ def producer_config(args):
        'value.serializer': StringSerializer('utf_8'),
        'security.protocol': 'sasl_plaintext',
        'sasl.mechanisms': 'OAUTHBEARER',
-        'sasl.oauthbearer.config': 'not-important',
+        # sasl.oauthbearer.config can be used to pass argument to your oauth_cb


edenhill · 2021-03-15T13:37:38Z

Thanks for your contribution!

mataralhawiti mentioned this pull request Nov 20, 2020

How to use jwt callback for consumer - works in python-kafka but zero examples on this lib #993

Closed

edenhill suggested changes Mar 9, 2021

View reviewed changes

edenhill suggested changes Mar 11, 2021

View reviewed changes

stevenylai added 2 commits March 13, 2021 18:56

Add OAuth callback

1c90092

update docstr Addressing review comments No need for 1k multiplication now that we are in secs Add doc for oauth_cb Check callable

Update according to comments

0bc6d48

edenhill suggested changes Mar 15, 2021

View reviewed changes

comment

99463c1

edenhill reviewed Mar 15, 2021

View reviewed changes

edenhill merged commit b7f8dce into confluentinc:master Mar 15, 2021

stevenylai deleted the oauth branch March 16, 2021 00:41

Add callback for oauth #960

Add callback for oauth #960

Uh oh!

Conversation

stevenylai commented Sep 25, 2020

Uh oh!

ghost commented Sep 25, 2020

Uh oh!

stevenylai commented Sep 25, 2020

Uh oh!

ghost commented Sep 25, 2020

Uh oh!

stevenylai commented Sep 29, 2020

Uh oh!

ghost commented Sep 29, 2020

Uh oh!

stevenylai commented Sep 29, 2020

Uh oh!

ghost commented Sep 29, 2020

Uh oh!

robooo commented Jan 27, 2021

Uh oh!

stevenylai commented Jan 31, 2021

Uh oh!

edenhill left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edenhill left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edenhill commented Mar 11, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edenhill commented Mar 15, 2021

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.