KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #4941

PratRanj07 · 2024-12-27T11:22:46Z

This PR intends to add support for calling DescribeConsumerGroups for groups created with the new consumer protocol and add backward compatibility to describe group created with the old Classic Protocol.
The working of the API remains similar. Users will continue to call the existing DescribeConsumerGroups api but internally we will be calling the new ConsumerGroupDescribe api to fetch the results. If we get GRP_NOT_FOUND or UNSUPPORTED_VERSION error, then in that case we will call the old DescribeConsumerGroup api.

confluent-cla-assistant · 2024-12-27T11:22:56Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

pranavrth

First pass comments.

pranavrth · 2025-02-04T08:28:42Z

src/rdkafka.h

+/**
+ * @brief Gets target assignment of \p member.
+ *
+ * @param member The group member.
+ *
+ * @return The target assignment.
+ *
+ * @remark The lifetime of the returned memory is the same
+ *         as the lifetime of the \p member object.
+ */
+RD_EXPORT
+const rd_kafka_MemberAssignment_t *rd_kafka_MemberDescription_target_assignment(
+    const rd_kafka_MemberDescription_t *member);
+
+/**
+ * @brief Gets target assigned partitions of a member \p assignment.
+ *
+ * @param assignment The group member assignment.
+ *
+ * @return The target assigned partitions.
+ *
+ * @remark The lifetime of the returned memory is the same
+ *         as the lifetime of the \p assignment object.
+ */
+RD_EXPORT
+const rd_kafka_topic_partition_list_t *
+rd_kafka_MemberAssignment_target_partitions(
+    const rd_kafka_MemberAssignment_t *assignment);
+


What are we doing for the old protocol on these new functions?

It is giving NULL as default value in these cases. I have updated it in the comments

pranavrth · 2025-02-06T07:25:51Z

CHANGELOG.md

@@ -2,6 +2,7 @@

 librdkafka v2.8.0 is a maintenance release:

+* DescribeConsumerGroup now supports new consumer protocol groups (#4922).


Add that new fields type and target assignment is added. Target assignment is only applicable for Consumer protocol. Explain about the default values as well.

pranavrth · 2025-02-06T07:27:54Z

examples/describe_consumer_groups.c

+        const rd_kafka_topic_partition_list_t *target_topic_partitions =
+            rd_kafka_MemberAssignment_partitions(target_assignment);


This will give NULL pointer exception as target_assignment is only applicable for consumer protocol. It should be NULL for classic protocol and default value should be NULL.

pranavrth · 2025-02-06T07:28:56Z

examples/describe_consumer_groups.c

+        if (!target_topic_partitions) {
+                printf("    No target assignment\n");
+        } else if (target_topic_partitions->cnt == 0) {
+                printf("    Empty target assignment\n");
+        } else {
+                printf("    Target assignment:\n");
+                print_partition_list(stdout, target_topic_partitions, 0,
+                                     "      ");
+        }


This is different for classic protocol.

pranavrth · 2025-02-06T07:39:52Z

src/rdkafka_proto.h

-            [RD_KAFKAP_ListTransactions]          = "ListTransactions",
-            [RD_KAFKAP_AllocateProducerIds]       = "AllocateProducerIds",
-            [RD_KAFKAP_ConsumerGroupHeartbeat]    = "ConsumerGroupHeartbeat",
+            [RD_KAFKAP_ConsumerGroupDescribe]  = "ConsumerGroupDescribeRequest",


Better add this at the end.

pranavrth · 2025-02-06T08:07:21Z

src/rdkafka_request.c

+ */
+rd_kafka_resp_err_t
+rd_kafka_ConsumerGroupDescribeRequest(rd_kafka_broker_t *rkb,
+                                      const rd_list_t *groups /*(char*)*/,


This should be char ** and accept count as well. Check older API.

pranavrth · 2025-02-06T08:28:59Z

src/rdkafka_request.c

+                                      rd_kafka_resp_cb_t *resp_cb,
+                                      void *opaque) {
+        rd_kafka_buf_t *rkbuf;
+        int16_t maxApiVersion = 0;


No need for this variable. Used only once.

pranavrth · 2025-02-06T08:29:42Z

src/rdkafka_request.c

+        int16_t ApiVersion    = rd_kafka_broker_ApiVersion_supported(
+            rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, maxApiVersion, NULL);
+        size_t ofGroupsArrayCnt;
+        int grp_ids_cnt = rd_list_cnt(groups);
+        int i, include_authorized_operations;
+        char *group;


Suggested change

int16_t ApiVersion = rd_kafka_broker_ApiVersion_supported(

rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, maxApiVersion, NULL);

size_t ofGroupsArrayCnt;

int grp_ids_cnt = rd_list_cnt(groups);

int i, include_authorized_operations;

char *group;

size_t ofGroupsArrayCnt;

int grp_ids_cnt = rd_list_cnt(groups);

int i, include_authorized_operations;

char *group;

int16_t ApiVersion = rd_kafka_broker_ApiVersion_supported(

rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, maxApiVersion, NULL);

pranavrth · 2025-02-06T17:20:44Z

src/rdkafka_request.c

+        rkbuf = rd_kafka_buf_new_flexver_request(
+            rkb, RD_KAFKAP_ConsumerGroupDescribe, 1,
+            4 /* rd_kafka_buf_write_arraycnt_pos */ +
+                1 /* IncludeAuthorizedOperations */ + 1 /* tags */ +
+                32 * grp_ids_cnt /* Groups */,
+            rd_true /* flexver */);


pranavrth · 2025-02-07T07:29:31Z

src/rdkafka_admin.c

+        if (newgroupres->type == RD_KAFKA_CONSUMER_GROUP_TYPE_CONSUMER &&
+            newgroupres->error &&
+            (newgroupres->error->code == RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND ||
+             newgroupres->error->code ==
+                 RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION)) {


Check for memory leak in this condition as well.

pranavrth

Still reviewing. Few comments. Check for possible SegFaults.

pranavrth · 2025-02-14T07:06:06Z

src/rdkafka_admin.c

-                                              src->assignment.partitions);
+        return rd_kafka_MemberDescription_new(
+            src->client_id, src->consumer_id, src->group_instance_id, src->host,
+            src->assignment.partitions, src->target_assignment.partitions);


This will give a SegFault if target assignment is NULL

pranavrth · 2025-02-14T07:06:30Z

src/rdkafka_admin.c

+        if (member->target_assignment.partitions)
+                rd_kafka_topic_partition_list_destroy(
+                    member->target_assignment.partitions);
        rd_free(member);


Same changes of SegFault.

pranavrth · 2025-02-14T07:18:34Z

src/rdkafka_admin.c

+        RD_LIST_FOREACH(group, groups, i) {
+                groups_arr[i] = rd_list_elem(groups, i);
+        }
+


No need for this. You can directly use groups->rl_elems

pranavrth

Comments on Implementation part.

pranavrth · 2025-04-10T12:32:07Z

CHANGELOG.md

@@ -160,6 +160,7 @@ librdkafka v2.10.0 is a feature release:

 librdkafka v2.8.0 is a maintenance release:

+* DescribeConsumerGroup now supports new consumer protocol groups. Two field type and target assignment has also been added. Target Assignment is only valid for the new Consumer protocol and its default value is NULL. (#4922).


Move changelog to 2.10.0. Add [KIP-848] tag.

pranavrth · 2025-04-10T12:58:34Z

CHANGELOG.md

@@ -160,6 +160,7 @@ librdkafka v2.10.0 is a feature release:

 librdkafka v2.8.0 is a maintenance release:

+* DescribeConsumerGroup now supports new consumer protocol groups. Two field type and target assignment has also been added. Target Assignment is only valid for the new Consumer protocol and its default value is NULL. (#4922).


Suggested change

* DescribeConsumerGroup now supports new consumer protocol groups. Two field type and target assignment has also been added. Target Assignment is only valid for the new Consumer protocol and its default value is NULL. (#4922).

* `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment has also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).

pranavrth · 2025-04-10T13:27:43Z

src/rdkafka_admin.c

+const rd_kafka_topic_partition_list_t *
+rd_kafka_MemberAssignment_target_partitions(
+    const rd_kafka_MemberAssignment_t *assignment) {
+        if (assignment == NULL)


Suggested change

if (assignment == NULL)

if (!assignment)

pranavrth · 2025-04-10T13:29:13Z

src/rdkafka.h

+ *
+ * @param member The group member.
+ *
+ * @return The target assignment. It returns NULL for CLASSIC GROUPS.


Suggested change

* @return The target assignment. It returns NULL for CLASSIC GROUPS.

* @return The target assignment for `consumer` group types. It returns NULL for the `classic` group types.

pranavrth · 2025-04-10T13:31:20Z

src/rdkafka.h

+RD_EXPORT
+const rd_kafka_topic_partition_list_t *
+rd_kafka_MemberAssignment_target_partitions(
+    const rd_kafka_MemberAssignment_t *assignment);
+


We can update rd_kafka_MemberDescription_assignment itself. If assignment is NULL there then it will raise a SegFault there. We can update that function instead of creating a new one.

Let it be as it is as this is getter function.

We should remove the new function. there is no field like target_partitions in MemberAssignment. We should modify rd_kafka_MemberAssignment_partitions like earlier.

pranavrth · 2025-04-10T22:34:21Z

src/rdkafka_admin.c

+            (newgroupres->error->code == RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND ||
+             newgroupres->error->code ==
+                 RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION)) {
+                rko_fanout->rko_u.admin_request.fanout.outstanding++;


Explain this in comments.

pranavrth · 2025-04-10T22:44:06Z

src/rdkafka_admin.c

+        if (newgroupres->type == RD_KAFKA_CONSUMER_GROUP_TYPE_CONSUMER &&
+            newgroupres->error &&
+            (newgroupres->error->code == RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND ||
+             newgroupres->error->code ==
+                 RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION)) {


Explain what we are doing here in comments.

pranavrth · 2025-04-10T22:49:10Z

src/rdkafka_op.h

+        RD_KAFKA_OP_ELECTLEADERS,          /**< Admin:
+                                            *   ElectLeaders
+                                            *   u.admin_request */
+        RD_KAFKA_OP_CONSUMERGROUPDESCRIBE, /**< Admin:


I think this is not needed altogether. Try using the old one only.

pranavrth · 2025-04-10T22:50:54Z

src/rdkafka_request.c

+        ofGroupsArrayCnt = rd_kafka_buf_write_arraycnt_pos(rkbuf);
+        rd_kafka_buf_finalize_arraycnt(rkbuf, ofGroupsArrayCnt, group_cnt);


Suggested change

ofGroupsArrayCnt = rd_kafka_buf_write_arraycnt_pos(rkbuf);

rd_kafka_buf_finalize_arraycnt(rkbuf, ofGroupsArrayCnt, group_cnt);

rd_kafka_buf_write_arraycnt(rkbuf, group_cnt);

pranavrth · 2025-04-10T22:51:35Z

src/rdkafka_request.c

+        while (group_cnt-- > 0) {
+                rd_kafka_buf_write_str(rkbuf, groups[group_cnt], -1);
+        }


Write in same order as request.

src/rdkafka_proto.h

pranavrth · 2025-04-11T20:47:23Z

src/rdkafka_admin.c

+        char *group_id = NULL, *group_state = NULL, *assignor_name = NULL,
+             *error_str = NULL, *host = NULL;
+        char *member_id = NULL, *instance_id = NULL, *client_id = NULL,
+             *client_host                   = NULL;


Merge these two

pranavrth · 2025-04-11T20:49:34Z

src/rdkafka_admin.c

+                        member_id   = RD_KAFKAP_STR_DUP(&MemberId);
+                        instance_id = RD_KAFKAP_STR_DUP(&InstanceId);
+                        client_id   = RD_KAFKAP_STR_DUP(&ClientId);
+                        client_host = RD_KAFKAP_STR_DUP(&ClientHost);


This is leaking. Free them after using and assign them as NULL.

pranavrth · 2025-04-11T20:51:55Z

src/rdkafka_admin.c

+        rd_assert(rko_partial->rko_evtype ==
+                      RD_KAFKA_EVENT_DESCRIBECONSUMERGROUPS_RESULT ||
+                  rko_partial->rko_evtype ==
+                      RD_KAFKA_EVENT_CONSUMERGROUPDESCRIBE_RESULT);


We can have only 1 of these. Keep the older one.

pranavrth · 2025-04-11T20:56:47Z

src/rdkafka.h

+RD_EXPORT
+const rd_kafka_topic_partition_list_t *
+rd_kafka_MemberAssignment_target_partitions(
+    const rd_kafka_MemberAssignment_t *assignment);
+


We should remove the new function. there is no field like target_partitions in MemberAssignment. We should modify rd_kafka_MemberAssignment_partitions like earlier.

src/rdkafka.h

pranavrth · 2025-04-14T06:54:38Z

src/rdkafka.h

+ * @param member The group member.
+ *
+ * @return The target assignment for `consumer` group types.
+ *         It returns NULL for the `classic` group types.


Suggested change

* It returns NULL for the `classic` group types.

* Returns NULL for the `classic` group types.

pranavrth · 2025-04-14T06:58:17Z

src/rdkafka_admin.c

+                if (member->target_assignment->partitions)
+                        rd_kafka_topic_partition_list_destroy(
+                            member->target_assignment->partitions);


Use RD_IF_FREE for this and other if conditions in the function.

pranavrth · 2025-04-14T07:03:09Z

src/rdkafka_admin.c

+        if (assignment)
+                return assignment->partitions;
+        else
+                return NULL;


Suggested change

if (assignment)

return assignment->partitions;

else

return NULL;

if (!assignment)

return NULL;

return assignment->partitions;

src/rdkafka.h

emasab · 2025-04-14T08:12:00Z

src/rdkafka.h

+ * @remark The lifetime of the returned memory is the same
+ *         as the lifetime of the \p grpdesc object.


Given it's not returning a pointer this isn't necessary

Suggested change

* @remark The lifetime of the returned memory is the same

* as the lifetime of the \p grpdesc object.

pranavrth · 2025-04-14T08:29:47Z

src/rdkafka_admin.c

+
+        nodeid = rkb->rkb_nodeid;
+        rd_kafka_broker_lock(rkb);
+        host = rd_strdup(rkb->rkb_origname);


Free host. Memory leak there.

Free node as well.

pranavrth · 2025-04-14T08:35:19Z

src/rdkafka_admin.c

+                        if (assignment)
+                                rd_kafka_topic_partition_list_destroy(
+                                    assignment);
+                        if (target_assignment)
+                                rd_kafka_topic_partition_list_destroy(
+                                    target_assignment);
+


pranavrth · 2025-04-14T08:39:24Z

src/rdkafka_admin.c

+                        rd_kafka_MemberDescription_t *member        = NULL;
+                        rd_kafka_topic_partition_list_t *assignment = NULL,
+                                                        *target_assignment =
+                                                            NULL;


These will leak in err_parse case. Free them properly.

pranavrth · 2025-04-14T08:40:27Z

src/rdkafka_admin.c

+                   identify it for further processing with the old protocol and
+                   eventually in rd_kafka_DescribeConsumerGroupsResponse_parse
+                   we will set the ConsumerGroupType to Unknown */
+                if (error == NULL) {


Suggested change

if (error == NULL) {

if (!error) {

pranavrth · 2025-04-14T08:43:28Z

src/rdkafka_admin.c

+        if ((newgroupres->error &&
+             newgroupres->error->code ==
+                 RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE) ||
+            (newgroupres->type == RD_KAFKA_CONSUMER_GROUP_TYPE_CONSUMER &&
+             newgroupres->error &&
+             (newgroupres->error->code ==
+                  RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND ||
+              newgroupres->error->code ==
+                  RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION))) {


Refactor this condition to a new function. Something like do_fallback_to_old_consumer_group_descibe()

pranavrth · 2025-04-14T08:49:51Z

src/rdkafka_admin.c

+                static const struct rd_kafka_admin_worker_cbs cbs = {
+                    rd_kafka_admin_DescribeConsumerGroupsRequest,
+                    rd_kafka_DescribeConsumerGroupsResponse_parse,
+                };
+                rd_kafka_op_t *rko = rd_kafka_admin_request_op_new(
+                    rko_fanout->rko_rk, RD_KAFKA_OP_DESCRIBECONSUMERGROUPS,
+                    RD_KAFKA_EVENT_DESCRIBECONSUMERGROUPS_RESULT, &cbs,
+                    &rko_fanout->rko_u.admin_request.options,
+                    rko_fanout->rko_rk->rk_ops);
+
+                rko->rko_u.admin_request.fanout_parent = rko_fanout;
+                rko->rko_u.admin_request.broker_id =
+                    RD_KAFKA_ADMIN_TARGET_COORDINATOR;
+                rko->rko_u.admin_request.coordtype = RD_KAFKA_COORD_GROUP;
+                rko->rko_u.admin_request.coordkey  = rd_strdup(grp);
+
+                /* Set the group name as the opaque so the fanout worker use it
+                 * to fill in errors.
+                 * References rko_fanout's memory, which will always outlive
+                 * the fanned out op. */
+                rd_kafka_AdminOptions_set_opaque(
+                    &rko->rko_u.admin_request.options, (void *)grp);
+
+                rd_list_init(&rko->rko_u.admin_request.args, 1, rd_free);
+                rd_list_add(&rko->rko_u.admin_request.args, rd_strdup(grp));
+
+                rd_kafka_q_enq(rko_fanout->rko_rk->rk_ops, rko);


Extract this into a method as well. We can use this method below as well.

pranavrth · 2025-04-14T08:53:42Z

src/rdkafka_admin.c

@@ -8453,8 +8786,8 @@ void rd_kafka_DescribeConsumerGroups(rd_kafka_t *rk,
         *        coordinator into one op. */
        for (i = 0; i < groups_cnt; i++) {
                static const struct rd_kafka_admin_worker_cbs cbs = {


We can use earlier created method here as well.

pranavrth · 2025-04-14T08:56:14Z

src/rdkafka_admin.c

@@ -8296,9 +8323,11 @@ rd_kafka_DescribeConsumerGroupsResponse_parse(rd_kafka_op_t *rko_req,
                        client_id   = RD_KAFKAP_STR_DUP(&ClientId);
                        client_host = RD_KAFKAP_STR_DUP(&ClientHost);

+                        /* Target Assignment is NULL for the classic protocol */


Suggested change

/* Target Assignment is NULL for the classic protocol */

/* Target Assignment is `NULL` for the `classic` protocol as there is no concept of Target Assignment there. */

pranavrth · 2025-04-14T08:58:30Z

src/rdkafka_admin.h

@@ -527,6 +527,8 @@ struct rd_kafka_MemberDescription_s {
        char *group_instance_id;                /**< Group instance id */
        char *host;                             /**< Group member host */
        rd_kafka_MemberAssignment_t assignment; /**< Member assignment */
+        rd_kafka_MemberAssignment_t
+            *target_assignment; /**< Target assignment */


Suggested change

*target_assignment; /**< Target assignment */

*target_assignment; /**< Target assignment. `NULL` for `classic` protocol */

pranavrth · 2025-04-14T10:12:12Z

tests/0081-admin.c

+                if (i < 2) {
+                        /* Classic Protocol */
+                        test_conf_init(&conf, NULL, 0);
+                        test_conf_set(conf, "group.protocol", "classic");
+                } else {
+                        /* Consumer Protocol */
+                        test_conf_init(&conf, NULL, 0);
+                        test_conf_set(conf, "group.protocol", "consumer");
+                }


Use protocols directly instead of this if-else.

Suggested change

if (i < 2) {

/* Classic Protocol */

test_conf_init(&conf, NULL, 0);

test_conf_set(conf, "group.protocol", "classic");

} else {

/* Consumer Protocol */

test_conf_init(&conf, NULL, 0);

test_conf_set(conf, "group.protocol", "consumer");

}

test_conf_init(&conf, NULL, 0);

test_conf_set(conf, "group.protocol", protocols[i]);

pranavrth · 2025-04-14T10:13:13Z

tests/0081-admin.c

+                         "client_id_%" PRId32, i);
+
+                test_conf_set(conf, "client.id", client_ids[i]);
+                test_conf_set(conf, "session.timeout.ms", "5000");


This property is not needed for this test.

pranavrth · 2025-04-14T10:25:04Z

tests/0081-admin.c

+        while (1) {
+                rkev = rd_kafka_queue_poll(q, tmout_multip(20 * 1000));
+                TEST_SAY("DescribeConsumerGroups: got %s in %.3fms\n",
+                         rd_kafka_event_name(rkev),
+                         TIMING_DURATION(&timing) / 1000.0f);
+                if (rkev == NULL)
+                        continue;
+                if (rd_kafka_event_error(rkev))
+                        TEST_SAY("%s: %s\n", rd_kafka_event_name(rkev),
+                                 rd_kafka_event_error_string(rkev));
+
+                if (rd_kafka_event_type(rkev) ==
+                    RD_KAFKA_EVENT_DESCRIBECONSUMERGROUPS_RESULT) {
+                        break;
+                }
+
+                rd_kafka_event_destroy(rkev);
+        }


We can extract this functionality into a method in the same file or in test.c file which can be used by other functions in this test file.

pranavrth · 2025-04-14T10:26:16Z

tests/0081-admin.c

+                expected_DescribeConsumerGroups_result_t *exp  = &expected[i];
+                rd_kafka_resp_err_t exp_err                    = exp->err;


Use expected[i] directly.

pranavrth · 2025-04-14T10:28:05Z

tests/0081-admin.c

+
+                if (strcmp(protocols[i], "classic") == 0) {
+                        TEST_ASSERT(
+                            state == RD_KAFKA_CONSUMER_GROUP_STATE_STABLE ||
+                                state == RD_KAFKA_CONSUMER_GROUP_STATE_EMPTY,
+                            "Expected Stable or Empty state for classic "
+                            "protocol, got %s.",
+                            rd_kafka_consumer_group_state_name(state));
+                } else if (strcmp(protocols[i], "consumer") == 0) {
+                        TEST_ASSERT(state ==
+                                        RD_KAFKA_CONSUMER_GROUP_STATE_STABLE,
+                                    "Expected Stable state, got %s.",
+                                    rd_kafka_consumer_group_state_name(state));
+                }


Why are we not checking the type here?

pranavrth · 2025-04-14T10:54:11Z

src/rdkafka_admin.c

+
+        nodeid = rkb->rkb_nodeid;
+        rd_kafka_broker_lock(rkb);
+        host = rd_strdup(rkb->rkb_origname);


Free node as well.

pranavrth · 2025-04-14T11:16:50Z

tests/0081-admin.c

+        } else {
+                do_test_DescribeConsumerGroups_Compatibility("temp queue", rk,
+                                                             NULL, -1);
+                do_test_DescribeConsumerGroups_Compatibility("main queue", rk,
+                                                             mainq, 1500);


These all test can be tested in all the scenarios starting Broker version 4.0. Including if condition as well.

You will need to use IncrementalAlterConfigs to change the session timeout and heartbeat timeout for consumer protocol case though for the consumer to leave the group and delete the group.

pranavrth · 2025-04-14T11:19:34Z

tests/0081-admin.c

+                } else {
+                        /* Consumer Protocol */
+                        test_conf_init(&conf, NULL, 0);
+                        test_conf_set(conf, "group.protocol", "consumer");


This is overriden in the test_create_consumer() function. This test would fail in that case if we test for the type as well.

pranavrth · 2025-04-14T13:14:50Z

src/rdkafka_admin.c

+                        rd_kafka_buf_skip_tags(reply);
+
+                        member_id   = RD_KAFKAP_STR_DUP(&MemberId);
+                        instance_id = RD_KAFKAP_STR_DUP(&InstanceId);


This can be NULL. It will be a Segmentation fault in this case. Fix this.

pranavrth · 2025-04-14T13:20:57Z

src/rdkafka_admin.c

+        RD_IF_FREE(host, rd_free);
+        RD_IF_FREE(node, rd_kafka_Node_destroy);
+        rd_kafka_buf_skip_tags(reply);


Suggested change

RD_IF_FREE(host, rd_free);

RD_IF_FREE(node, rd_kafka_Node_destroy);

rd_kafka_buf_skip_tags(reply);

rd_kafka_buf_skip_tags(reply);

RD_IF_FREE(host, rd_free);

RD_IF_FREE(node, rd_kafka_Node_destroy);

pranavrth · 2025-04-14T13:27:57Z

src/rdkafka_admin.c

+             groupres->error->code == RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE) ||
+            (groupres->type == RD_KAFKA_CONSUMER_GROUP_TYPE_CONSUMER &&


Check RD_KAFKA_CONSUMER_GROUP_TYPE_CONSUMER for UNSUPPORTED_FEATURE error as well. If we get this error (which might happen in old brokers), we will keep on looping.

emasab

Thanks Pratyush, these are my comments about the PR

emasab · 2025-04-14T16:56:22Z

CHANGELOG.md

@@ -39,6 +39,8 @@ librdkafka v2.10.0 is a feature release:
   leader change and offset validation (#4970).
 * Fix the Nagle algorithm (TCP_NODELAY) on broker sockets to not be enabled
   by default (#4986).
+ * [KIP-848] `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment has also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).
+ * [KIP-848] Group Config is now supported in IncrementalAlterConfigs and DescribeConfigs. (#4939).


Suggested change

* [KIP-848] Group Config is now supported in IncrementalAlterConfigs and DescribeConfigs. (#4939).

* [KIP-848] Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs. (#4939).

emasab · 2025-04-14T16:57:01Z

CHANGELOG.md

@@ -39,6 +39,8 @@ librdkafka v2.10.0 is a feature release:
   leader change and offset validation (#4970).
 * Fix the Nagle algorithm (TCP_NODELAY) on broker sockets to not be enabled
   by default (#4986).
+ * [KIP-848] `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment has also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).


Suggested change

* [KIP-848] `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment has also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).

* [KIP-848] `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).

emasab · 2025-04-14T17:02:59Z

examples/describe_consumer_groups.c

+        const rd_kafka_MemberAssignment_t *target_assignment =
+            rd_kafka_MemberDescription_target_assignment(member);
+        const rd_kafka_topic_partition_list_t *target_topic_partitions =
+            rd_kafka_MemberAssignment_partitions(target_assignment);


This function shouldn't check the argument is NULL, instead it mustn't be called with a NULL argument.

Suggested change

rd_kafka_MemberAssignment_partitions(target_assignment);

target_assignment

? rd_kafka_MemberAssignment_partitions(target_assignment)

: NULL;

emasab · 2025-04-14T17:03:41Z

examples/describe_consumer_groups.c

@@ -212,9 +228,10 @@ static void print_group_info(const rd_kafka_ConsumerGroupDescription_t *group) {
                         rd_kafka_Node_port(coordinator));
        }
        printf(
-            "Group \"%s\", partition assignor \"%s\", "
+            "Group \"%s\", partition assignor \"%s\", type \"%s\" "
            " state %s%s, with %" PRId32 " member(s)\n",


As with other field let's add the quotes

Suggested change

" state %s%s, with %" PRId32 " member(s)\n",

" state \"%s\"%s, with %" PRId32 " member(s)\n",

emasab · 2025-04-14T17:05:11Z

src/rdkafka_admin.c

+        if (!assignment)
+                return NULL;


Same, here it shouldn't check for NULL

src/rdkafka_admin.h

src/rdkafka_proto.h

src/rdkafka_request.c

emasab · 2025-04-14T17:46:13Z

src/rdkafka_request.c

+        size_t i;
+
+        int16_t ApiVersion = rd_kafka_broker_ApiVersion_supported(
+            rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, 0, NULL);


We need to create the local _UNSUPPORTED_FEATURE error after getting the ApiVersion

Suggested change

rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, 0, NULL);

rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, 0, NULL);

if (ApiVersion == -1) {

return rd_kafka_error_new(

RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE,

"ConsumerGroupDescribe (KIP-848) "

"not supported by broker, "

"requires broker version >= 4.0.0");

}

This returns the error on the request part itself and then the admin worker fails. We removed it during the review. Check -

librdkafka/src/rdkafka_admin.c

Lines 8176 to 8186 in 85b9cb4

error = rd_kafka_ConsumerGroupDescribeRequest(

rkb, (char **)groups->rl_elems, groups_cnt,

include_authorized_operations, replyq, resp_cb, opaque);

if (error) {

rd_snprintf(errstr, errstr_size, "%s",

rd_kafka_error_string(error));

err = rd_kafka_error_code(error);

rd_kafka_error_destroy(error);

return err;

}

We are getting same error in the response merge function if we remove it from here through rd_kafka_send I think.

In case returning this error as in other requests, the duplicate

rd_kafka_enq_once_del_source(eonce, "coordinator response");

on rd_kafka_admin_coord_request should be removed otherwise it the eonce id destroyed before calling the callback.

We are getting same error in the response merge function if we remove it from here through rd_kafka_send I think.

Yes the error is returned by rd_kafka_send but it's not correct to continue with request construction if the ApiVersion is -1

Changed it to not fail when the admin result when request is rd_kafka_admin_ConsumerGroupDescribeRequest and the err returned is Unsupported feature

I don't know if this is correct to hardcode a condition for this specific case in general rd_kafka_admin_coord_request function. Let's understand this offline.

src/rdkafka_admin.c

examples/describe_consumer_groups.c

…kafka_admin_ConsumerGroupDescribeRequest

src/rdkafka_proto.h

src/rdkafka_admin.c

pranavrth · 2025-04-15T08:06:00Z

src/rdkafka_admin.c

+           UNSUPPORTED_VERSION(35) we need to send a request to the old
+           protocol.
+ */
+static rd_bool_t rd_kafka_admin_describe_consumer_group_fallback_to_classic(


Suggested change

static rd_bool_t rd_kafka_admin_describe_consumer_group_fallback_to_classic(

static rd_bool_t rd_kafka_admin_describe_consumer_group_do_fallback_to_classic(

pranavrth · 2025-04-15T08:06:52Z

src/rdkafka_admin.c

+        if (groupres->error &&
+            (groupres->error->code == RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND ||
+             groupres->error->code == RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION ||
+             groupres->error->code == RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE)) {
+                return rd_true;
+        }
+        return rd_false;


Suggested change

if (groupres->error &&

(groupres->error->code == RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND ||

groupres->error->code == RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION ||

groupres->error->code == RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE)) {

return rd_true;

}

return rd_false;

return groupres->error &&

(groupres->error->code == RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND ||

groupres->error->code == RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION ||

groupres->error->code == RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE);

emasab

Thanks Prayush and Pranav! Approving

pranavrth

LGTM!. Thanks Pratyush and Emanuele.

PratRanj07 requested a review from a team as a code owner December 27, 2024 11:22

pranavrth requested changes Feb 7, 2025

View reviewed changes

PratRanj07 requested a review from pranavrth February 12, 2025 11:38

pranavrth requested changes Feb 14, 2025

View reviewed changes

airlock-confluentinc bot force-pushed the dev_ConsumerGroupDescribeApi branch from 2b0883d to b09e721 Compare April 9, 2025 11:13

pranavrth requested changes Apr 10, 2025

View reviewed changes

airlock-confluentinc bot force-pushed the dev_ConsumerGroupDescribeApi branch 2 times, most recently from 7075a8c to ee0e92f Compare April 11, 2025 14:11

pranavrth requested changes Apr 11, 2025

View reviewed changes

airlock-confluentinc bot force-pushed the dev_ConsumerGroupDescribeApi branch 2 times, most recently from 55b2fe9 to 5b7ed97 Compare April 14, 2025 06:27

pranavrth reviewed Apr 14, 2025

View reviewed changes

src/rdkafka.h Show resolved Hide resolved

pranavrth reviewed Apr 14, 2025

View reviewed changes

emasab requested changes Apr 14, 2025

View reviewed changes

pranavrth reviewed Apr 14, 2025

View reviewed changes

pranavrth requested changes Apr 14, 2025

View reviewed changes

pranavrth reviewed Apr 14, 2025

View reviewed changes

pranavrth requested changes Apr 14, 2025

View reviewed changes

emasab requested changes Apr 14, 2025

View reviewed changes

emasab reviewed Apr 14, 2025

View reviewed changes

examples/describe_consumer_groups.c Outdated Show resolved Hide resolved

PratRanj07 added 2 commits April 15, 2025 01:17

first pass

05b1aa6

debugging

ba04853

PratRanj07 added 18 commits April 15, 2025 01:17

removing unused variables

17331c8

removing more unused variables

9aa3b73

removing segfault error-freeing groups_arr

a4e0a7e

Code simplifications and Error handling change

3cc4c56

removing unused variable

87d2ad7

Changed error handling in ConsumerGroupDescribeResponseParse

e58f2dc

Removing target partition getter and memory leak

9e135f8

Removing RD_KAFKA_EVENT_CONSUMERGROUPDESCRIBE_RESULT

4fe6c16

unnecessary change

e88ab31

Changelog change for configs

b1e4b4c

Changed error handling in destroy method

026f45c

Style fix

d4530b9

Code refactoring and comments addressed

258ccf1

Code refactoring and comments addressed-admin file

c1250f2

Changed error handling

db24d44

style fix and moving test to another PR

24c6af4

Removing unused variables

f84bf53

error handling for previous versions and comments addressed

39f225d

airlock-confluentinc bot force-pushed the dev_ConsumerGroupDescribeApi branch from 85b9cb4 to 39f225d Compare April 14, 2025 19:48

PratRanj07 added 3 commits April 15, 2025 02:16

Unsupported error being returned by request

0d86c64

Request should not fail only when unsupported feature returned by rd_…

02cb969

…kafka_admin_ConsumerGroupDescribeRequest

Style fix

572f7a8

emasab requested changes Apr 15, 2025

View reviewed changes

src/rdkafka_proto.h Outdated Show resolved Hide resolved

src/rdkafka_admin.c Outdated Show resolved Hide resolved

src/rdkafka_admin.c Outdated Show resolved Hide resolved

src/rdkafka_admin.c Outdated Show resolved Hide resolved

pranavrth requested changes Apr 15, 2025

View reviewed changes

Remove hardcoded error handling in coord request

18065ee

emasab approved these changes Apr 15, 2025

View reviewed changes

pranavrth approved these changes Apr 15, 2025

View reviewed changes

emasab merged commit 8bf0a67 into master Apr 15, 2025
2 checks passed

emasab deleted the dev_ConsumerGroupDescribeApi branch April 15, 2025 09:44

PratRanj07 mentioned this pull request Apr 17, 2025

KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups confluentinc/confluent-kafka-go#1418

Merged

		@@ -2,6 +2,7 @@

		librdkafka v2.8.0 is a maintenance release:

		* DescribeConsumerGroup now supports new consumer protocol groups (#4922).

		const rd_kafka_topic_partition_list_t *target_topic_partitions =
		rd_kafka_MemberAssignment_partitions(target_assignment);

		@@ -160,6 +160,7 @@ librdkafka v2.10.0 is a feature release:

		librdkafka v2.8.0 is a maintenance release:

		* DescribeConsumerGroup now supports new consumer protocol groups. Two field type and target assignment has also been added. Target Assignment is only valid for the new Consumer protocol and its default value is NULL. (#4922).

	* DescribeConsumerGroup now supports new consumer protocol groups. Two field type and target assignment has also been added. Target Assignment is only valid for the new Consumer protocol and its default value is NULL. (#4922).
	* `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment has also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).

	* @return The target assignment. It returns NULL for CLASSIC GROUPS.
	* @return The target assignment for `consumer` group types. It returns NULL for the `classic` group types.

		ofGroupsArrayCnt = rd_kafka_buf_write_arraycnt_pos(rkbuf);
		rd_kafka_buf_finalize_arraycnt(rkbuf, ofGroupsArrayCnt, group_cnt);

	ofGroupsArrayCnt = rd_kafka_buf_write_arraycnt_pos(rkbuf);
	rd_kafka_buf_finalize_arraycnt(rkbuf, ofGroupsArrayCnt, group_cnt);
	rd_kafka_buf_write_arraycnt(rkbuf, group_cnt);

	* It returns NULL for the `classic` group types.
	* Returns NULL for the `classic` group types.

		* @remark The lifetime of the returned memory is the same
		* as the lifetime of the \p grpdesc object.

	/* Target Assignment is NULL for the classic protocol */
	/* Target Assignment is `NULL` for the `classic` protocol as there is no concept of Target Assignment there. */

	target_assignment; /< Target assignment /
	target_assignment; /< Target assignment. `NULL` for `classic` protocol /

		expected_DescribeConsumerGroups_result_t *exp = &expected[i];
		rd_kafka_resp_err_t exp_err = exp->err;

		groupres->error->code == RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE) \|\|
		(groupres->type == RD_KAFKA_CONSUMER_GROUP_TYPE_CONSUMER &&

	* [KIP-848] Group Config is now supported in IncrementalAlterConfigs and DescribeConfigs. (#4939).
	* [KIP-848] Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs. (#4939).

	* [KIP-848] `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment has also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).
	* [KIP-848] `rd_kafka_DescribeConsumerGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).

	" state %s%s, with %" PRId32 " member(s)\n",
	" state \"%s\"%s, with %" PRId32 " member(s)\n",

-            rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, 0, NULL);
+            rkb, RD_KAFKAP_ConsumerGroupDescribe, 0, 0, NULL);
+        if (ApiVersion == -1) {
+                return rd_kafka_error_new(
+                    RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE,
+                    "ConsumerGroupDescribe (KIP-848) "
+                    "not supported by broker, "
+                    "requires broker version >= 4.0.0");
+        }

	error = rd_kafka_ConsumerGroupDescribeRequest(
	rkb, (char **)groups->rl_elems, groups_cnt,
	include_authorized_operations, replyq, resp_cb, opaque);

	if (error) {
	rd_snprintf(errstr, errstr_size, "%s",
	rd_kafka_error_string(error));
	err = rd_kafka_error_code(error);
	rd_kafka_error_destroy(error);
	return err;
	}

	static rd_bool_t rd_kafka_admin_describe_consumer_group_fallback_to_classic(
	static rd_bool_t rd_kafka_admin_describe_consumer_group_do_fallback_to_classic(

KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #4941

KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #4941

Uh oh!

Conversation

PratRanj07 commented Dec 27, 2024

Uh oh!

confluent-cla-assistant bot commented Dec 27, 2024

Uh oh!

pranavrth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pranavrth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pranavrth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment