List consumer group offsets #49

PratRanj07 · 2024-05-21T10:01:25Z

Created ListConsumerGroupOffsets API with the name of fetchOffsets and wrote examples and test cases for it.

cla-assistant · 2024-05-21T10:01:45Z

All committers have signed the CLA.

…roupOffsets

milindl

Good work @PratRanj07 to implement this.

I have done a first round of review.
Things still pending to be reviewed by me:

example added
run leak checks with valgrind

and of course any changes you make due to the comments.

milindl · 2024-05-29T05:47:38Z

lib/kafkajs/_admin.js

@@ -366,6 +366,38 @@ class Admin {
      });
    });
  }
+
+  /**
+   *   Fetch Offsets


nit: Just 1 space before Fetch Offsets

milindl · 2024-05-29T05:48:44Z

lib/kafkajs/_admin.js

+
+  /**
+   *   Fetch Offsets
+   * @param {string} options.groupId - The group id.


documentation:

Suggested change

* @param {string} options.groupId - The group id.

* @param {string} options.groupId - The group ID to fetch offsets for.

milindl · 2024-05-29T05:49:42Z

lib/kafkajs/_admin.js

+    if (this.#state !== AdminState.CONNECTED) {
+      throw new error.KafkaJSError("Admin client is not connected.", { code: error.ErrorCodes.ERR__STATE });
+    }
+    //console.log(options);


milindl · 2024-05-29T05:51:53Z

lib/kafkajs/_admin.js

+    }
+    //console.log(options);
+    if(options.resolveOffsets){
+      throw new Error("resolveOffsets is not yet implemented.");


Throw the correct error type. Check the file to see how it's done for waitForLeaders.

Change the check within the if block also. Currently the check won't work if someone has set resolveOffsets as false. We want to indicate non-support of the key entirely.

milindl · 2024-05-29T05:54:29Z

lib/kafkajs/_admin.js

+   * @param {boolean} options.resolveOffsets - not yet implemented 
+   * @param {number?} options.timeout - The request timeout in milliseconds.
+   *                                    May be unset (default: 5000)
+   * @param {boolean?} options.requireStableOffsets


documentation: Requires documentation and default value

milindl · 2024-05-29T07:48:35Z

src/common.h

@@ -38,6 +38,8 @@ template<> std::vector<std::string> GetParameter<std::vector<std::string> >(
 template<> v8::Local<v8::Array> GetParameter<v8::Local<v8::Array> >(
  v8::Local<v8::Object>, std::string, v8::Local<v8::Array>);
 // template int GetParameter<int>(v8::Local<v8::Object, std::string, int);
+rd_kafka_topic_partition_list_t *v8ArrayToTopicPartitionList(


This should probably be within namespace TopicPartition. With a different name. Maybe FromGroupedTopicPartitionV8Array. Not really sure about the name for now.

milindl · 2024-05-29T08:13:08Z

test/promisified/admin/list_consumer_group_offsets.spec.js

+    ).rejects.toHaveProperty("code", ErrorCodes.ERR__TIMED_OUT);
+  });
+
+  test("should return correct offset after consuming messages", async () => {


add a test with topics unset or null too

add a test with multiple topics each with more than 1 partition so the result format can be verified

milindl · 2024-05-29T08:14:40Z

test/promisified/admin/list_consumer_group_offsets.spec.js

@@ -0,0 +1,129 @@
+jest.setTimeout(30000);


rename this file to be same as the function name. Let's call it 'list consumer groups' only when we actually need to call into librdkafka. Otherwise it gets confusing to mix the names.

milindl · 2024-05-29T08:18:44Z

types/kafkajs.d.ts

@@ -259,6 +259,10 @@ export interface PartitionOffset {
  offset: string
 }

+export type FetchOffsetsPartition = PartitionOffset & { metadata: string | null }
+
+export type TopicInput = string | { topic: string; partitions: number[] }


To be consistent with how we have done things before, define these types in rdkafka.d.ts, and import them here. Re-export from this file. (See the top of this file for examples, one such type is GroupOverview)

milindl · 2024-05-29T08:19:14Z

types/rdkafka.d.ts

@@ -11,7 +11,7 @@ import {

 export * from './config';
 export * from './errors';
-import { Kafka } from './kafkajs';
+import { FetchOffsetsPartition, Kafka, TopicInput } from './kafkajs';


Remove this line entirely after following the comment from kafkajs.d.ts (Kafka was unused anyway)

EricMCornelius · 2024-10-18T17:58:46Z

Any ETA on this? You guys are promoting this client as a KafkaJS replacement while lacking some pretty basic functionality on the admin side here...

EricMCornelius · 2024-10-18T18:00:48Z

https://github.com/confluentinc/confluent-kafka-javascript/blob/master/MIGRATION.md#admin-client

At the very least, maybe keep major omissions up to date in any migration guides?

emasab · 2024-10-21T12:03:57Z

types/rdkafka.d.ts

@@ -438,6 +442,12 @@ export interface IAdminClient {
    deleteGroups(groupIds: string[],
        options?: { timeout?: number },
        cb?: (err: LibrdKafkaError, result: DeleteGroupsResult[]) => any): void;
+    fetchOffsets(options: { groupId: string,


for the node-rdkafka API, it's a new method, so we should have the same signature and name as listConsumerGroupOffsets, allowing multiple groups. In future, we'll deprecate fetchOffsets and have a new method named listConsumerGroupOffsets in KafkaJS API too.

milindl

Thanks for the changes!

Please also add a CHANGELOG.md entry, and merge the master branch to it. Once you do that we will change the base branch of this PR to master.

milindl · 2024-07-11T08:53:37Z

lib/admin.js

+   * @param {number?} options.timeout - The request timeout in milliseconds.
+   *                                    May be unset (default: 5000)
+   * @param {boolean?} options.requireStableOffsets - Whether broker should return stable offsets
+ *                                      (transaction-committed). (default: false)


indentation for the * is off

milindl · 2024-07-11T09:05:34Z

lib/admin.js

+      options.requireStableOffsets = false;
+    }
+
+    let originalTopics = null;


About the comment:
I see that you have added the comment which explains what you are doing, but a better comment should explain why you are doing it rather than the 'what' (which a developer can figure out through reading the code also). However, the 'why' is something that the other developer can't figure out on their own.

So for instance, you should say that, 'If the input is a list of topic string, the user expects us to fetch offsets for all all partitions of all the input topics. In librdkafka, we can only fetch offsets by topic partitions, or else, we can fetch all of them. This, we must fetch offsets for all topic partitions (by settings topics to null) and filter by the topic strings later.'

^ something like this. You can add this to your comment to annotate both the 'what' and the 'why'.

milindl · 2024-07-11T09:07:10Z

lib/kafkajs/_admin.js

+   * @param {number?} options.timeout - The request timeout in milliseconds.
+   *                                    May be unset (default: 5000)
+   * @param {boolean?} options.requireStableOffsets - Whether broker should return stable offsets
+ *                                      (transaction-committed). (default: false)


again the indentation is off for this *

nit: "(transaction-committed)" should be aligned to "Whether"

milindl · 2024-07-11T09:13:01Z

lib/admin.js

+   * Fetch Offsets
+   * 
+   * @param {string} options.groupId - The group ID to fetch offsets for.
+   * @param {import('../types/rdkafka').TopicInput} options.topics - The topics to fetch offsets for.


Use instead

import("../../types/kafkajs").TopicInput

milindl · 2024-07-11T09:13:27Z

lib/kafkajs/_admin.js

+   * Fetch Offsets
+   * 
+   * @param {string} options.groupId - The group ID to fetch offsets for.
+   * @param {import('../../types/rdkafka').TopicInput} options.topics - The topics to fetch offsets for.


Use instead

import("../../types/kafkajs").TopicInput

milindl · 2024-10-22T07:18:36Z

src/admin.cc

+    const rd_kafka_group_result_t **results =
+        rd_kafka_ListConsumerGroupOffsets_result_groups(result, &result_cnt);
+
+    // Change the type of the 'error' pointer to 'const rd_kafka_error_t *'


Let's move this processing outside this function and let the caller process this. This sort of works for now, but when/if we add the capacity for multiple groups, it won't.

Let's handle only top level errors here, and move this group-specific errors out.

So with the change that Emanuele suggested, you will need to change the FromFetchOffsetsResult function as well, since the return type signature will be different for listConsumerGroupOffsets.

However, the kafkajs-compatible signature remains the same, so you'll need to convert it in fetchOffsets in _admin.js

milindl · 2024-10-22T07:28:17Z

test/promisified/admin/fetch_offsets.spec.js

+                offset: (parseInt(message.offset, 10) + 1).toString(),
+              },
+            ]);
+            await consumer.stop();


Remove as this has no effect, here and elsewhere

milindl · 2024-10-22T07:28:54Z

test/promisified/admin/fetch_offsets.spec.js

+            await consumer.stop();
+          }
+        } catch (error) {
+          if (error.message.includes("Offset out of range")) {


I'm not sure when you'll get this error, because it's handled automatically. Maybe remove the try-catch block entirely?

Here, and elsewhere.

milindl · 2024-10-22T07:39:47Z

src/common.h

+
+v8::Local<v8::Array> FromFetchOffsetsResult(
+    const rd_kafka_ListConsumerGroupOffsets_result_t *result);
+}// namespace Admin


nit

Suggested change

}// namespace Admin

} // namespace Admin

milindl · 2024-10-22T07:41:59Z

src/workers.cc

+ * @brief Fetch Offsets in an asynchronous worker
+ *
+ * This callback will list all the consumer group offsets for the specified
+ * topic offsets.


Suggested change

* topic offsets.

* groups' topic partitions.

* Add data-governance to code owners for schema registry clients * Fix ownership

* npmignore more files and fix to the update versions script * Improved Releasing section

* Changelog entry for #123

confluent-cla-assistant · 2024-10-25T08:46:03Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

milindl

Pretty good work to address the review comments. I added a final set of comments, please take a look

milindl · 2024-10-30T10:46:59Z

types/rdkafka.d.ts

@@ -86,6 +86,10 @@ export interface TopicPartitionOffsetAndMetadata extends TopicPartitionOffset {

 export type TopicPartitionTime = TopicPartitionOffset;

+export type FetchOffsetsPartition = PartitionOffset & { metadata: string | null, leaderEpoch: number | null, error?: LibrdKafkaError };  


This isn't used in this file. It's just exported. Define this instead in kafkajs.d.ts.

milindl · 2024-10-30T10:47:30Z

types/rdkafka.d.ts

@@ -11,7 +11,7 @@ import {

 export * from './config';
 export * from './errors';
-import { Kafka } from './kafkajs';
+import { PartitionOffset } from './kafkajs';


Remove this line entirely after following the comment related to FetchOffsetsPartition.

milindl · 2024-10-30T10:50:56Z

types/rdkafka.d.ts

@@ -86,6 +86,10 @@ export interface TopicPartitionOffsetAndMetadata extends TopicPartitionOffset {

 export type TopicPartitionTime = TopicPartitionOffset;

+export type FetchOffsetsPartition = PartitionOffset & { metadata: string | null, leaderEpoch: number | null, error?: LibrdKafkaError };  
+
+export type TopicInput = string[] | { topic: string; partitions: number[] }[]


Same as FetchOffsetsPartition, it is not used in this file so define in the other file only.

milindl · 2024-10-30T11:14:42Z

lib/kafkajs/_admin.js

+           */
+          const topicPartitionMap = new Map();
+
+          offsets.forEach(groupResult => {


You should make sure here that groupResult.length is exactly 1 and just use that element. The forEach is redundant

milindl · 2024-10-30T11:17:42Z

lib/kafkajs/_admin.js

+          }));
+
+          if (originalTopics !== null) {
+            convertedOffsets = convertedOffsets.filter(convertedOffsets => originalTopics.includes(convertedOffsets.topic));


nit

Suggested change

convertedOffsets = convertedOffsets.filter(convertedOffsets => originalTopics.includes(convertedOffsets.topic));

convertedOffsets = convertedOffsets.filter(convertedOffset => originalTopics.includes(convertedOffset.topic));

milindl · 2024-10-30T11:47:45Z

examples/kafkajs/fetchOffsets.js

@@ -0,0 +1,68 @@
+// require('kafkajs') is replaced with require('@confluentinc/kafka-javascript').KafkaJS.


This file still needs to be moved to examples/kafkajs/admin/fetch-offsets.js, and use parseArgs appropriately for argument parsing. You can follow this file for an idea of how to do it: examples/kafkajs/admin/describe-groups.js

Use arguments for bootstrap server, requireStableOffsets and timeout. Use positionals for the group name and the topic/partitions as you are currently doing, it will be something like

const args = parseArgs({ allowPositionals: true, options: { 'bootstrap-servers': { type: 'string', short: 'b', default: 'localhost:9092', }, 'timeout': { type: 'string', short: 'm', default: 5000, }, ... }, }); and then args.values will contain option values, and args.positionals will contain the rest of the arguments which you can parse with your function that you've already written.

milindl · 2024-10-30T12:22:24Z

src/admin.cc

+    v8::Local<v8::Object> listGroupOffsetObj =
+        listGroupOffsets->Get(context, i).ToLocalChecked().As<v8::Object>();


Here and at other places, use Nan::Get instead. You don't need to make the context either. Handle the error case properly too, in case the user has not provided a proper object.

v8::Local<v8::Value> listGroupOffsetValue = Nan::Get(listGroupOffsets, i).ToLocalChecked(); if (!listGroupOffsetValue->IsObject()) { return Nan::ThrowError("Each entry must be an object"); } v8::Local<v8::Object> listGroupOffsetObj = listGroupOffsetValue.As<v8::Object>();

milindl · 2024-10-30T12:29:25Z

src/common.cc

+    v8::Isolate* isolate = v8::Isolate::GetCurrent();
+    v8::Local<v8::Context> context = isolate->GetCurrentContext();
+
+    v8::Local<v8::Value> topicVal =


Just use GetParameter<std::string> to get the string of the topic name directly, no need for isolate/context. The "item" is an object and thus can be used with GetParameter<std::string>.

Same goes for partitionVal and offset also.

milindl · 2024-10-30T12:30:43Z

src/admin.cc

+    return Nan::ThrowError("'listGroupOffsets' cannot be empty");
+  }
+
+  rd_kafka_ListConsumerGroupOffsets_t **requests =


Add a comment above this telling the reader that the ownership for this is taken by Workers::AdminClientListConsumerGroupOffsets and freeing it is also handled by that class.

milindl · 2024-10-30T12:40:09Z

src/common.cc

+
+      // Set partition-level error (if any)
+      if (partition->err != RD_KAFKA_RESP_ERR_NO_ERROR) {
+        RdKafka::ErrorCode code = static_cast<RdKafka::ErrorCode>(partition->err);


Add // NOLINT or split into two lines.

milindl

Great work. I'm approving the PR. There are some minor changes suggested. You can make them and merge this in.

milindl · 2024-11-04T06:09:04Z

CHANGELOG.md

+## Features
+
+1. Add support for an Admin API to list a consumer group's offsets (#49).
+
 # confluent-kafka-javascript v0.3.1


0.3.1 isn't released yet so remove this section and add "1. Fixes an issue where headers were not passed correctly to the eachBatch callback (#130)." within this list only. Call it "Enhancements" instead of features.

milindl · 2024-11-04T06:09:46Z

types/kafkajs.d.ts

+  fetchOffsets(options: { 
+    groupId: string,
+    topics?: TopicInput,  
+    resolveOffsets?: boolean,


Sorry I missed this earlier, remove resolveOffsets from the types. For users of typescript this makes their life better as they get a compile-time error instead of a run time error.

milindl · 2024-11-04T06:16:43Z

examples/kafkajs/admin/fetchOffsets.js

nit:
rename this file to fetch-offsets.js

PratRanj07 added 5 commits May 20, 2024 12:07

1st commit

63f05de

2nd commit

4445321

3rd commit

28aff0a

Added tests and examples

0b3a617

Formatting

3c4a907

PratRanj07 requested a review from a team as a code owner May 21, 2024 10:01

PratRanj07 requested a review from milindl May 21, 2024 10:01

PratRanj07 added 4 commits May 21, 2024 15:49

Formatting

76639a6

little change

6ff253b

some small changes

925fd21

Merge branch 'dev_early_access_development_branch' into listConsumerG…

5b769a8

…roupOffsets

milindl requested changes May 29, 2024

View reviewed changes

PratRanj07 added 5 commits May 30, 2024 12:53

changes requested

195a922

requested changes

380ee83

requested changes

254765b

name change

3ba54a1

indentation

ffe16f7

PratRanj07 requested a review from milindl June 3, 2024 13:43

indentation

a98699a

PratRanj07 requested a review from a team as a code owner June 4, 2024 06:36

emasab reviewed Oct 21, 2024

View reviewed changes

milindl requested changes Oct 22, 2024

View reviewed changes

emasab and others added 5 commits October 25, 2024 13:38

Add data-governance to code owners for schema registry clients (#52)

160f762

* Add data-governance to code owners for schema registry clients * Fix ownership

Fix deprecation warning

2a99288

Separate eachMessage and eachBatch internal consume loop

28e28e3

Update performance example with more cases

ffbbafd

Add per-partition cache with global expiry

42004b1

emasab and others added 11 commits October 25, 2024 13:41

v0.3.0 (#126)

b3712ba

* npmignore more files and fix to the update versions script * Improved Releasing section

Minor optimization to reduce schema ID lookups (#123)

1501a64

v0.3.0-RC2 (#127)

29bc526

v0.3.0 final release (#128)

4e42726

* Changelog entry for #123

Fix header conversion in eachBatch (#130)

e3de7e4

1st commit

0f3a167

2nd commit

1bde73a

3rd commit

ce5a4e9

changes requested

beafa7c

requested changes

a7c5aca

required Changes

99b0252

PratRanj07 changed the base branch from dev_early_access_development_branch to master October 25, 2024 08:46

PratRanj07 added 9 commits October 25, 2024 14:49

Merge master

f6f5b54

remove unnecessary changes

603ca2e

indentation and unnecessary changes

2f86c63

indentation

a758b90

comment removed

7085111

comment added

b2e28fa

changelog entry

92f262d

Changed topic partition js to c conversion structure

2d90d5b

refactoring

c435397

milindl requested changes Oct 30, 2024

View reviewed changes

Requested changes

7f6dd40

milindl approved these changes Nov 4, 2024

View reviewed changes

PratRanj07 added 2 commits November 4, 2024 11:59

final changes

1c1cfe8

Merge master

3c494e4

milindl approved these changes Nov 4, 2024

View reviewed changes

PratRanj07 merged commit e1bd8e8 into master Nov 4, 2024
2 checks passed

PratRanj07 deleted the listConsumerGroupOffsets branch November 4, 2024 06:52

	* @param {string} options.groupId - The group id.
	* @param {string} options.groupId - The group ID to fetch offsets for.

		@@ -86,6 +86,10 @@ export interface TopicPartitionOffsetAndMetadata extends TopicPartitionOffset {

		export type TopicPartitionTime = TopicPartitionOffset;

		export type FetchOffsetsPartition = PartitionOffset & { metadata: string \| null, leaderEpoch: number \| null, error?: LibrdKafkaError };

	convertedOffsets = convertedOffsets.filter(convertedOffsets => originalTopics.includes(convertedOffsets.topic));
	convertedOffsets = convertedOffsets.filter(convertedOffset => originalTopics.includes(convertedOffset.topic));

		@@ -0,0 +1,68 @@
		// require('kafkajs') is replaced with require('@confluentinc/kafka-javascript').KafkaJS.

		v8::Local<v8::Object> listGroupOffsetObj =
		listGroupOffsets->Get(context, i).ToLocalChecked().As<v8::Object>();

List consumer group offsets #49

List consumer group offsets #49

Uh oh!

Conversation

PratRanj07 commented May 21, 2024

Uh oh!

cla-assistant bot commented May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

milindl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EricMCornelius commented Oct 18, 2024

Uh oh!

EricMCornelius commented Oct 18, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milindl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

confluent-cla-assistant bot commented Oct 25, 2024

Uh oh!

milindl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cla-assistant bot commented May 21, 2024 •

edited

Loading