Description

In our environment, we noticed significant performance change between the

• 1.6.0 version (version() ('1.6.0', 17170432), libversion() ('1.6.0', 17170687)) and
• 1.5.0 version (version() ('1.5.0', 17104896), libversion() ('1.5.0', 17105151))
  by our consumers running on Debian Buster.

The older version manages to process all messages and the gap overall remains close to zero. If the newer version is installed in the same environment, the consumers performance drops significantly and the gap begins to grow.

Our client config looks like this:

  "kafka_client_config": {
      "bootstrap.servers": "s1,s2,s3,s4,s5,s6,s7",
      "group.id": "group_id",
      "auto.offset.reset": "latest",
      "session.timeout.ms": 16000, 
      "heartbeat.interval.ms": 3900,
      "partition.assignment.strategy": "roundrobin,range",
  }

and we instantiate the client as follows:

from confluent_kafka import Consumer as KafkaConsumer

...
consumer = KafkaConsumer(**kafka_client_config)
consumer.subscribe(topics)
...

In librdkafka release notes I noticed some notes on consumer changes:

The consumer assignment and consumer group implementations have been
decoupled, simplified and made more strict and robust

Maybe the robustness could have some impact on the performance? Has somebody else noticed similar problem? Perhaps, we only missed some configuration change needed to use the newer 1.6.0 version..

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system:
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue