Description

Hi,

This is a question concerning DeserializingConsumer (and to some extent SerializingProducer, i.e. should we be using these classes at all?), and generally a question about how to construct a robust consumer.

I noticed that the code for DeserializingConsumer is strange.
I'm wondering if I should follow the recommendation in the docstring and not use this class at all.

The code for DeserializingConsumer, has this error handling step where it checks if msg contains an error and then raises an exception no matter what error occurred:
https://github.com/confluentinc/confluent-kafka-python/blob/master/src/confluent_kafka/deserializing_consumer.py#L103-L104

if msg.error() is not None:
    raise ConsumeError(msg.error(), kafka_message=msg)

Specifically:
During poll, if msg contains an error: ConsumeError is raised.

Is this actually what you want to have happen when polling?

It seems to me that it should return a msg (KafkaMessage) with an error attached and let the application handle the error on it's own.

There are two error cases that I think should be handled differently to each other in a consumer:

1. If the message contains a fatal error: The consumer is now bad and needs to be recreated in some way (either recreate it by re-instantiating it, or kill the application instance and restart). Either of these behaviors seem fine to me: Return the message and let the application check if it's a fatal error OR raise an exception because this is unrecoverable and we need to recreate the consumer. Though I would probably let the application itself manage the error handling.
2. If the message contains a non-fatal error: Continue using the consumer: It should return msg with the error attached, and let the application manage the failure by continuing and calling poll again.

Because of what the DeserializingConsumer does, it prevents the common error-checking interface to be used and instead you must now go in and see if it's a ConsumeError, and if it is and then check the error through that.

As an example, I would expect this kind of error checking to happen if you use a Consumer, similar to the example given on the Confluent docs page, https://docs.confluent.io/kafka-clients/python/current/overview.html#basic-poll-loop:

while True:
    msg = consumer.poll(1.0)
    if msg is None:
        continue  # Received None from Kafka, continuing.
    elif msg is not None and msg.error():
        error = msg.error()  # Error is set, we have to check whether it's fatal or not.
        if error.fatal():
            break  # Ran into a fatal error: Consumer is bad. We have to stop using it.
        else:
            continue  # Ran into a non-fatal error: Consumer is fine but we have to poll again.
    else:
        success = fn(msg)  # Process the message
        if success:
            consumer.commit()
        else:
            pass  # Failed for some reason, retry. Unimportant for this example.
consumer.close()

This is not possible to do with DeserializingConsumer.poll, because it has a different output/behavior when compared to Consumer.poll. DeserializingConsumer.poll forces you to check the inner error message in the exception if you should continue or not. Ideally, the output of Consumer.poll and DeserializingConsumer.poll would be identical such that you don't need to modify how you interface with them EXCEPT when deserialization fails.
Does the Liskov Substitution Principle not get violated here?

I also took a look at the Protobuf Consumer example, where there are still problems:
https://github.com/confluentinc/confluent-kafka-python/blob/master/examples/protobuf_consumer.py

The error handling here is:

1. If msg is None: Continue (explicitly checked).
2. If msg contains an error: Deserializing will return None because msg.value() is None (this happens in the Deserializer class here).
3. If the msg.value() is incorrectly encoded or the Deserializer is not correctly configured to decode the message: Deserializer decoding fails, will raise a SerializationError which will crash the entire application unhandled.

There is no explicit checking for msg.error().fatal() or msg.error(). So if a non-fatal error occurs, it'll fail at the deserialization step because the msg.value() is None, but if a fatal error occurs there is seemingly no handling for that. Rather it seems like it should do error checks beforehand, as in my example above.

Is there an example of a robust deserializing consumer somewhere, so I can take inspiration from that?

How to reproduce

Just a question, see above.

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
  confluent_kafka.version()
  ('2.1.1', 33620224)
  confluent_kafka.libversion()
  ('2.1.1', 33620479)
• Apache Kafka broker version: Confluent 7.3.2
• Client configuration: N/A
• Operating system: Debian
• Provide client logs (with 'debug': '..' as necessary): N/A
• Provide broker log excerpts: N/A
• Critical issue: No