KafkaConsumer恢复分区无法继续接收未提交的消息 [英] KafkaConsumer resume partition cannot continue to receive uncommitted messages
问题描述
我正在使用一个主题，一个分区，一个使用者，Kafka客户端版本为0.10.
I'm using one topic, one partition, one consumer, Kafka client version is 0.10.
我得到两个不同的结果:
I got two different results:
-
如果我已暂停分区，然后产生一条消息并调用
If I paused partition first, then to produce a message and to invoke resume method. KafkaConsumer can poll the uncommitted message successfully.
但是，如果我先生成消息但未提交其偏移量，则在几秒钟后暂停分区，以调用resume方法. KafkaConsumer将不会收到未提交的消息.我使用kafka-consumer-groups.sh在Kafka服务器上对其进行了检查，它显示了LOG-END-OFFSET minus CURRENT-OFFSET = LAG = 1.
But If I produced message first and didn't commit its offset, then to pause the partition, after several seconds, to invoke the resume method. KafkaConsumer would not receive the uncommitted message. I checked it on Kafka server using kafka-consumer-groups.sh, it shows LOG-END-OFFSET minus CURRENT-OFFSET = LAG = 1.
我已经尝试了两天，我重复了很多次这样的测试，结果总是这样.我需要一些建议，或者有人可以告诉我其Kafka的原始机制.
I have been trying to figure out it for two days, I repeated such tests a lot of times, the results are always like so. I need some suggestion or someone can tell me its Kafka's original mechanism.
推荐答案
为您的观察#2，如果重新启动应用程序，它将为您提供所有未提交偏移量的记录，即丢失的记录以及您的使用者再次不提交，它将在应用程序重新启动时向Kafka注册使用者时再次发送.是预期的.
For your observation#2, if you restart the application, it will supply you all records from the un-committed offset, i.e. the missing record and if your consumer again does not commit, it will be sent again when application registers consumer with Kafka upon restart. It is expected.
假设您正在使用consumer.poll()，它将创建一个混合流接口，即是否累积了提到的duration进入Kafka的数据，并在持续时间结束后将其提供给消费者进行处理.这种连续的累积发生在后端，而不取决于您是否已提交偏移量.
Assuming you are using consumer.poll() which creates a hybrid-streaming interface i.e. if accumulates data coming into Kafka for the duration mentioned and provides it to the consumer for processing once the duration is finished. This continuous accumulation happens in the backend and is not dependent on whether you have committed offset or not.
使用者的位置给出了下一条记录的偏移量 将被给出.它比最大偏移量大一个 消费者已经看到了该分区.它会自动前进 消费者在轮询(long)的呼叫中收到消息的时间.
The position of the consumer gives the offset of the next record that will be given out. It will be one larger than the highest offset the consumer has seen in that partition. It automatically advances every time the consumer receives messages in a call to poll(long).
这篇关于KafkaConsumer恢复分区无法继续接收未提交的消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
