Kafka - 如何在使用高级消费者的每条消息后提交偏移量? [英] Kafka - How to commit offset after every message using High-Level consumer?
问题描述
我正在使用 Kafka 的高级消费者.因为我将 Kafka 用作我的应用程序的事务队列"，所以我需要绝对确保我不会错过或重新阅读任何消息.我有两个关于此的问题:
I'm using Kafka's high-level consumer. Because I'm using Kafka as a 'queue of transactions' for my application, I need to make absolutely sure I don't miss or re-read any messages. I have 2 questions regarding this:
如何向zookeeper提交偏移量?每条消息成功消费后，我将关闭自动提交和提交偏移量.我似乎无法找到有关如何使用高级使用者执行此操作的实际代码示例.有人能帮我解决这个问题吗?
How do I commit the offset to zookeeper? I will turn off auto-commit and commit offset after every message successfully consumed. I can't seem to find actual code examples of how to do this using high-level consumer. Can anyone help me with this?
另一方面，我听说提交到 zookeeper 可能很慢，所以另一种方法可能是在本地跟踪偏移量?这种替代方法是否可取?如果是，您将如何处理?
On the other hand, I've heard committing to zookeeper might be slow, so another way may be to locally keep track of the offsets? Is this alternative method advisable? If yes, how would you approach it?
推荐答案
有两个相关设置来自 http://kafka.apache.org/documentation.html#consumerconfigs.
There are two relevant settings from http://kafka.apache.org/documentation.html#consumerconfigs.
auto.commit.enable
和
auto.commit.interval.ms
如果你想设置它使得消费者在每条消息之后提交偏移量，这将是困难的，因为唯一的设置是在一个计时器间隔之后，而不是每条消息之后.您必须对传入消息进行一些速率预测并相应地设置时间.
If you want to set it such that the consumer commits the offset after each message, that will be difficult since the only setting is after a timer interval, not after each message. You will have to do some rate prediction of the incoming messages and accordingly set the time.
一般来说，不建议将此间隔保持得太小，因为它会大大增加 zookeeper 的读/写速率，而 zookeeper 会变慢，因为它在其仲裁中具有很强的一致性.
In general, it is not recommended to keep this interval too small because it vastly increases the read/write rates in zookeeper and zookeeper gets slowed down because it's strongly consistent across its quorum.
这篇关于Kafka - 如何在使用高级消费者的每条消息后提交偏移量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
