Kafka-如何使用高级使用者在每条消息后提交偏移量? [英] Kafka - How to commit offset after every message using High-Level consumer?

问题描述

我正在使用Kafka的高级消费者.因为我将Kafka用作应用程序的交易队列"，所以我需要确保绝对不会错过或重读任何消息.我对此有2个问题:

I'm using Kafka's high-level consumer. Because I'm using Kafka as a 'queue of transactions' for my application, I need to make absolutely sure I don't miss or re-read any messages. I have 2 questions regarding this:

1. 我如何将偏移量提交给zookeeper?每条消息成功使用后，我将关闭自动提交和提交偏移.我似乎找不到使用高级使用者如何执行此操作的实际代码示例.有人可以帮我吗?

1. How do I commit the offset to zookeeper? I will turn off auto-commit and commit offset after every message successfully consumed. I can't seem to find actual code examples of how to do this using high-level consumer. Can anyone help me with this?

另一方面，我听说提交给zookeeper可能很慢，所以另一种方法可能是本地跟踪偏移量?这种替代方法是否明智?如果是，您将如何处理?

On the other hand, I've heard committing to zookeeper might be slow, so another way may be to locally keep track of the offsets? Is this alternative method advisable? If yes, how would you approach it?

推荐答案

http://kafka中有两个相关设置.apache.org/documentation.html#consumerconfigs .

auto.commit.enable

和

auto.commit.interval.ms

如果要进行设置，以使使用者在每条消息后提交偏移量，这将很困难，因为唯一的设置是在计时器间隔之后，而不是在每条消息之后.您将需要对传入消息进行一些速率预测，并相应地设置时间.

If you want to set it such that the consumer commits the offset after each message, that will be difficult since the only setting is after a timer interval, not after each message. You will have to do some rate prediction of the incoming messages and accordingly set the time.

通常，不建议将此间隔设置得太小，因为它会极大地增加Zookeeper的读/写速率，并且由于其在整个仲裁群体中的一致性极强，因此Zookeeper的运行速度会变慢.

In general, it is not recommended to keep this interval too small because it vastly increases the read/write rates in zookeeper and zookeeper gets slowed down because it's strongly consistent across its quorum.

这篇关于Kafka-如何使用高级使用者在每条消息后提交偏移量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！