需要澄清有关Kafka自动提交和auto.commit.interval.ms的信息 [英] Need clarification about Kafka auto commit and auto.commit.interval.ms

问题描述

文档 https://www.safaribooksonline. com/library/view/kafka-the-definitive/9781491936153/ch04.html 表示请注意，启用自动提交功能后，对民意调查的调用将始终提交上次民意调查返回的最后一个偏移量它不知道实际处理了哪些事件，因此在再次调用poll之前(或调用close()之前，它也会自动提交偏移量)始终处理poll返回的所有事件至关重要." 如果是这样的话，如果auto.commit.interval.ms大于处理从先前poll()接收到的消息所花费的时间，它将如何工作.

The document https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html says that "Note that with auto-commit enabled, a call to poll will always commit the last offset returned by the previous poll. It doesn’t know which events were actually processed, so it is critical to always process all the events returned by poll before calling poll again (or before calling close(), it will also automatically commit offsets)". If that's the case how does it work if auto.commit.interval.ms is larger than the time if takes to process the messages received from previous poll().

要使其更具体，请考虑以下情况:

To make it more concrete, consider the scenario where I have following:

enable.auto.commit=true

auto.commit.interval.ms=10

然后我循环调用poll().

1)第一次调用poll()时，我收到1000条消息(偏移量为2000-3000)，处理所有1000条消息需要1毫秒

1) On first call to poll(), I get 1000 messages (offset 2000-3000) and it takes 1 ms to process all 1000 messages

2)我再次致电poll().在第二个poll()调用中，它应提交从先前poll()返回的最新偏移量3000，但由于auto.commit.interval.ms设置为10 ms，它还不会提交偏移量，对吗?

2) I call poll() again. In this 2nd poll() call, it should commit the latest offset 3000 returned from previous poll() but since auto.commit.interval.ms is set to 10 ms, it won't commit the offset yet, right?

在这种情况下，提交的偏移量将越来越滞后于实际处理的最新偏移量?

In this scenario, the committed offset will lag further and further behind the latest offset that was actually processed?

有人可以澄清/确认吗?

Could someone clarify/confirm?

推荐答案

您正确地描述了行为.但是，您的结论是不正确的.承诺的偏移量不会越来越滞后.经过自动提交间隔后，下一次轮询将提交所有处理过的消息.

You describe the behavior correctly. However, you conclusion is not correct. The committed offset will not lag further and further. After auto-commit interval passed, the next call to poll will commit all processed messages.

比方说，您每10毫秒调用一次poll，然后将commit-interval设置为100ms.因此，每10次轮询调用都会提交一次(该提交涵盖了最近10次轮询调用中的所有消息).

Let's say, you call poll each 10 ms, and set commit-interval to 100ms. Thus, in every 10th call to poll will commit (and this commit covers all messages from the last 10 poll calls).

这篇关于需要澄清有关Kafka自动提交和auto.commit.interval.ms的信息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！