Kafka 偏移管理 [英] Kafka offset management

问题描述

我们正在使用 Kafka 0.10...我在网上(和文档中)看到一些关于当 enable.auto.commit 为 TRUE 时如何在 kafka 中管理偏移量的相互矛盾的信息.检索消息的同一个 poll() 方法是否也按配置的时间间隔处理提交?

We are using Kafka 0.10... I'm seeing some conflicting information online (and in documentation) regarding how offsets are managed in kafka when enable.auto.commit is TRUE. Does the same poll() method that retrieves messages also handle the commits at the configured intervals?

如果我在单线程应用程序中从 poll 检索消息，则在 SAME 线程中处理消息直至完成(包括处理错误)，这意味着在我的处理完成之前不会再次调用 poll()，然后我假设那里不害怕丢失消息，对吗?这仅在 poll() 在随后的调用中尝试提交时才有效(当然，如果 auto.commit.interval.ms 已经过去).如果在收到消息后立即完成提交(在我的应用程序处理消息之前)，这对我们不起作用......

If i retrieve messages from poll in a single threaded application, process the messages to completion (including handling errors) in the SAME thread, meaning poll() will not be invoked again until after my processing is complete, then I presume there is no fear in losing messages, correct? This only works if poll() attempts the commit at the subsequent invocation (if the auto.commit.interval.ms has passed, of course). If the commits are done immediately upon receiving the messages (prior to my app processing the messages), this will not work for us....

这很重要，因为我想确保如果我们使用自动提交策略，我们不会丢失消息.重复的消息对我们来说是可以容忍的，我们只是不能容忍丢失的数据.

This is important, as I want to be certain we won't lose messages if we use the automatic commit policy. Duplicate messages are tolerable for us, we just have no tolerance for lost data.

感谢您的澄清！

推荐答案

检索消息的同一个 poll() 方法是否也按配置的时间间隔处理提交?

Does the same poll() method that retrieves messages also handle the commits at the configured intervals?

是的.(如果enable.auto.commit=true.)

如果我在单线程应用程序中从 poll 检索消息，则在 SAME 线程中处理消息直至完成(包括处理错误)，这意味着在我的处理完成之前不会再次调用 poll()，然后我假设那里不害怕丢失消息，对吗?

If i retrieve messages from poll in a single threaded application, process the messages to completion (including handling errors) in the SAME thread, meaning poll() will not be invoked again until after my processing is complete, then I presume there is no fear in losing messages, correct?

是的.

这仅在 poll() 在随后的调用中尝试提交时才有效(当然，如果 auto.commit.interval.ms 已经过去)

This only works if poll() attempts the commit at the subsequent invocation (if the auto.commit.interval.ms has passed, of course)

这正是它的做法.

请参阅此处了解更多详情:http://docs.confluent.io/current/clients/consumer.html

See here for further details: http://docs.confluent.io/current/clients/consumer.html

这篇关于Kafka 偏移管理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！