如果消费者保留消息的时间长于自动提交间隔时间，kafka 会丢失消息吗? [英] Does kafka lose message if consumer holds message longer then auto commit interval time?

问题描述

假设自动提交间隔时间为 30 秒，消费者由于某些原因无法处理消息并保持超过 30 秒然后崩溃.自动提交偏移机制是否会在消费者崩溃之前提交此偏移量?

Say if auto-commit interval time is 30 seconds, consumer for some reasons could not process the message and hold it longer than 30 seconds then crash. does the auto-commit offset mechanism commits this offset anyway right before consumer crash?

如果我的假设是正确的，消息会因为提交的偏移量而丢失，但消息本身还没有被处理?

If my assumption is correct, the message is lost as its offset committed but the message itself has not been processed?

推荐答案

假设您的消费者组名称是 Test，并且您在消费者组中有一个消费者.

Lets consider your Consumer group name is Test and you have a single consumer in the Consumer Group.

启用自动提交后，仅在 poll() 调用和关闭消费者期间提交偏移量.

When Auto-Commit is enabled, offsets are committed only during poll() calls and during closing of a consumer.

例如- auto.commit.interval.ms 是 5 秒，每次调用 poll() 需要 7 秒.每次调用 poll() 时，它都会检查自动提交间隔是否已经过去，如果已经过去，就像上面的例子一样，它将提交偏移量.

For example- auto.commit.interval.ms is 5 secs, and every call to poll() takes 7 secs. When making every call to poll(), it will check if the auto commit interval has elapsed, if it has, like in the above example, it will commit the offset.

在关闭消费者期间也会提交偏移量.

Offsets are also committed during closing of a consumer.

来自文档 -

关闭消费者，等待最多 30 秒的默认超时时间以进行任何需要的清理.如果启用了自动提交，这将尽可能在默认超时内提交当前偏移量".

"Close the consumer, waiting for up to the default timeout of 30 seconds for any needed cleanup. If auto-commit is enabled, this will commit the current offsets if possible within the default timeout".

您可以在这里阅读更多相关信息 -

You can read more about it here -

https:///kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

现在，关于你的问题，如果没有再次调用 poll() 或者消费者没有关闭，它不会提交偏移量.

Now, onto your question, if poll() is not called again or consumer is not closed, it won't commit the offset.

这篇关于如果消费者保留消息的时间长于自动提交间隔时间，kafka 会丢失消息吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！