Kafka Stream:消费者提交频率 [英] Kafka Stream: Consumer commit frequency

问题描述

至少一次保证，我知道在失败的情况下有可能重复.然而，
1) Kafka Stream 库执行提交的频率如何?
2) 除了上述之外，用户是否还需要考虑提交?
3)是否有关于提交频率的最佳实践?

With atleast-once guarantee, I understand that there is a possibility of duplicates in case of failures. However,
1) How frequent does the Kafka Stream library performs commits?
2) Does the users ever need consider committing in addition to the above?
3) Is there a best practice on how frequent the commit should be performed?

推荐答案

Kafka Streams 定期提交，可以通过参数 commit.interval.ms 进行配置(默认为 30 秒；如果恰好-启用处理后，默认为 100 毫秒).

Kafka Streams commits in regular intervals that can be configured via parameter commit.interval.ms (default is 30 seconds; if exactly-once processing is enabled, default is 100ms).

通常，用户不需要手动提交.注意思想，用户不能完全控制提交，但只能请求提交:cf.如何使用 Kafka Stream 手动提交?

Usually, it's not necessary for users to commit manually. Note thought, that users don't have complete control over committing, but can only request commits: cf. How to commit manually with Kafka Stream?

提交是同步点，如果您提交太频繁(例如在每个处理过的记录之后的极端示例)，您的吞吐量可能会显着下降.它也高度依赖于应用程序，因为提交频率决定了应用程序进程有多少潜在的重复(这也取决于输入数据速率).因此，您需要考虑在失败的情况下您愿意容忍多少重复.它还取决于应用程序重新处理数据所需的时间:在此期间，应用程序可能无法完全可用.总的来说，很难给出建议，您需要单独考虑每个应用程序的所述权衡.

Commits are synchronizations point and if you commit too frequently (for an extreme example after every processed record) your throughput can drop significantly. It also highly application dependent, because the commit frequency determines how many potential duplicates the application processes (this also depends on the input data rate). Thus, you need to consider how many duplicates in case of failure your are willing to tolerate. It also depends how long it will take for the application to reprocess the data: during this time the application might not be fully available. Overall, it's hard to give a recommendation and you need to consider the described trade-offs for each application individually.

这篇关于Kafka Stream:消费者提交频率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！