卡夫卡消费群再平衡 [英] Kafka Consumer group rebalancing

问题描述

我正在使用kafka消费者组管理来处理我的消息.

I'm using kafka consumer group management for processing my messages.

我的邮件的处理时间互不相同.因此，我将最大记录间隔为20条，将最大轮询间隔设置为20分钟.我正在使用5个分区和5个使用者实例，这些实例的默认配置值不同于上述两个.

The processing time for my messages vary from one another. So I have set the max poll interval to 20 min for max records of 20. And I'm using 5 partition and 5 consumer instances with default config values apart from the above two.

但是我仍然间歇性地收到以下错误:

But still I'm getting the following error intermittently:

[Consumer clientId=consumer-3, groupId=amc_dashboard_analytics] Attempt to heartbeat failed since group is rebalancing

了解是，除非达到消费者配置文档中所写的最大轮询间隔之前未调用轮询，否则不会发生重新平衡.但是对我来说，重新平衡仅发生在20分钟之前.

The understanding is that rebalancing won't occur unless poll is not called before max poll interval is reached as written in consumer config Docs. But for me rebalancing occurs before 20 minutes only.

在运行了几个小时之后，所有分配的消费者都只是说由于组正在重新平衡而导致心跳失败"，并且不再再次加入(理想情况下应该再次加入).

Also after few hours of running, all the assigned consumers just leave saying "Attempt to heartbeat failed since group is rebalancing" and do not join back again(Ideally should join back again).

我在这里错过了什么吗?任何线索都将有所帮助.

Am I missing something here? Any leads would be helpful.

推荐答案

再平衡的另一个原因是session.timeout.ms到期而没有发送心跳.您可以考虑增加此使用者配置.

Another reason of rebalance is expiring session.timeout.ms without sending heartbeat. You can consider to increase this consumer config.

来自Kafka文档:

heartbeat.interval.ms :两次心跳之间的预期时间 使用Kafka的群组管理工具时的消费者协调员. 心跳用于确保使用者的会话保持活动状态 并在新消费者加入或离开时促进平衡 团体.该值必须设置为低于session.timeout.ms，但是 通常应将其设置为不大于该值的1/3.有可能 调整甚至更低以控制正常的预期时间 重新平衡. (默认值:3000)

heartbeat.interval.ms: The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances. (default: 3000)

---

session.timeout.ms :该超时值用于在以下情况下检测客户端故障 使用Kafka的群组管理工具.客户端定期发送 向经纪人表明其活力的心跳.如果没有心跳 在此会话期满之前由经纪人收到 超时，则代理会将该客户端从组中删除，然后 开始重新平衡.请注意，该值必须在允许范围内 代理配置中配置的范围 group.min.session.timeout.ms和group.max.session.timeout.ms. (默认值:10000)

session.timeout.ms: The timeout used to detect client failures when using Kafka's group management facility. The client sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this client from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms. (default: 10000)

您可以检查链接以获取更多信息.

You can check this link for more information.

即使通过单独的线程以固定的时间间隔发送心跳，在某些情况下，心跳也无法在session.timeout.ms中发送给代理.这种情况的一些可能原因是:

Even if heartbeat is sent in fixed time intervals via separate thread, in some cases heartbeat cannot be sent to broker in session.timeout.ms. Some of the possible reasons of this situation is:

• 网络问题
• 在消费者或经纪人方面停止世界垃圾收集

这篇关于卡夫卡消费群再平衡的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！