Kafka 一遍又一遍地重播消息 - 心跳会话已过期 - 将协调器标记为死 [英] Kafka replays messages over and over - Heartbeat session expired - marking coordinator dead

问题描述

使用 python kafka api 从一个主题中读取消息，其中只有少数消息.Kafka 不断地一遍又一遍地重放队列中的消息.

Using python kafka api to read messages from a topic with only a handful of messages in it. Kafka keeps on replaying the messages in the queue over and over again.

它从我的主题收到一条消息(每个消息内容都回来)，然后抛出 ERROR - Heartbeat session expired - 标记协调器死了 并继续遍历其余消息并继续重播它们.更多日志:

It receives a message from my topic (comes back with each message content), then throws ERROR - Heartbeat session expired - marking coordinator dead and keeps on looping through rest of messages and keeps on replaying them. more logs:

kafka.coordinator - ERROR - Heartbeat session expired - marking coordinator dead
kafka.coordinator - WARNING - Marking the coordinator dead (node 1) for group GROUPID1: Heartbeat session expired.
kafka.coordinator.consumer - WARNING - Auto offset commit failed for group GROUPID1: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
kafka.cluster - INFO - Group coordinator for GROUPID1 is BrokerMetadata(nodeId=1, host='HOST', port=PORT, rack=None)
kafka.coordinator - INFO - Discovered coordinator 1 for group GROUPID1
kafka.coordinator - INFO - Skipping heartbeat: no auto-assignment or waiting on rebalance
kafka.coordinator.consumer - ERROR - Offset commit failed: This is likely to cause duplicate message delivery
Traceback (most recent call last):
  File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 407, in _maybe_auto_commit_offsets_sync
    self.commit_offsets_sync(self._subscription.all_consumed_offsets())
  File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 398, in commit_offsets_sync
    raise future.exception # pylint: disable-msg=raising-bad-type
kafka.errors.CommitFailedError: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
kafka.coordinator.consumer - INFO - Revoking previously assigned partitions {TopicPartition(topic='TOPIC1', partition=0)} for group GROUPID1

推荐答案

看来你需要调整你的消费者配置，最有可能的是查看日志，似乎消费者的心跳会话即将到期并且无法提交最后一次轮询由于会话过期而产生的记录并且它正在触发重新平衡，因此它将从最后一个未提交的记录再次轮询

Seems like you need to tune your consumer configuration , most likely looking the logs it seems that the consumer's heartbeat session is expiring and it is unable to commit the last polled records due to expired session and it is triggering the rebalance so it will poll again from last uncommitted record

要检查的配置 -

• heartbeat.interval.ms
• session.timeout.ms
• max.poll.interval.ms

这篇关于Kafka 一遍又一遍地重播消息 - 心跳会话已过期 - 将协调器标记为死的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！