Kafka一遍又一遍地重播消息-心跳会话已过期-指示协调器已死 [英] Kafka replays messages over and over - Heartbeat session expired - marking coordinator dead

问题描述

使用python kafka api从其中仅包含少量消息的主题读取消息. Kafka不断重复播放队列中的消息.

Using python kafka api to read messages from a topic with only a handful of messages in it. Kafka keeps on replaying the messages in the queue over and over again.

它从我的主题接收到一条消息(每条消息的内容返回)，然后抛出ERROR - Heartbeat session expired - marking coordinator dead并继续循环浏览其余消息并继续重播它们.更多日志:

It receives a message from my topic (comes back with each message content), then throws ERROR - Heartbeat session expired - marking coordinator dead and keeps on looping through rest of messages and keeps on replaying them. more logs:

kafka.coordinator - ERROR - Heartbeat session expired - marking coordinator dead
kafka.coordinator - WARNING - Marking the coordinator dead (node 1) for group GROUPID1: Heartbeat session expired.
kafka.coordinator.consumer - WARNING - Auto offset commit failed for group GROUPID1: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
kafka.cluster - INFO - Group coordinator for GROUPID1 is BrokerMetadata(nodeId=1, host='HOST', port=PORT, rack=None)
kafka.coordinator - INFO - Discovered coordinator 1 for group GROUPID1
kafka.coordinator - INFO - Skipping heartbeat: no auto-assignment or waiting on rebalance
kafka.coordinator.consumer - ERROR - Offset commit failed: This is likely to cause duplicate message delivery
Traceback (most recent call last):
  File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 407, in _maybe_auto_commit_offsets_sync
    self.commit_offsets_sync(self._subscription.all_consumed_offsets())
  File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 398, in commit_offsets_sync
    raise future.exception # pylint: disable-msg=raising-bad-type
kafka.errors.CommitFailedError: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
kafka.coordinator.consumer - INFO - Revoking previously assigned partitions {TopicPartition(topic='TOPIC1', partition=0)} for group GROUPID1

推荐答案

似乎您需要调整使用者配置，最有可能是查看使用者心跳会话即将到期且无法提交上次轮询的日志会话过期导致记录失败，并触发了重新平衡，因此它将从上次未提交的记录中再次轮询

Seems like you need to tune your consumer configuration , most likely looking the logs it seems that the consumer's heartbeat session is expiring and it is unable to commit the last polled records due to expired session and it is triggering the rebalance so it will poll again from last uncommitted record

配置以进行检查-

• heartbeat.interval.ms
• session.timeout.ms
• max.poll.interval.ms

这篇关于Kafka一遍又一遍地重播消息-心跳会话已过期-指示协调器已死的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！