Kafka 消费者(组)触发再平衡的条件 [英] Conditions in which Kafka Consumer (Group) triggers a rebalance

问题描述

我正在浏览 Kafka 的消费者配置.

I was going through the Consumer Config for Kafka.

• https://kafka.apache.org/documentation/#newconsumerconfigs

触发重新平衡的参数是什么?.例如，以下参数将 ?.我们需要更改或默认的任何其他参数就足够了

what are the parameters that will trigger a rebalance ?. For instance the following parameter will ?. Any other parameters which we need to change or will defaults suffice

connections.max.idle.ms 在此配置指定的毫秒数后关闭空闲连接.长540000中

connections.max.idle.ms Close idle connections after the number of milliseconds specified by this config. long 540000 medium

我们还有三个不同的主题

Also we have three different topics

1. 让同一个消费者组(相同 ID)从多个主题中消费是不是一个坏主意.

2. 假设上述场景是否有效(不一定是最佳实践) - 如果主题之一的流量非常小，是否会导致消费者组重新平衡.

1. Is it a bad idea to have the same Consumer Group (Same ID) consuming from multiple topics.

2. Assuming if the above scenario is valid (not necessarily the best practice) - if one of the topic is very light traffic, will it cause the Consumer group to rebalance.

一个后续问题 - 哪些因素会影响重新平衡及其性能.

A follow up question - what factors affect the rebalancing and its performance.

推荐答案

这些条件将触发组重新平衡:

任何订阅的主题列表的分区数都发生了变化

Number of partitions change for any of the subscribed list of topics

创建或删除主题

消费者组的现有成员死亡

An existing member of the consumer group dies

通过加入 API 将新成员添加到现有消费者组

A new member is added to an existing consumer group via the join API

让同一个消费者组(相同 ID)从多个主题中消费是不是一个坏主意.

至少它是有效的，至于好坏，这取决于您的具体情况.这个是官方java客户端api支持的，看这个方法定义:

At least it is valid, as for good or bad, it depends on your detailed case. This is supported by the official java client api, see this method definition:

public void subscribe(Collection<String> topics,
             ConsumerRebalanceListener listener)

它接受一组主题.

如果其中一个主题的流量非常小，是否会导致消费者组重新平衡.

否，因为这未在条件中列出.如果只从话题方面考虑的话.只有当主题被删除或分区数量改变时，才会发生重新平衡.

No, because this is not listed in conditions. If we just consider it from the topic aspect. only when the topic is deleted or partition counts changed, the rebalcance will happens,.

更新.

感谢@Hans Jespersen 对会话和心跳的评论.

Thanks for @Hans Jespersen's comment about session and hearbeat.

这是由 kafka Consumer javadoc 引用的:

this is quoted by kafka Consumer javadoc:

订阅一组主题后，消费者将在调用poll(long) 时自动加入该组.poll API 旨在确保消费者的活跃度.只要您继续调用 poll，消费者就会留在组中并继续从分配给它的分区接收消息.在幕后，poll API 会定期向服务器发送心跳；当您停止调用 poll(可能是因为抛出了异常)时，将不会发送心跳.如果在服务器收到心跳之前 配置的会话超时时间过去了，那么消费者将被踢出组并重新分配其分区.

After subscribing to a set of topics, the consumer will automatically join the group when poll(long) is invoked. The poll API is designed to ensure consumer liveness. As long as you continue to call poll, the consumer will stay in the group and continue to receive messages from the partitions it was assigned. Underneath the covers, the poll API sends periodic heartbeats to the server; when you stop calling poll (perhaps because an exception was thrown), then no heartbeats will be sent. If a period of the configured session timeout elapses before the server has received a heartbeat, then the consumer will be kicked out of the group and its partitions will be reassigned.

在您的问题中，您会问触发重新平衡的参数是什么

在这种情况下，有两个配置与重新平衡有关.它是 session.timeout.ms 和 ma​​x.poll.records.其手段显而易见.

In this case, there are two configs has relation with the rebalance. It is session.timeout.ms and max.poll.records. Its means is obvious.

由此，我们还可以了解到在轮询之间做大量工作是一种不好的做法，开销工作可能会阻止心跳并导致会话超时.

And from this, We also could learn that it is a bad practice to do a lot work between the poll, overhead work maybe block the heartbeat and cause session timeout.

这篇关于Kafka 消费者(组)触发再平衡的条件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！