Kafka Consumer(Group)触发重新平衡的条件 [英] Conditions in which Kafka Consumer (Group) triggers a rebalance

问题描述

我正在查看Kafka的Consumer Config.

I was going through the Consumer Config for Kafka.

• https://kafka.apache.org/documentation/#newconsumerconfigs

将触发重新平衡的参数是什么?例如，以下参数将为?.我们需要更改或将其设为默认值的任何其他参数

what are the parameters that will trigger a rebalance ?. For instance the following parameter will ?. Any other parameters which we need to change or will defaults suffice

connections.max.idle.ms在此配置指定的毫秒数后关闭空闲连接.长540000中

connections.max.idle.ms Close idle connections after the number of milliseconds specified by this config. long 540000 medium

我们还有三个不同的主题

Also we have three different topics

1. 拥有来自多个主题的同一个消费者组(相同ID)是一个坏主意.

2. 假设上述情况是否有效(不一定是最佳实践)-如果主题之一是流量非常小，将导致消费者组重新平衡.

1. Is it a bad idea to have the same Consumer Group (Same ID) consuming from multiple topics.

2. Assuming if the above scenario is valid (not necessarily the best practice) - if one of the topic is very light traffic, will it cause the Consumer group to rebalance.

一个后续问题-哪些因素会影响再平衡及其性能.

A follow up question - what factors affect the rebalancing and its performance.

推荐答案

这些情况将触发组重新平衡:

任何已订阅主题列表的分区数更改

Number of partitions change for any of the subscribed list of topics

主题已创建或删除

该消费者组的现有成员死亡

An existing member of the consumer group dies

通过加入API将新成员添加到现有的消费者组

A new member is added to an existing consumer group via the join API

让来自多个主题的同一个消费者组(相同ID)消费是一个坏主意.

至少它是有效的，无论好坏，都取决于您的详细情况. 官方的Java客户端api支持此功能，请参见以下方法定义:

At least it is valid, as for good or bad, it depends on your detailed case. This is supported by the official java client api, see this method definition:

public void subscribe(Collection<String> topics,
             ConsumerRebalanceListener listener)

它接受一系列主题.

如果其中一个主题的访问量很小，是否会导致Consumer组重新平衡.

否，因为未在条件中列出.如果仅从主题方面考虑.仅当删除主题或更改分区计数时，才会发生重新平衡.

No, because this is not listed in conditions. If we just consider it from the topic aspect. only when the topic is deleted or partition counts changed, the rebalcance will happens,.

更新.

感谢@Hans Jespersen对会话和心跳的评论.

Thanks for @Hans Jespersen's comment about session and hearbeat.

这是由kafka Consumer Javadoc引用的:

this is quoted by kafka Consumer javadoc:

订阅了一组主题后，当调用民意测验(长)时，消费者将自动加入该组.轮询API旨在确保消费者的活力.只要您继续呼叫民意调查，使用者就将留在组中并继续从分配给它的分区中接收消息. 在后台，轮询API将定期的心跳发送到服务器；当您停止致电民意调查(可能是因为引发异常)时，将不会发送任何心跳信号.如果在服务器收到心跳之前经过了配置的会话超时的时间段，则使用方将被踢出该组，并重新分配其分区.

After subscribing to a set of topics, the consumer will automatically join the group when poll(long) is invoked. The poll API is designed to ensure consumer liveness. As long as you continue to call poll, the consumer will stay in the group and continue to receive messages from the partitions it was assigned. Underneath the covers, the poll API sends periodic heartbeats to the server; when you stop calling poll (perhaps because an exception was thrown), then no heartbeats will be sent. If a period of the configured session timeout elapses before the server has received a heartbeat, then the consumer will be kicked out of the group and its partitions will be reassigned.

然后在您的问题中，您问哪些参数会触发重新平衡

And In your question, you ask what are the parameters that will trigger a rebalance

在这种情况下，有两个与重新平衡有关的配置.这是 session.timeout.ms 和 max.poll.records .它的手段很明显.

In this case, there are two configs has relation with the rebalance. It is session.timeout.ms and max.poll.records. Its means is obvious.

由此，我们还可以了解到，在轮询之间做很多工作是不明智的做法，开销的工作可能会阻塞心跳并导致会话超时.

And from this, We also could learn that it is a bad practice to do a lot work between the poll, overhead work maybe block the heartbeat and cause session timeout.

这篇关于Kafka Consumer(Group)触发重新平衡的条件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！