在 Kafka 消费者配置中将 max.poll.interval.ms 设置为大于 request.timeout.ms 的负面影响是什么 [英] What is negative effects of setting max.poll.interval.ms larger than request.timeout.ms in Kafka consumer configs

问题描述

根据 Kafka 文档；

According to Kafka documentation;

新的 Java Consumer 现在支持来自后台的心跳线.有一个新的配置 max.poll.interval.ms控制消费者之前轮询调用之间的最长时间将主动离开组(默认为 5 分钟).的价值配置 request.timeout.ms 必须始终大于max.poll.interval.ms 因为这是 JoinGroup 的最长时间当消费者重新平衡时，请求可以在服务器上阻塞，所以我们已将其默认值更改为略高于 5 分钟.

The new Java Consumer now supports heartbeating from a background thread. There is a new configuration max.poll.interval.ms which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configuration request.timeout.ms must always be larger than max.poll.interval.ms because this is the maximum time that a JoinGroup request can block on the server while the consumer is rebalancing, so we have changed its default value to just above 5 minutes.

但我无法理解使 max.poll.interval.ms 大于 request.timeout.ms 的可能结果.

But I couldn't understand the possible results of making max.poll.interval.ms larger than request.timeout.ms.

谁能解释一下这部分文档:

Can anybody explain this part of document:

因为这是 JoinGroup 请求可以阻塞的最长时间消费者重新平衡时的服务器

because this is the maximum time that a JoinGroup request can block on the server while the consumer is rebalancing

现在我在我的项目中使用这些消费者参数:

Now I'am using these consumer parameters in my project:

kafka.consumer.session.timeout.ms=30000
kafka.consumer.heartbeat.interval.ms=10000
kafka.consumer.request.timeout.ms=31000
kafka.consumer.max.poll.interval.ms=259200000
kafka.consumer.max.partition.fetch.bytes=10242880

这些使用会有什么负面影响?

What would be the negative effects of these usage?

注意:我的 kafka-clients 版本是 1.1.0

Note: My kafka-clients version is 1.1.0

从 Kafka 2.0.0 版开始，此规则不再有效.来自 Kafka 文档 2.0.0 中的显着变化:

From Kafka version 2.0.0 this rule is not valid anymore. From Kafka docs Notable changes in 2.0.0:

同样作为 KIP-266 的一部分，request.timeout.ms 的默认值有已更改为 30 秒.之前的值有点高超过 5 分钟来考虑重新平衡的最长时间拿.现在我们将重新平衡中的 JoinGroup 请求视为一个特殊的case 并使用从 max.poll.interval.ms 派生的值作为请求超时.所有其他请求类型使用由定义的超时request.timeout.ms

Also as part of KIP-266, the default value of request.timeout.ms has been changed to 30 seconds. The previous value was a little higher than 5 minutes to account for maximum time that a rebalance would take. Now we treat the JoinGroup request in the rebalance as a special case and use a value derived from max.poll.interval.ms for the request timeout. All other request types use the timeout defined by request.timeout.ms

推荐答案

request.timeout.ms 指定消费者等待代理响应的时间.

request.timeout.ms specifies how long the consumer will wait for a response from the broker.

max.poll.interval.ms 用于几个用例.它首先指定消费者与代理交互的频率(以确认它是活着的)，但在消费者加入组时也会使用它.在这种情况下，它表示代理可以花费多长时间来响应加入组请求.

max.poll.interval.ms is used in a couple of use cases. It first specifies how often the consumer was to intereact with brokers (to confirm it's alive) but it's also used when the consumer joins a group. In this case, it indicates how long the broker can take to respond to the join group request.

因此，如果 max.poll.interval.ms 大于 request.timeout.ms，您会看到代理可能需要比消费者等待更长的时间来响应.因此消费者可以超时请求.

So if max.poll.interval.ms is larger than request.timeout.ms, you see that the broker may take longer to respond than the consumer will wait. So the consumer could timeout requests.

在很多情况下，这两种配置的默认值都比较好.与其将它们覆盖为我在您的问题中看到的大值，您应该首先清楚地确定您想要如何更改默认行为.

In many cases, the defaults values for these 2 configurations are relatively good. Instead of overriding them to the large values I see in your question, you should first start by clearly identify how you want to change the default behaviour.

这篇关于在 Kafka 消费者配置中将 max.poll.interval.ms 设置为大于 request.timeout.ms 的负面影响是什么的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！