在Kafka使用者配置中将max.poll.interval.ms设置为大于request.timeout.ms有什么负面影响 [英] What is negative effects of setting max.poll.interval.ms larger than request.timeout.ms in Kafka consumer configs

问题描述

根据Kafka文档；

According to Kafka documentation;

新的Java Consumer现在支持从后台进行心跳 线.有一个新的配置max.poll.interval.ms 控制使用者前两次轮询调用之间的最长时间 将主动离开小组(默认为5分钟).的价值 配置request.timeout.ms必须始终大于 max.poll.interval.ms，因为这是JoinGroup的最长时间 使用者重新平衡时，请求可能会在服务器上阻止，因此 我们将其默认值更改为刚好超过5分钟.

The new Java Consumer now supports heartbeating from a background thread. There is a new configuration max.poll.interval.ms which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configuration request.timeout.ms must always be larger than max.poll.interval.ms because this is the maximum time that a JoinGroup request can block on the server while the consumer is rebalancing, so we have changed its default value to just above 5 minutes.

但是我无法理解使max.poll.interval.ms大于request.timeout.ms的可能结果.

But I couldn't understand the possible results of making max.poll.interval.ms larger than request.timeout.ms.

任何人都可以解释文档的这一部分吗?

Can anybody explain this part of document:

因为这是JoinGroup请求可以阻止的最长时间 消费者重新平衡服务器时

because this is the maximum time that a JoinGroup request can block on the server while the consumer is rebalancing

现在我在项目中使用这些使用者参数:

Now I'am using these consumer parameters in my project:

kafka.consumer.session.timeout.ms=30000
kafka.consumer.heartbeat.interval.ms=10000
kafka.consumer.request.timeout.ms=31000
kafka.consumer.max.poll.interval.ms=259200000
kafka.consumer.max.partition.fetch.bytes=10242880

这些用法的负面影响是什么?

What would be the negative effects of these usage?

注意:我的kafka-clients版本是1.1.0

Note: My kafka-clients version is 1.1.0

从Kafka 2.0.0版开始，此规则不再有效.来自Kafka文档2.0.0中的显着更改:

From Kafka version 2.0.0 this rule is not valid anymore. From Kafka docs Notable changes in 2.0.0:

作为KIP-266的一部分，request.timeout.ms的默认值具有 已更改为30秒.以前的值有点高 少于5分钟的时间来解决重新平衡所需的最长时间 拿. 现在，我们将重新平衡中的JoinGroup请求视为特殊 大小写，并使用从max.poll.interval.ms派生的值进行请求 .所有其他请求类型都使用由定义的超时 request.timeout.ms

Also as part of KIP-266, the default value of request.timeout.ms has been changed to 30 seconds. The previous value was a little higher than 5 minutes to account for maximum time that a rebalance would take. Now we treat the JoinGroup request in the rebalance as a special case and use a value derived from max.poll.interval.ms for the request timeout. All other request types use the timeout defined by request.timeout.ms

推荐答案

request.timeout.ms指定使用者将等待经纪人响应的时间.

request.timeout.ms specifies how long the consumer will wait for a response from the broker.

max.poll.interval.ms在几个用例中使用.它首先指定了消费者与经纪人进行互动的频率(以确认其是否还活着)，但是当消费者加入一个团体时也使用它.在这种情况下，它指示代理可以花多长时间来响应加入组请求.

max.poll.interval.ms is used in a couple of use cases. It first specifies how often the consumer was to intereact with brokers (to confirm it's alive) but it's also used when the consumer joins a group. In this case, it indicates how long the broker can take to respond to the join group request.

因此，如果max.poll.interval.ms大于request.timeout.ms，您会发现代理响应时间可能比消费者等待时间长.这样消费者可以使请求超时.

So if max.poll.interval.ms is larger than request.timeout.ms, you see that the broker may take longer to respond than the consumer will wait. So the consumer could timeout requests.

在许多情况下，这两种配置的默认值都比较好.与其将它们覆盖为我在问题中看到的较大值，不如首先应该清楚地确定要更改默认行为的方式.

In many cases, the defaults values for these 2 configurations are relatively good. Instead of overriding them to the large values I see in your question, you should first start by clearly identify how you want to change the default behaviour.

这篇关于在Kafka使用者配置中将max.poll.interval.ms设置为大于request.timeout.ms有什么负面影响的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！