max.poll.records结合fetch.min.bytes [英] max.poll.records in conjunction with fetch.min.bytes

问题描述

我正在阅读 max.poll.records如何影响消费者调查以及apache kafka文档，但我仍然不确定 fetch.min.bytes 是否未更改，默认值为1，是否需要kafka经纪人返回记录的 max.poll.records (如果有那么多的话)?

I am reading this How does max.poll.records affect the consumer poll, as well as apache kafka docs, and I am still not sure if fetch.min.bytes is not changed, and the default is 1, is kafka broker obligated to return max.poll.records of records, if that much is available, or not?

根据我们的测试，即使某个主题中有足够的数据，它也不总是返回那么多，并且从文档及其绝对名称中解释该参数并不意味着应该如此，但是有些人倾向于认为相反.我们还增加了可能阻止这种情况发生的限制，例如 message.max.bytes ， max.message.bytes ， max.partition.fetch.bytes和 fetch.max.bytes (实际上我们不必增加，因为默认值很高，为50 MB)，但这并没有改变.

According to our tests, it does not always return that much even if there is sufficient data available in a topic, and the explanation of that parameter from documentation and its sheer name does not imply it should, but some people tend to think the opposite. We also increased the limits that could potentially prevent this from happen, like message.max.bytes, max.message.bytes, max.partition.fetch.bytes, and fetch.max.bytes (that one we actually didn't have to increase, since the default is rather high, 50 MB), but that didn't change a thing.

我们也没有更改 fetch.max.wait.ms ，默认值为500，即半秒，因此，如果 fetch.min.bytes 设置为不超过1个字节，则此设置生效，即，它确定实际返回多少记录?这是否意味着如果返回的 max.poll.records 少，是因为要花500毫秒以上的时间才能提取这么多的东西?

We also didn't change fetch.max.wait.ms, and default is 500, that is a half of a second, so, if fetch.min.bytes is not set to something more than 1 byte, then this setting becomes effective, ie, it determines how much records is actually returned? Which would mean that if less then max.poll.records was returned, it is because it would take more than 500 ms to fetch that much?

推荐答案

这2种配置可能会令人困惑，乍一看它们看上去很相似，但是它们的工作方式却大不相同.

These 2 configurations can be confusing and while at first sight they look similar they work in very different ways.

• fetch.min.bytes :此值是Fetch Requests字段之一(min_bytes .apache.org/protocol#The_Messages_Fetch"rel =" noreferrer> http://kafka.apache.org/protocol#The_Messages_Fetch ).代理使用此值来决定何时将获取响应"发送回客户端.如果没有可供使用的 fetch.min.bytes 个字节，则当代理接收到Fetch请求时，它最多可以保留 fetch.max.wait.ms 个字节.例如，使用者位于日志末尾，或者要使用的消息增加到小于该大小).

• fetch.min.bytes: This value is one of the fields of Fetch Requests (it's min_bytes in http://kafka.apache.org/protocol#The_Messages_Fetch). This value is used by the broker to decide when to send a Fetch Response back to the client. When a broker receives a Fetch Request it can hold it for up to fetch.max.wait.ms if there are not fetch.min.bytes bytes available for consumption (for example the consumer is at the end of the log or the messages to be consumed add to less than that size).

max.poll.records :此设置仅在使用者中使用，并且永远不会发送给代理.在后台(异步)，消费者客户端主动从代理中获取记录并对其进行缓冲，因此在调用 poll()时，它可以返回已获取的记录.顾名思义，此设置控制最多可以从使用者缓冲区返回多少 poll()条记录.

max.poll.records: This setting is only used within the Consumer and is never sent to brokers. In the background (asynchronously), the consumer client actively fetches records from the broker and buffers them so when poll() is called, it can return records already fetched. As the name suggest, this settings control how many records at most poll() can return from the consumer buffer.

这篇关于max.poll.records结合fetch.min.bytes的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！