Kafka 批量监听器，轮询固定数量的记录(尽可能多) [英] Kafka batch listener, polling fixed numbers of records ( as much as possible)

问题描述

我使用的是 Spring Boot 版本 1.5.4.RELEASE &spring Kafka 版本 1.3.8.RELEASE.

I'm using Spring boot version 1.5.4.RELEASE & spring Kafka version 1.3.8.RELEASE.

我的 Kafka 消费者正在以 100 个块为单位进行批处理.我尝试使用的主题有 10 个分区 &我确实有 10 个 Kafka 消费者实例.

My Kafka consumer is doing batch processing in chunks of 100. Topic I'm trying to consume has 10 partitions & I do have 10 instances of Kafka consumer.

除了特定分区中的最后一个块外，我是否可以强制获得 100 个固定数量的记录(尽可能多).

推荐答案

Kafka没有属性fetch.min.records.

你能做的最好的事情就是模拟它:

The best you can do is simulate it with:

fetch.min.bytes:服务器应该为获取请求返回的最小数据量.如果没有足够的数据可用，则请求将在回答请求之前等待积累足够多的数据.1 字节的默认设置意味着只要有一个字节的数据可用或获取请求超时等待数据到达就会响应获取请求.将此值设置为大于 1 的值将导致服务器等待大量数据累积，这可以稍微提高服务器吞吐量，但会增加一些延迟.

fetch.min.bytes: The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

和

fetch.max.wait.ms:如果没有足够的数据来立即满足 fetch.min 给出的要求，服务器将在响应 fetch 请求之前阻塞的最长时间.字节.

fetch.max.wait.ms: The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

如果您的记录长度相似，这将起作用.

Which will work if your records have similar lengths.

顺便说一下，Spring Boot 1.5.x 已结束生命周期，不再受支持.当前启动版本为 2.2.3.

By the way Spring Boot 1.5.x is end of life and no longer supported. The current Boot version is 2.2.3.

这篇关于Kafka 批量监听器，轮询固定数量的记录(尽可能多)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！