Kafka 减少消费者的延迟 [英] Kafka Reduce Lag for Consumer

问题描述

我正在设置新的 Kafka 集群，出于测试目的，我创建了具有 1 个分区和 3 个副本的主题.

I am setting up the new Kafka cluster and for testing purpose I created the topic with 1 partition and 3 replicas.

现在，我通过生产者并行地发送消息，比如每秒 50K 条消息.我在一个组内创建了一个消费者，它每秒只能获取 30K 条消息.

Now, I am firing the messages via producer in parallel say 50K messages per Second. And I have create One Consumer inside a Group and its only able to fetch 30K messages per second.

我可以更改主题级别、分区级别、消费者级别的配置.

I can change topic level, partition level, consumer level configurations.

我正在通过 grafana + prometheus 监控一切.

I am monitoring everything via grafana + prometheus.

知道哪些配置或其他东西可以帮助我消费更多数据吗??

Any Idea which configuration or something else can help me to consumer more data??

提前致谢

推荐答案

Kafka 消费者轮询代理以获取消息并获取可用于消费的任何消息，具体取决于使用的消费者配置.通常，如果您的目标是提高吞吐量，则在单个轮询请求中传输尽可能多的数据是有效的.但是在一次轮询中传输的数据量取决于消息的大小、记录的数量以及一些控制等待消息可用的时间的参数.

A Kafka consumer polls the broker for messages and fetches whatever messages are available for consumption, depending upon the consumer configuration used. In general, it is efficient to transfer as much data is possible in a single poll request if increasing throughput is your aim. But how much data is transferred in a single poll is determined by the size of messages, number of records and some parameters which control how much time to wait for messages to be available.

通常，您可以使用以下一项或多项消费者配置来影响吞吐量:

In general, you can influence throughput using one or more of the following consumer configurations:

fetch.min.bytes

max.partition.fetch.bytes

fetch.max.bytes

max.poll.records

fetch.max.wait.ms

这篇关于Kafka 减少消费者的延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！