如何设置Kafka Producer的每秒消息速率? [英] How to set Kafka Producer message rate per second?

问题描述

我正在读取一个csv文件，并将此输入的行提供给我的Kafka Producer.现在我希望我的Kafka Producer以每秒100条消息的速度产生消息.

I am reading a csv file and giving the rows of this input to my Kafka Producer. now I want my Kafka Producer to produce messages at a rate of 100 messages per second.

推荐答案

看看Kafka Producer的 linger.ms 和 batch.size 属性.您必须相应地调整这些属性才能获得所需的速率.

Take a look at linger.ms and batch.size properties of Kafka Producer. You have to adjust these properties correspondingly to get desired rate.

生产者将在请求传输之间到达的所有记录归为一个批处理的请求.通常，只有在记录到达速度快于记录发送速度时，才在负载下发生这种情况.但是，在某些情况下，即使在中等负载下，客户端也可能希望减少请求的数量.此设置通过添加少量的人为延迟来实现此目的-也就是说，生产者将立即等待直到给定的延迟以允许其他记录被发送，以便发送者可以被批处理在一起，而不是立即发送记录.可以认为这类似于TCP中的Nagle算法.此设置提供了批处理延迟的上限:一旦获得分区的记录的batch.size值，无论此设置如何，它都会立即发送，但是，如果我们为该分区积累的字节数少于这个数量，我们将持续指定的时间，以等待更多的记录显示.此设置默认为0(即无延迟).例如，将linger.ms = 5设置会减少发送请求的数量，但会导致在没有负载的情况下发送记录的延迟最多增加5ms.

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absense of load.

这篇关于如何设置Kafka Producer的每秒消息速率?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！