发送大量消息 Kafka Producer [英] Send bulk of messages Kafka Producer

问题描述

我正在使用 Kafka.

I'm using Kafka.

我有一个包含 10k json 的列表.

I have a list with 10k jsons.

目前我发送 Jsons 如下:

Currently I send the Jsons as follow:

for(int i=0 ;i< jsonList.size(); i++){
     ProducerRecord<K,V> record = new ProducerRecord(topic, jsonList[i]);
     producer.send(record);
}

发送每条消息.

我想将列表发送到 kafka 并让 kafka 在 json 之后发送 json(不是一个包含所有 json 字符串的消息)，例如:

I want to send the list to kafka and make kafka to send it json after json (not one message with all json strings), Something like:

ProducerRecord<K,V> record = new ProducerRecord(topic, jsonList);
producer.send(record);

我该怎么做?

谢谢

推荐答案

正式使用 KafkaProducer 和 producerRecord 你不能这样做，但你可以通过在 ProducerConfig

Officially by using KafkaProducer and producerRecord you can't do that, but you can do this by configuring some properties in ProducerConfig

batch.size 来自文档生产者的记录将记录批量发送到同一分区的请求中并立即发送

batch.size from document producer batch up the records into requests that are sending to same partition and send them at once

每当多条记录被发送到同一个分区时，生产者将尝试将记录一起批处理成更少的请求.这有助于提高客户端和服务器的性能.此配置以字节为单位控制默认批处理大小.不会尝试批量处理大于此大小的记录.

The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size.

linger.ms 此设置用于生产者的延迟时间，为生产者保留一段时间，以便在此期间的所有请求都将被批量发送，但batch.size 是上限，如果生产者获得足够的批量大小，它将忽略此属性并将批量消息发送到 kafka

linger.ms This setting is used for delay time for producer, to hold producer some time so that all request in meantime will be batched up and sent, but batch.size is upper bound on this, if producer get enough batch size it will ignore this property and send batch messages to kafka

生产者将请求传输之间到达的所有记录组合到一个批处理请求中.此设置通过添加少量人为延迟来实现此目的 - 也就是说，生产者不会立即发送记录，而是等待给定的延迟以允许发送其他记录，以便可以将发送一起批处理.此设置给出了批处理延迟的上限:一旦我们获得了分区的 batch.size 值记录，无论此设置如何，它都会立即发送.

The producer groups together any records that arrive in between request transmissions into a single batched request. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it will be sent immediately regardless of this setting.

这篇关于发送大量消息 Kafka Producer的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！