发送大量消息Kafka Producer [英] Send bulk of messages Kafka Producer

问题描述

我正在使用Kafka.

I'm using Kafka.

我有1万个json列表，

I have list with 10k jsons,

当前，我将按以下方式发送Jsons:

Currently i'm send the Jsons as follow:

for(int i=0 ;i< jsonList.size(); i++){
     ProducerRecord<K,V> record = new ProducerRecord(topic, jsonList[i]);
     producer.send(record);
}

发送每封邮件.

我想将列表发送到kafka，并使kafka在json之后发送json(没有一条消息包含所有json字符串)，类似:

I want to send the list to kafka and make kafka to send it json after json (not one message with all json strings), Something like:

ProducerRecord<K,V> record = new ProducerRecord(topic, jsonList);
producer.send(record);

我该怎么办?

谢谢

推荐答案

通过使用 KafkaProducer 和 producerRecord ，您不能这样做，但是您可以通过以下方法来做到这一点在 ProducerConfig

Officially by using KafkaProducer and producerRecord you can't do that, but you can do this by configuring some properties in ProducerConfig

batch.size 将记录分批处理成发送到同一分区的请求，并立即发送

batch.size from document producer batch up the records into requests that are sending to same partition and send them at once

每当将多个记录发送到同一分区时，生产者将尝试将记录一起批处理成更少的请求.这有助于提高客户端和服务器的性能.此配置控制默认的批处理大小(以字节为单位).不会尝试批处理大于此大小的记录.

The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size.

linger.ms 此设置用于延迟生产者的时间，以保留生产者一些时间，以便同时打包并发送所有请求，但 batch.size 是上限，如果生产者获得足够的批处理大小，它将忽略此属性并将批处理消息发送到kafka

linger.ms This setting is used for delay time for producer, to hold producer some time so that all request in meantime will be batched up and sent, but batch.size is upper bound on this, if producer get enough batch size it will ignore this property and send batch messages to kafka

生产者将在请求传输之间到达的所有记录归为一个批处理的请求.此设置通过添加少量的人为延迟来实现此目的-也就是说，生产者将立即等待直到给定的延迟以允许其他记录被发送，以便发送者可以被批处理在一起，而不是立即发送记录.此设置提供了批处理延迟的上限:一旦获得分区的记录的batch.size值，无论此设置如何，它都会立即发送.

The producer groups together any records that arrive in between request transmissions into a single batched request. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it will be sent immediately regardless of this setting.

这篇关于发送大量消息Kafka Producer的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！