在kafka中创建多少个生产者? [英] How many producers to create in kafka?

问题描述

在一个大容量实时 Java Web 应用程序中，我正在向 apache kafka 发送消息.目前我正在向单个主题发送消息，但将来我可能需要向多个主题发送消息.

In a high volume real time java web app I'm sending messages to apache kafka. Currently I'm sending to a single topic, but in the future I might need to send messages to multiple topics.

在这种情况下，我不确定是否适合为每个主题创建一个生产者，还是应该为所有主题使用一个生产者?

In this case I'm not sure weather to create a producer per topic or should I use a single producer to all my topics?

这是我的代码:

props = new Properties();
props.put("zk.connect", <zk-ip1>:<2181>,<zk-ip3>:<2181>,<zk-ip3>:<2181>);
props.put("zk.connectiontimeout.ms", "1000000");
props.put("producer.type", "async");

Producer<String, Message> producer = new kafka.javaapi.producer.Producer<String, Message>(new ProducerConfig(props));

ProducerData<String, Message> producerData1 = new ProducerData<String, Message>("someTopic1", messageTosend);
ProducerData<String, Message> producerData2 = new ProducerData<String, Message>("someTopic2", messageTosend);

producer.send(producerData1);
producer.send(producerData2);

如您所见，一旦创建了生产者，我就可以使用它向不同的主题发送数据.我想知道什么是最佳实践?如果我的应用发送到多个主题(每个主题获取不同的数据)我可以/应该使用单个生产者还是应该创建多个生产者?什么时候(一般来说)我应该使用多个生产者?

As you can see, once the producer has been created I can use it to send data to different topics. I wonder what is the best practice? If my app sends to multiple topics (each topic gets different data) can/should I use a single producer or should I create multiple producers? When (generaly speaking) should I use more than a single producer?

推荐答案

一般来说，所有主题的单一生产者会更有效地提高网络效率.

In general, a single producer for all topics will be more network efficient.

如果 kafka 客户端在同一个 Kafka 节点上看到多个主题+分区，它可以在一条消息中同时发送两个主题+分区的消息.Kafka 针对消息批次进行了优化，因此非常高效.

If the kafka client sees more than one topic+partition on the same Kafka Node, it can send messages for both topic+partitions in a single message. Kafka optimizes for message batches so this is efficient.

此外，您的 Web 服务器最多只需要与每个 Kafka 节点保持一个 tcp 连接，而不是每个生产者、每个节点一个连接.

In addition, your web servers only need to maintain at-most one tcp connection to each Kafka node, instead of one connection per producer, per node.

有关 Kafka 设计的更多信息:https://kafka.apache.org/documentation.html#design

For more info on Kafka's design: https://kafka.apache.org/documentation.html#design

正如您在评论中提到的，锁争用可能成为限制因素，YMMV.

As you mention in comments, lock contention may become a limiting factor, YMMV.

这篇关于在kafka中创建多少个生产者?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！