ProducerStream仅产生单个分区 [英] ProducerStream producing only to single partition
本文介绍了ProducerStream仅产生单个分区的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试向具有2个分区的单个主题生成一些消息.所有消息都将仅分配到2号分区. 我希望生产者流可以将消息分布在所有分区上.
I am trying to produce some messages to a single topic having 2 partitions. All the messages are going to partition number 2 only. I would expect that a producer stream would distribute the messages across all partitions.
const kafka = require('kafka-node')
const { Transform } = require('stream');
const _ = require('lodash');
const client = new kafka.KafkaClient({ kafkaHost: 'localhost:9092' })
, streamproducer = new kafka.ProducerStream({kafkaClient: client});
const stdinTransform = new Transform({
objectMode: true,
decodeStrings: true,
transform (text, encoding, callback) {
let num = parseInt(text);
let message = { num: num, method: 'two' }
console.log('pushing message')
callback(null, {
topic: 'topic356',
messages: JSON.stringify(message)
});
}
});
stdinTransform.pipe(streamproducer);
function send() {
var message = new Date().toString();
stdinTransform.write([{ messages: [message] }]);
}
setInterval(send, 100);
ConsumerGroup:
ConsumerGroup:
var consumerOptions = {
kafkaHost: '127.0.0.1:9092',
groupId: 'ExampleTestGroup',
sessionTimeout: 15000,
protocol: ['roundrobin'],
fromOffset: 'latest' // equivalent of auto.offset.reset valid values are 'none', 'latest', 'earliest'
};
var topics = 'topic356';
var consumerGroup = new ConsumerGroup(Object.assign({ id: 'consumer1' }, consumerOptions), topics);
consumerGroup.on('data', onMessage);
var consumerGroup2 = new ConsumerGroup(Object.assign({ id: 'consumer2' }, consumerOptions), topics);
consumerGroup2.on('data', onMessage);
consumerGroup2.on('connect', function () {
setTimeout(function () {
consumerGroup2.close(true, function (error) {
console.log('consumer2 closed', error);
});
}, 25000);
});
function onMessage (message) {
console.log(
` partition: ${message.partition} `
);
}
推荐答案
您是否生成带有密钥的消息?在Kafka中,具有相同密钥的邮件将发布到同一分区.
Do you produce messages with a key? In Kafka, messages with the same key are published to the same partition.
这篇关于ProducerStream仅产生单个分区的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文