什么是 Kafka ConsumerConnector 的主题计数参数 [英] what is topic count parameter for Kafka ConsumerConnector

问题描述

我是 apache kafka 的新手，并尝试使用给出的示例.

I am new to apache kafka, and try with the examples given.

以下代码片段用于初始化一个ConsumerConnector，我被topic count参数搞糊涂了；似乎这会导致 kafka 为该主题分发相应数量的流.但是，我尝试了几次，只有第一个流产生消息.所以，两个问题:1. 如何确定主题的计数?2. 消息如何跨流拆分?

The following code snippet is used to initialize a ConsumerConnector, I am confused by the topic count parameter; it seems it will cause kafka hands out corresponding number of streams for that topic. however, I tried several times, only the first stream produces messages. So, two questions: 1. how can I determine the count number for a topic? 2. how does the messages split cross over the streams?

提前致谢.

    Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
    **topicCountMap.put(topic, new Integer(a_numThreads));**
    Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer
            .createMessageStreams(topicCountMap);
    List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);

推荐答案

如果我有点不正确，其他用户可以随时纠正我，但是:

Another user may feel free to correct me if I am slightly incorrect but:

我之前看过这个例子，代码通常被初始化为

I have looked at this example before and the code is typically initialized as

topicCountMap.put(topic, new Integer(1));

一个是整数，因为对于给定的示例，示例主题通常只创建一个分区.当您的主题只有一个分区时，就没有并行能力(就像您在同一组中有多个消费者一样，会有一些消费者实例没有获取任何数据)，因此，没有理由创建多个流.因此，对于这个单个分区，只会创建一个流，这就是为什么您只有第一个(也是唯一一个)流生成消息.

With one being the integer because for the examples given the sample topic is usually created with only a single partition. When your topic only has a single partition, there is no capability of parallelism (as if you had multiple consumers in the same group there will be some consumer instances that do not get any data), and thus, no reason to create multiple streams. As such, for this single partition only a single stream is created and that is why you only have the first (and only) stream producing messages.

这篇关于什么是 Kafka ConsumerConnector 的主题计数参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！