Kafka ConsumerConnector的主题计数参数是什么 [英] what is topic count parameter for Kafka ConsumerConnector

问题描述

我是apache kafka的新手，请尝试使用给出的示例.

I am new to apache kafka, and try with the examples given.

以下代码片段用于初始化ConsumerConnector，我对topic count参数感到困惑；看来这将导致kafka为该主题分发相应数量的流.但是，我尝试了几次，只有第一个流产生消息.因此，有两个问题: 1.如何确定主题的计数? 2.消息如何拆分穿过流?

The following code snippet is used to initialize a ConsumerConnector, I am confused by the topic count parameter; it seems it will cause kafka hands out corresponding number of streams for that topic. however, I tried several times, only the first stream produces messages. So, two questions: 1. how can I determine the count number for a topic? 2. how does the messages split cross over the streams?

先谢谢了.

    Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
    **topicCountMap.put(topic, new Integer(a_numThreads));**
    Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer
            .createMessageStreams(topicCountMap);
    List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);

推荐答案

如果我稍有不正确，另一个用户可能会随意纠正我，但是:

Another user may feel free to correct me if I am slightly incorrect but:

我之前已经看过这个示例，并且代码通常初始化为

I have looked at this example before and the code is typically initialized as

topicCountMap.put(topic, new Integer(1));

其中一个是整数，因为对于给定的示例，示例主题通常仅使用一个分区来创建.当您的主题只有一个分区时，就没有并行能力(就像您在同一组中有多个使用者一样，会有一些使用者实例不获取任何数据)，因此，没有理由创建多个流.因此，对于此单个分区，仅创建一个流，这就是为什么您只有第一个(也是唯一的)流生成消息的原因.

With one being the integer because for the examples given the sample topic is usually created with only a single partition. When your topic only has a single partition, there is no capability of parallelism (as if you had multiple consumers in the same group there will be some consumer instances that do not get any data), and thus, no reason to create multiple streams. As such, for this single partition only a single stream is created and that is why you only have the first (and only) stream producing messages.

这篇关于Kafka ConsumerConnector的主题计数参数是什么的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！