如何整合Storm和Kafka [英] How to integrate Storm and Kafka

问题描述

我曾在 Storm 中工作，并开发了使用本地文本文件作为输入源.但是现在我必须处理来自外部系统的连续流数据.为此，Kafka是最佳选择.

I have worked in Storm and developed a basic program which is using a local text file as input source. But now I have to work on streaming data coming continuously from external systems. For this purpose, Kafka is the best choice.

问题在于如何使我的Spout从Kafka获取流数据.或者如何将Storm与Kafka集成.我该怎么做才能处理来自Kafka的数据?

The problem is how to make my Spout get streaming data from Kafka. Or how to integrate Storm with Kafka. How can I do that so I may process data, coming from Kafka?

推荐答案

查找这是一个普通的Storm Spout实现，它从Kafka集群中读取.您所需要做的就是使用list of brokers，topic name等参数配置该喷口.您只需将输出链接到相应的螺栓即可进行进一步处理.

It's a normal Storm Spout implementation that reads from a Kafka cluster. All you need is to configure that spout with parameters like list of brokers, topic name, etc. You can simply then chain the output to corresponding bolts for further processing.

根据上述同一文档，配置如下:

From the same doc mentioned above, the configuration goes like this:

SpoutConfig spoutConfig = new SpoutConfig(
      ImmutableList.of("kafkahost1", "kafkahost2"), // List of Kafka brokers
      8, // Number of partitions per host
      "clicks", // Topic to read from
      "/kafkastorm", // The root path in Zookeeper for the spout to store the consumer offsets
      "discovery"); // An id for this consumer for storing the consumer offsets in Zookeeper

KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

这篇关于如何整合Storm和Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！