Kafka Spout 的字段分组 [英] Field Grouping for a Kafka Spout

问题描述

可以对 kafka spout 发出的元组进行字段分组吗?如果是，那么 Storm 是如何知道 Kafka 记录中的字段的?

Can field grouping be done on tuples emitted by a kafka spout? If yes, then how does Storm gets to know the fields in a Kafka record?

推荐答案

Kafka Spout 像任何其他组件一样声明其输出字段.我的解释基于 KafkaSpout 的当前 implementation.

Kafka Spout declared its output fields like any other component. My explanation is based on current implementation of KafkaSpout.

在KafkaSpout.java 类中，我们看到declareOutputFields 方法调用KafkaConfig Scheme 的getOutputFields() 方法.

In KafkaSpout.java class we see declareOutputFields method that call getOutputFields() method of KafkaConfig Scheme.

@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
    declarer.declare(_spoutConfig.scheme.getOutputFields());
}

默认情况下，KafkaConfig 使用 RawMultiScheme 以这种方式实现此方法.

By default, KafkaConfig uses RawMultiScheme that implements this method in this way.

  @Override
  public Fields getOutputFields() {
    return new Fields("bytes");
  }

那是什么意思?，如果你声明了用 fieldGrouping 从 KafkaSpout 读取元组的 bolt，你就知道每个包含等于字段bytes"的元组都将由同一个任务执行.如果你想发出任何字段，你应该根据你的需要实现新的方案.

So what does it mean?, if you declared bolt which reads tuples from KafkaSpout with fieldGrouping you know that every tuple that contains equals field "bytes" is going to be executed by the same task. If you want to emit any field, you should implement new scheme for your needs.

这篇关于Kafka Spout 的字段分组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！