Spark Direct流-在多个使用者中使用同一条消息 [英] Spark Direct Streaming - consume same message in multiple consumers

问题描述

如何使用Direct Stream approach?

有可能吗?由于直接流方法没有Consumer Group概念.

Is it possible? Since Direct Stream approach doesn't have Consumer Group concept.

如果我通过group.id作为DirectStream方法的kafkaparams，会发生什么?以下代码with group.id与Kafka Params以及without group.id一样工作.

What happens, if i pass group.id as kafkaparams for the DirectStream method? The below code works with group.id as Kafka Params also without group.id.

示例代码:

val kafkaParams = Map(
  "group.id" -> "group1", 
  CommonClientConfigs.SECURITY_PROTOCOL_CONFIG -> sasl,
  ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer",
  ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer",
  "metadata.broker.list" -> brokerList,
  "zookeeper.connect" -> zookeeperURL
)

val dStream = 
  KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
    ssc, kafkaParams, topicSet
  ).map(_._2)

推荐答案

没有任何反应，Sparks直接流方法完全不使用group id参数，因为它使用较低级别的SimpleConsumer.您不能在不同的Spark直接方法流中使用同一主题.您可以采用确实采用分组的基于接收者的较早方法.

Nothing happens, Sparks direct streaming approach doesn't take into account the group id parameter at all as it uses the lower level SimpleConsumer. You can't consume the same topic with different Spark direct approach streams. You can defer to the older receiver based approach which does utilize groups.

这篇关于Spark Direct流-在多个使用者中使用同一条消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！