如何将JavaPairDStream的结果写入Spark Streaming的输出kafka主题中? [英] How can I write results of JavaPairDStream into output kafka topic on Spark Streaming?

问题描述

我正在寻找一种在输出kafka主题中编写Dstream的方法，仅当微型批处理RDD吐出某些东西时.

I'm looking for a way to write a Dstream in an output kafka topic, only when the micro-batch RDDs spit out something.

我正在Java8(均为最新版本)中使用Spark Streaming和spark-streaming-kafka连接器

I'm using Spark Streaming and spark-streaming-kafka connector in Java8 (both latest versions)

我不知道.

感谢您的帮助.

推荐答案

如果dStream包含要发送到Kafka的数据:

if dStream contains data that you want to send to Kafka:

dStream.foreachRDD(rdd -> {
    rdd.foreachPartition(iter ->{
        Producer producer = createKafkaProducer();  
        while (iter.hasNext()){
               sendToKafka(producer, iter.next())
        }
    }

});

因此，您为每个RDD分区创建一个生产者.

So, you create one producer per each RDD partition.

这篇关于如何将JavaPairDStream的结果写入Spark Streaming的输出kafka主题中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！