如何在apache spark中使用kafka主题中的scala读取json数据 [英] How to read json data using scala from kafka topic in apache spark

问题描述

我是新的spark，请告诉我如何使用scala从apache spark中的kafka主题读取json数据.

I am new spark, Could you please let me know how to read json data using scala from kafka topic in apache spark.

谢谢.

推荐答案

最简单的方法是利用 Spark 附带的 DataFrame 抽象.

The simplest method would be to make use of the DataFrame abstraction shipped with Spark.

val sqlContext = new SQLContext(sc)
val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
                  ssc, kafkaParams, Set("myTopicName"))

stream.foreachRDD(
  rdd => {
     val dataFrame = sqlContext.read.json(rdd.map(_._2)) //converts json to DF
     //do your operations on this DF. You won't even require a model class.
        })

这篇关于如何在apache spark中使用kafka主题中的scala读取json数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！