如何在Apache Spark中使用来自Kafka主题的Scala读取JSON数据 [英] How to read json data using scala from kafka topic in apache spark

问题描述

我是新火花，请让我知道如何使用Apache Spark中来自kafka主题的scala读取json数据.

I am new spark, Could you please let me know how to read json data using scala from kafka topic in apache spark.

谢谢.

推荐答案

最简单的方法是利用Spark附带的DataFrame抽象.

The simplest method would be to make use of the DataFrame abstraction shipped with Spark.

val sqlContext = new SQLContext(sc)
val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
                  ssc, kafkaParams, Set("myTopicName"))

stream.foreachRDD(
  rdd => {
     val dataFrame = sqlContext.read.json(rdd.map(_._2)) //converts json to DF
     //do your operations on this DF. You won't even require a model class.
        })

这篇关于如何在Apache Spark中使用来自Kafka主题的Scala读取JSON数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！