如何将json字符串转换为spark上的数据帧 [英] how to convert json string to dataframe on spark
本文介绍了如何将json字符串转换为spark上的数据帧的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想将下面的字符串变量转换为 spark 上的数据帧.
val jsonStr = "{ "metadata": { "key": 84896, "value": 54 }}"
我知道如何从 json 文件创建数据框.
sqlContext.read.json("file.json")
但我不知道如何从字符串变量创建数据框.
如何将 json 字符串变量转换为数据框.
解决方案
对于 Spark 2.2+:
import spark.implicits._val jsonStr = """{ "metadata": { "key": 84896, "value": 54 }}"""val df = spark.read.json(Seq(jsonStr).toDS)
val events = sc.parallelize("""{"action":"create","timestamp":"2016-01-07T00:01:17Z"}""" :: Nil)val df = sqlContext.read.json(events)
<块引用>
提示:这是使用 sqlContext.read.json(jsonRDD: RDD[Stirng])
重载.还有 sqlContext.read.json(path: String)
直接读取 Json 文件.
对于旧版本:
val jsonStr = """{ "metadata": { "key": 84896, "value": 54 }}"""val rdd = sc.parallelize(Seq(jsonStr))val df = sqlContext.read.json(rdd)
I want to convert string variable below to dataframe on spark.
val jsonStr = "{ "metadata": { "key": 84896, "value": 54 }}"
I know how to create dataframe from json file.
sqlContext.read.json("file.json")
but I don't know how to create dataframe from string variable.
How can I convert json String variable to dataframe.
解决方案
For Spark 2.2+:
import spark.implicits._
val jsonStr = """{ "metadata": { "key": 84896, "value": 54 }}"""
val df = spark.read.json(Seq(jsonStr).toDS)
For Spark 2.1.x:
val events = sc.parallelize("""{"action":"create","timestamp":"2016-01-07T00:01:17Z"}""" :: Nil)
val df = sqlContext.read.json(events)
Hint: this is using
sqlContext.read.json(jsonRDD: RDD[Stirng])
overload. There is alsosqlContext.read.json(path: String)
where it reads a Json file directly.
For older versions:
val jsonStr = """{ "metadata": { "key": 84896, "value": 54 }}"""
val rdd = sc.parallelize(Seq(jsonStr))
val df = sqlContext.read.json(rdd)
这篇关于如何将json字符串转换为spark上的数据帧的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文