使用PySpark将JSON文件读取为Pyspark Dataframe吗? [英] Read JSON file as Pyspark Dataframe using PySpark?
问题描述
如何读取以下JSON结构以使用PySpark触发数据帧?
How can I read the following JSON structure to spark dataframe using PySpark?
我的JSON结构
{"results":[{"a":1,"b":2,"c":"name"},{"a":2,"b":5,"c":"foo"}]}
我尝试过:
df = spark.read.json('simple.json');
我希望将输出a,b,c作为列,并将值作为相应的行.
I want the output a,b,c as columns and values as respective rows.
谢谢.
推荐答案
Json字符串变量
如果您将 json字符串作为变量,则可以
simple_json = '{"results":[{"a":1,"b":2,"c":"name"},{"a":2,"b":5,"c":"foo"}]}'
rddjson = sc.parallelize([simple_json])
df = sqlContext.read.json(rddjson)
from pyspark.sql import functions as F
df.select(F.explode(df.results).alias('results')).select('results.*').show(truncate=False)
这将为您提供
+---+---+----+
|a |b |c |
+---+---+----+
|1 |2 |name|
|2 |5 |foo |
+---+---+----+
Json字符串作为文件(sparkContext和sqlContext)中的单独行
如果文件中有 json字符串作为单独的行 ,则可以如上所述使用sparkContext将其读取到rdd [string] 中,其余过程相同如上
If you have json strings as separate lines in a file then you can read it using sparkContext into rdd[string] as above and the rest of the process is same as above
rddjson = sc.textFile('/home/anahcolus/IdeaProjects/pythonSpark/test.csv')
df = sqlContext.read.json(rddjson)
df.select(F.explode(df['results']).alias('results')).select('results.*').show(truncate=False)
Json字符串作为文件中的单独行(仅适用于sqlContext)
如果文件中有 json字符串作为单独的行 ,则只能使用sqlContext
.但是该过程很复杂,因为您必须为其创建架构
If you have json strings as separate lines in a file then you can just use sqlContext
only. But the process is complex as you have to create schema for it
df = sqlContext.read.text('path to the file')
from pyspark.sql import functions as F
from pyspark.sql import types as T
df = df.select(F.from_json(df.value, T.StructType([T.StructField('results', T.ArrayType(T.StructType([T.StructField('a', T.IntegerType()), T.StructField('b', T.IntegerType()), T.StructField('c', T.StringType())])))])).alias('results'))
df.select(F.explode(df['results.results']).alias('results')).select('results.*').show(truncate=False)
应该与上述结果相同
我希望答案会有所帮助
这篇关于使用PySpark将JSON文件读取为Pyspark Dataframe吗?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!