使用 PySpark 将 JSON 文件读取为 Pyspark 数据帧? [英] Read JSON file as Pyspark Dataframe using PySpark?

问题描述

如何使用 PySpark 读取以下 JSON 结构以触发数据帧?

How can I read the following JSON structure to spark dataframe using PySpark?

我的 JSON 结构

{"results":[{"a":1,"b":2,"c":"name"},{"a":2,"b":5,"c":"foo"}]}

我试过:

df = spark.read.json('simple.json');

我希望输出 a,b,c 作为列和值作为相应的行.

I want the output a,b,c as columns and values as respective rows.

谢谢.

推荐答案

Json 字符串变量

如果你有 json 字符串作为变量 那么你可以做

If you have json strings as variables then you can do

simple_json = '{"results":[{"a":1,"b":2,"c":"name"},{"a":2,"b":5,"c":"foo"}]}'
rddjson = sc.parallelize([simple_json])
df = sqlContext.read.json(rddjson)

from pyspark.sql import functions as F
df.select(F.explode(df.results).alias('results')).select('results.*').show(truncate=False)

这会给你

+---+---+----+
|a  |b  |c   |
+---+---+----+
|1  |2  |name|
|2  |5  |foo |
+---+---+----+

Json 字符串作为文件中的单独行(sparkContext 和 sqlContext)

如果您有 json 字符串作为文件中的单独行，那么您可以使用 sparkContext 将其读取到 rdd[string] 中，如上所述，其余过程相同如上

If you have json strings as separate lines in a file then you can read it using sparkContext into rdd[string] as above and the rest of the process is same as above

rddjson = sc.textFile('/home/anahcolus/IdeaProjects/pythonSpark/test.csv')
df = sqlContext.read.json(rddjson)
df.select(F.explode(df['results']).alias('results')).select('results.*').show(truncate=False)

Json 字符串作为文件中的单独行(仅限 sqlContext)

如果您有 json 字符串作为文件中的单独行，那么您只能使用 sqlContext.但是这个过程很复杂，因为你必须为它创建模式

If you have json strings as separate lines in a file then you can just use sqlContext only. But the process is complex as you have to create schema for it

df = sqlContext.read.text('path to the file')

from pyspark.sql import functions as F
from pyspark.sql import types as T
df = df.select(F.from_json(df.value, T.StructType([T.StructField('results', T.ArrayType(T.StructType([T.StructField('a', T.IntegerType()), T.StructField('b', T.IntegerType()), T.StructField('c', T.StringType())])))])).alias('results'))
df.select(F.explode(df['results.results']).alias('results')).select('results.*').show(truncate=False)

这应该给你与上面相同的结果

which should give you same as above result

希望回答对你有帮助

这篇关于使用 PySpark 将 JSON 文件读取为 Pyspark 数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！