将 JSON 文件读入 Spark 时出现 _corrupt_record 错误 [英] _corrupt_record error when reading a JSON file into Spark

问题描述

我有这个 JSON 文件

I've got this JSON file

{
    "a": 1, 
    "b": 2
}

已通过 Python json.dump 方法获得.现在，我想使用 pyspark 将此文件读入 Spark 中的 DataFrame.按照文档，我正在这样做

which has been obtained with Python json.dump method. Now, I want to read this file into a DataFrame in Spark, using pyspark. Following documentation, I'm doing this

sc = SparkContext()

sc = SparkContext()

sqlc = SQLContext(sc)

sqlc = SQLContext(sc)

df = sqlc.read.json('my_file.json')

df = sqlc.read.json('my_file.json')

打印df.show()

print df.show()

打印语句虽然吐出了这个:

The print statement spits out this though:

+---------------+
|_corrupt_record|
+---------------+
|              {|
|       "a": 1, |
|         "b": 2|
|              }|
+---------------+

有人知道发生了什么以及为什么它不能正确解释文件吗?

Anyone knows what's going on and why it is not interpreting the file correctly?

推荐答案

您的输入文件中的每一行都需要有一个 json 对象，请参阅 http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader.json

You need to have one json object per row in your input file, see http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader.json

如果您的 json 文件如下所示，它将为您提供预期的数据框:

If your json file looks like this it will give you the expected dataframe:

{ "a": 1, "b": 2 }
{ "a": 3, "b": 4 }

....
df.show()
+---+---+
|  a|  b|
+---+---+
|  1|  2|
|  3|  4|
+---+---+

这篇关于将 JSON 文件读入 Spark 时出现 _corrupt_record 错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！