如何使用Spark(pyspark)编写镶木地板文件? [英] How can I write a parquet file using Spark (pyspark)?

问题描述

我在Spark中还很陌生，我一直在尝试将Dataframe转换为Spark中的镶木地板文件，但还没有成功. 文档说，我可以使用 write.parquet 函数来创建文件.但是，当我运行脚本时，它向我显示: AttributeError:'RDD'对象没有属性'write'

from pyspark import SparkContext
sc = SparkContext("local", "Protob Conversion to Parquet ")

# spark is an existing SparkSession
df = sc.textFile("/temp/proto_temp.csv")

# Displays the content of the DataFrame to stdout
df.write.parquet("/output/proto.parquet")

您知道如何进行这项工作吗?

我使用的Spark版本是为Hadoop 2.7.3构建的Spark 2.0.1.

解决方案

该错误是由于SparkContext中的textFile方法返回了RDD而我需要的是DataFrame./p>

SparkSession在引擎盖下有一个SQLContext.所以我需要使用DataFrameReader正确读取CSV文件，然后再将其转换为镶木地板文件.

spark = SparkSession \
    .builder \
    .appName("Protob Conversion to Parquet") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

# read csv
df = spark.read.csv("/temp/proto_temp.csv")

# Displays the content of the DataFrame to stdout
df.show()

df.write.parquet("output/proto.parquet")

I'm pretty new in Spark and I've been trying to convert a Dataframe to a parquet file in Spark but I haven't had success yet. The documentation says that I can use write.parquet function to create the file. However, when I run the script it shows me: AttributeError: 'RDD' object has no attribute 'write'

from pyspark import SparkContext
sc = SparkContext("local", "Protob Conversion to Parquet ")

# spark is an existing SparkSession
df = sc.textFile("/temp/proto_temp.csv")

# Displays the content of the DataFrame to stdout
df.write.parquet("/output/proto.parquet")

Do you know how to make this work?

The spark version that I'm using is Spark 2.0.1 built for Hadoop 2.7.3.

解决方案

The error was due to the fact that the textFile method from SparkContext returned an RDD and what I needed was a DataFrame.

SparkSession has a SQLContext under the hood. So I needed to use the DataFrameReader to read the CSV file correctly before converting it to a parquet file.

spark = SparkSession \
    .builder \
    .appName("Protob Conversion to Parquet") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

# read csv
df = spark.read.csv("/temp/proto_temp.csv")

# Displays the content of the DataFrame to stdout
df.show()

df.write.parquet("output/proto.parquet")

这篇关于如何使用Spark(pyspark)编写镶木地板文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！