PySpark:写入时吐出单个文件而不是多个部分文件 [英] PySpark: spit out single file when writing instead of multiple part files

问题描述

有没有办法防止 PySpark 在将 DataFrame 写入 JSON 文件时创建多个小文件?

Is there a way to prevent PySpark from creating several small files when writing a DataFrame to JSON file?

如果我跑:

 df.write.format('json').save('myfile.json')

或

df1.write.json('myfile.json')

它创建名为 myfile 的文件夹，在其中我找到了几个名为 part-*** 的小文件，这是 HDFS 方式.有什么办法可以让它吐出单个文件吗?

it creates the folder named myfile and within it I find several small files named part-***, the HDFS way. Is it by any means possible to have it spit out a single file instead?

推荐答案

好吧，您的确切问题的答案是 coalesce 函数.但正如已经提到的那样，它根本没有效率，因为它会强制一个工人获取所有数据并按顺序写入.

Well, the answer to your exact question is coalesce function. But as already mentioned it is not efficient at all as it will force one worker to fetch all data and write it sequentially.

df.coalesce(1).write.format('json').save('myfile.json')

附言顺便说一句，结果文件不是有效的 json 文件.它是一个每行一个 json 对象的文件.

P.S. Btw, the result file is not a valid json file. It is a file with a json object per line.

这篇关于PySpark:写入时吐出单个文件而不是多个部分文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！