保存DataFrame时如何避免生成crc文件和SUCCESS文件? [英] How to avoid generating crc files and SUCCESS files while saving a DataFrame?
问题描述
我使用以下代码将 spark DataFrame 保存到 JSON 文件
I am using the following code to save a spark DataFrame to JSON file
unzipJSON.write.mode("append").json("/home/eranw/Workspace/JSON/output/unCompressedJson.json")
输出结果为:
part-r-00000-704b5725-15ea-4705-b347-285a4b0e7fd8
.part-r-00000-704b5725-15ea-4705-b347-285a4b0e7fd8.crc
part-r-00001-704b5725-15ea-4705-b347-285a4b0e7fd8
.part-r-00001-704b5725-15ea-4705-b347-285a4b0e7fd8.crc
_SUCCESS
._SUCCESS.crc
- 如何生成单个 JSON 文件而不是每行一个文件?
- 如何避免 *crc 文件?
- 如何避免 SUCCESS 文件?
推荐答案
如果你想要单个文件,你需要在调用 write 之前对单个分区做一个coalesce
,所以:
If you want a single file, you need to do a coalesce
to a single partition before calling write, so:
unzipJSON.coalesce(1).write.mode("append").json("/home/eranw/Workspace/JSON/output/unCompressedJson.json")
就个人而言,我发现输出文件的数量取决于您在调用 write
之前拥有的分区数量这一点很烦人 - 特别是如果您使用 write
执行 write
code>partitionBy - 但据我所知,目前没有其他方法.
Personally, I find it rather annoying that the number of output files depend on number of partitions you have before calling write
- especially if you do a write
with a partitionBy
- but as far as I know, there are currently no other way.
我不知道是否有一种方法可以禁用 .crc 文件 - 我不知道有一种方法 - 但是您可以通过在 Spark 上下文的 hadoop 配置中设置以下内容来禁用 _SUCCESS 文件.>
I don't know if there is a way to disable the .crc files - I don't know of one - but you can disable the _SUCCESS file by setting the following on the hadoop configuration of the Spark context.
sc.hadoopConfiguration.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")
请注意,您可能还想通过以下方式禁用元数据文件的生成:
Note, that you may also want to disable generation of the metadata files with:
sc.hadoopConfiguration.set("parquet.enable.summary-metadata", "false")
显然,生成元数据文件需要一些时间(参见 这篇博文) 但实际上并不那么重要(根据 这个).就个人而言,我总是禁用它们,我没有遇到任何问题.
Apparently, generating the metadata files takes some time (see this blog post) but aren't actually that important (according to this). Personally, I always disable them and I have had no issues.
这篇关于保存DataFrame时如何避免生成crc文件和SUCCESS文件?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!