如何配置 spark 以创建“_$folder$"?S3 中的条目? [英] How can I configure spark so that it creates "_$folder$" entries in S3?

问题描述

当我使用

df.write
  .format("parquet")
  .mode("overwrite")
  .partitionBy("year", "month", "day", "hour", "gen", "client")
  .option("compression", "gzip")
  .save("s3://xxxx/yyyy")

我在 S3 中得到以下内容

I get the following in S3

year=2018
year=2019

但我想要这个:

year=2018
year=2018_$folder$
year=2019
year=2019_$folder$

从该 S3 位置读取的脚本取决于 *_$folder$ 条目，但我还没有找到配置 spark/hadoop 来生成它们的方法.

The scripts that are reading from that S3 location depend on the *_$folder$ entries, but I haven't found a way to configure spark/hadoop to generate them.

知道什么 hadoop 或 spark 配置设置控制 *_$folder$ 文件的生成吗?

Any idea on what hadoop or spark configuration setting control the generation of *_$folder$ files?

推荐答案

那些标记是遗留功能；我认为不再有任何东西可以创建它们……尽管在实际列出目录时它们经常被忽略.(也就是说，即使在那里，它们也会从列表中删除并替换为目录条目).

those markers a legacy feature; I don't think anything creates them any more...though they are often ignored when actually listing directories. (that is, even if there, they get stripped from listings and replaced with directory entries).

这篇关于如何配置 spark 以创建“_$folder$"?S3 中的条目?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！