每种格式的写入或读取选项的参考在哪里? [英] Where is the reference for options for writing or reading per format?
问题描述
我使用 Spark 1.6.1.
I use Spark 1.6.1.
我们正在尝试使用 HiveContext 和 DataFrameWriter 将 ORC 文件写入 HDFS.虽然我们可以使用
We are trying to write an ORC file to HDFS using HiveContext and DataFrameWriter. While we can use
df.write().orc(<path>)
我们宁愿做类似的事情
df.write().options(Map("format" -> "orc", "path" -> "/some_path")
这样我们就可以根据使用此帮助程序库的应用程序灵活地更改格式或根路径.我们在哪里可以找到对可以传递给 DataFrameWriter 的选项的引用?我在这里的文档中什么也没找到
This is so that we have the flexibility to change the format or root path depending on the application that uses this helper library. Where can we find a reference to the options that can be passed into the DataFrameWriter? I found nothing in the docs here
推荐答案
我们在哪里可以找到对可以传递给 DataFrameWriter 的选项的引用?
Where can we find a reference to the options that can be passed into the DataFrameWriter?
最权威的答案是来源:
您可能会在文档中找到一些描述，但没有单个页面(可能会从来源自动生成以保持最新状态).
Some description you may find in the docs, but there is no single page (that could possibly be auto-generated from the sources to stay up-to-date the most).
原因是有意将选项与格式实现分开，以便为每个用例提供您希望提供的灵活性(如您所见):
The reason being that the options are separated from the format implementation on purpose to have the flexibility you want to offer per use case (as you duly noted):
这样我们就可以根据使用此帮助程序库的应用程序灵活地更改格式或根路径.
This is so that we have the flexibility to change the format or root path depending on the application that uses this helper library.
您的问题似乎类似于 如何知道 Databricks 支持的文件格式? 我说:
在哪里可以获得每种文件格式支持的选项列表?
Where can I get the list of options supported for each file format?
这是不可能的，因为没有 API 可以遵循(如在 Spark MLlib 中)来定义选项.每种格式都会自行执行此操作……不幸的是，您最好的选择是阅读文档或(更权威的)源代码.
That's not possible as there is no API to follow (like in Spark MLlib) to define options. Every format does this on its own...unfortunately and your best bet is to read the documentation or (more authoritative) the source code.
这篇关于每种格式的写入或读取选项的参考在哪里?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
