如何在 DataFrame Spark 1.6 中加载特定的 Hive 分区? [英] How to load specific Hive partition in DataFrame Spark 1.6?

问题描述

根据官方 doc<，Spark 1.6 起/a> 我们无法向 DataFrame 添加特定的 hive 分区

Spark 1.6 onwards as per the official doc we cannot add specific hive partitions to DataFrame

直到 Spark 1.5 使用以下内容，数据框将具有实体列和数据，如下所示:

Till Spark 1.5 the following used to work and the dataframe would have entity column and the data, as shown below:

DataFrame df = hiveContext.read().format("orc").load("path/to/table/entity=xyz")

但是，这在 Spark 1.6 中不起作用.

However, this would not work in Spark 1.6.

如果我提供如下所示的基本路径，它不包含我在 DataFrame 中想要的实体列，如下所示 -

If I give base path like the following it does not contain entity column which I want in DataFrame, as shown below -

DataFrame df = hiveContext.read().format("orc").load("path/to/table/") 

如何在数据帧中加载特定的 hive 分区?删除此功能的驱动因素是什么?

How do I load specific hive partition in a dataframe? What was the driver behind removing this feature?

我相信它是有效的.在 Spark 1.6 中是否有替代方案可以实现这一点?

I believe it was efficient. Is there an alternative to achieve that in Spark 1.6?

根据我的理解，Spark 1.6 加载所有分区，如果我过滤特定分区效率不高，它会命中内存并抛出 GC(垃圾收集)错误，因为数千个分区被加载到内存中而不是特定分区.

As per my understanding, Spark 1.6 loads all partitions and if I filter for specific partitions it is not efficient, it hits memory and throws GC(Garbage Collection) errors because of thousands of partitions get loaded into memory and not the specific partition.

推荐答案

要使用 Spark 1.6 在 DataFrame 中添加特定分区，我们必须执行以下操作，首先设置 basePath 然后给出分区需要的路径待加载

To add specific partition in a DataFrame using Spark 1.6 we have to do the following first set basePath and then give path of partition needs to be loaded

DataFrame df = hiveContext.read().format("orc").
               option("basePath", "path/to/table/").
               load("path/to/table/entity=xyz")

所以上面的代码只会加载 DataFrame 中的特定分区.

So above code will load only specific partition in a DataFrame.

这篇关于如何在 DataFrame Spark 1.6 中加载特定的 Hive 分区?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！