从Spark删除分区 [英] Drop partitions from Spark

问题描述

我正在使用Java-Spark(Spark 2.2.0).

I'm using Java-Spark (Spark 2.2.0).

我正在尝试删除Hive分区，如下所示:

I'm trying to drop Hive partitions as follow:

spark.sql(""ALTER TABLE backup DROP PARTITION (date < '20180910')"

并得到以下异常:

org.apache.spark.sql.catalyst.parser.ParseException: 输入'<'不匹配期待{')'，'，'}(第1行，pos 42)

org.apache.spark.sql.catalyst.parser.ParseException: mismatched input '<' expecting {')', ','}(line 1, pos 42)

我知道这是一个未解决的问题 ALTER TABLE DROP PARTITION应该支持比较器应该会在我的版本中修复，但我仍然会遇到异常.

I know this is open issue ALTER TABLE DROP PARTITION should support comparators that should be fixed in my version but I'm still get exception.

从Spark删除分区的替代方法是什么?还有另一种实现方法吗?

What is my alternative to drop the partitions from Spark? There is another implementation to do that?

谢谢.

推荐答案

您可以使用spark编程执行相同的操作.同样，它在Spark 2、2.1及更高版本中未修复. 2.2 ref https://issues.apache.org/jira/browse/SPARK-14922

You can do the same with spark programming. Also its not fixed in Spark 2 , 2.1 & 2.2 for ref https://issues.apache.org/jira/browse/SPARK-14922

    Steps 

        1 . Create hive context 
        2 . Get the table for getTable method from the hive context and you need to pass dbName, tableName and a boolean value if any error
        3 . From table Object hive.getPartitions(table) you can get the partitions from hive context (you need to decide which partitions you are going delete )
            4 . You can remove partitions using dropPartition with partition values , table name and db info (hive.dropPartition) 

    hiveContext.getPartitions(table)
    hiveContext.dropPartition(dbName, tableName, partition.getValues(), true)


You need to validate the partition name and check whether it needs to be deleted or not (you need to write custom method ).

       Or you can get the partition list sql using show partitions and from there also you can use drop partition to remove it.

This may give you some pointers .

这篇关于从Spark删除分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！