是否可以更改Hive中分区表上的列的元数据？ [英] Is it possible to change the metadata of a column that is on a partitioned table in Hive?

问题描述

这是我之前提出的问题的扩展：可以在HIVE中更改分区元数据吗？

This is an extension of a previous question I asked: Is it possible to change partition metadata in HIVE?

我们正在探索更改表上元数据的想法，而不是对表上执行CAST操作。 SELECT语句中的数据。更改MySQL Metastore中的元数据非常容易。但是，是否有可能将元数据更改应用于分区表（每天）上的列？注意：该列本身不是分区列。这是一个简单的ID字段，已从STRING更改为BIGINT。

We are exploring the idea of changing the metadata on the table as opposed to performing a CAST operation on the data in SELECT statements. Changing the metadata in the MySQL metastore is easy enough. But, is it possible to have that metadata change applied to a column that is on a partitioned table (they are daily)? Note: the column itself is not the partitioning column. It is a simple ID field that is being changed from STRING to BIGINT.

否则，当历史记录为STRING时，我们可能会遇到当前和将来的数据为BIGINT类型的情况

Otherwise, we might be stuck with current and future data being of type BIGINT while the historical is STRING.

问题：是否可以在Hive中更改分区元数据？如果是，怎么办？

Question: Is it possible to change partition meta data in Hive? If yes, how?

注意：我将其作为一个单独的问题进行询问，因为原始答案似乎是针对分区表中的一列，该表也是分区列。因此，我不想弄混水域。

Note: I am asking this as a separate question as the original answer appears to be for a column on a partitioned table that is also the partitioning column. So, I do not want to muddy the waters.

更新：

我运行了 ALTER TABLE .. CHANGE COLUMN ... CASCADE 命令，但出现以下错误：

I ran the ALTER TABLE .. CHANGE COLUMN ... CASCADE command, but I get the following error:

处理语句时出错：失败：执行错误，从org.apache.hadoop.hive.ql.exec.DDLTask返回代码
1。不允许更改具有外部架构的Avro存储表的
架构。考虑从表属性中删除
avro.schema.literal或avro.schema.url。

Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Not allowed to alter schema of Avro stored table having external schema. Consider removing avro.schema.literal or avro.schema.url from table properties.

元数据存储在单独的avro文件。我可以确认更新的元数据在avro文件中，而不在单个分区文件中。

The metadata is stored in a separate avro file. I can confirm that the updated metadata is in the avro file, but not in the individual partition file.

注意：该表存储为EXTERNAL。

Note: The table is stored as EXTERNAL.

推荐答案

您可以轻松更改列类型：

You can easily change column type:

在Hive中使用alter table，更改类型到STRING，等等：

Use alter table in Hive, change type to STRING, etc:

alter table table_name change column col_name col_name string cascade; --change to string

请参见文档。

使用 CASCADE 命令更改表更改列表的元数据的列，并将相同的更改层叠到所有分区元数据。

ALTER TABLE CHANGE COLUMN with CASCADE command changes the columns of a table's metadata, and cascades the same change to all the partition metadata.

或者，您可以像下面的答案那样重新创建表：https://stackoverflow.com/a/58299056/2700344

Alternatively you can recreate table like in this answer: https://stackoverflow.com/a/58299056/2700344

这篇关于是否可以更改Hive中分区表上的列的元数据？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！