Sqoop:如何在将数据从 RDBMS 导入 Hive 表时处理重复值 [英] Sqoop: How to deal with duplicate values while importing data from RDBMS to Hive tables

问题描述

Sqoop:如何在将数据从 RDBMS 导入 Hive 表时处理重复值.

Sqoop: How to deal with duplicate values while importing data from RDBMS to Hive tables.

或者如果 Hive 表中的值已经可用，则处理冗余选项?

Or to deal with redundancy option if the values are already available in Hive Tables?

推荐答案

如果您的数据具有唯一标识符并且您正在运行增量导入，您可以在导入的 -mergeKey 值上指定它.这会将表中已有的值与最新的值合并.较新的将覆盖最旧的.

If your data has a unique identifier and you are running incremental imports you can specify it on the -mergeKey value of the import. This will merge the values that where already on the table with the newest one. The newer will override the oldest.

如果您没有运行增量导入，您可以使用 sqoop 合并来统一数据.来自 sqoop 文档:

If you are not running incremental imports you can use sqoop merge to unify data. From sqoop docs :

在合并数据集时，假设每条记录中都有一个唯一的主键值.主键的列用 --merge-key 指定.同一数据集中的多行不能有相同的主键，否则会造成数据丢失.

When merging the datasets, it is assumed that there is a unique primary key value in each record. The column for the primary key is specified with --merge-key. Multiple rows in the same dataset should not have the same primary key, or else data loss may occur.

重要的是每个记录都有一个唯一的主键.否则，您可能会在导入数据时生成一个.为此，您可以使用 --query 生成导入，并使用连接现有列的数据选择上的唯一键生成新列，直到获得唯一组合.

The important is that you do have a single unique primary key for each record. Otherwise you might generate one when importing the data. To do so you could generate the import with the --query and generate the new column with the unique key on the select of the data concatenating existing columns until you get a unique combination.

--query "SELECT CONVERT(VARCHAR(128), [colum1]) + '_' + CONVERT(VARCHAR(128), [column2]) AS CompoundKey ,* FROM [dbo].[tableName] WHERE \$CONDITIONS" \

这篇关于Sqoop:如何在将数据从 RDBMS 导入 Hive 表时处理重复值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！