如何将mysql表转移到配置单元？ [英] How to transfer mysql table to hive?

问题描述

我有一个很大的mysql表，我想转移到Hadoop / Hive表。是否有标准的命令或技术将一个简单（但很大）的表从Mysql传输到Hive？该表主要存储分析数据。 首先下载mysql-connector- java-5.0.8并将jar放到Sqoop的lib和bin文件夹中 使用在Hive中创建表格定义 >确切的字段名称和类型在MySQL中

sqoop import --verbose --fields-terminated-by'，'--connect jdbc：mysql： // localhost / test --table employee --hive-import --warehouse-dir / user / hive / warehouse --fields-terminated-by'，'--split-by id --hive-table employee

test - 数据库名称

employee - 表名（存在于测试）

/ user / hive / warehouse - 需要导入数据的HDFS中的目录

- -split-by id-id可以是表'employee'的主键。
$ b - hive-table employee - 雇员表，其定义是pres ent in Hive

Sqoop用户指南（学习Sqoop的最佳指南之一）

I have a large mysql table that I would like to transfer to a Hadoop/Hive table. Are there standard commands or techniques to transfer a simple (but large) table from Mysql to Hive? The table stores mostly analytics data.

解决方案

1. First of all download mysql-connector-java-5.0.8 and put the jar to lib and bin folder of Sqoop

2. Create the table definition in Hive with exact field names and types as in mysql

   sqoop import --verbose --fields-terminated-by ',' --connect jdbc:mysql://localhost/test --table employee --hive-import --warehouse-dir /user/hive/warehouse --fields-terminated-by ',' --split-by id --hive-table employee

test - Database name

employee - Table name (present in test)

/user/hive/warehouse - Directory in HDFS where the data has to be imported

--split-by id - id can be the primary key of the table 'employee'

--hive-table employee - employee table whose definition is present in Hive

Sqoop User Guide (One of the best guide for learning Sqoop)

这篇关于如何将mysql表转移到配置单元？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！