如何在不使用Sqoop的情况下自动将Hive外部表与MySQL表同步? [英] How to automatically sync a Hive external table with a MySQL table without using Sqoop?

问题描述

我在本地计算机(Linux)本身中已经有一个MySQL表，并且我有一个Hive外部表，该表具有与MySQL表相同的架构.

I'm already having a MySQL table in my local machine (Linux) itself, and I have a Hive external table with the same schema as the MySQL table.

无论何时插入或更新新记录，我都希望同步我的配置单元外部表.我可以每小时进行一次批量更新. 在不使用sqoop的情况下实现相同效果的最佳方法是什么?

I want to sync my hive external table whenever a new record is inserted or updated. Batch update is ok with me to say hourly. What is the best possible approach to achieve the same without using sqoop?

谢谢， 提交

推荐答案

如果没有任何发现，您可以创建表STORED BY JdbcStorageHandler .项目存储库: https://github.com/qubole/Hive-JDBC- Storage-Handler 它将像通常的蜂巢表一样工作，但是查询将在MySQL上运行.谓词下推将起作用.

Without scoop, you can create table STORED BY JdbcStorageHandler. Project repository: https://github.com/qubole/Hive-JDBC-Storage-Handler It will work as usual hive table, but query will run on MySQL. Predicate pushdown will work.

DROP TABLE HiveTable;
CREATE EXTERNAL TABLE HiveTable(
  id INT,
  id_double DOUBLE,
  names STRING,
  test INT
)
STORED BY 'org.apache.hadoop.hive.jdbc.storagehandler.JdbcStorageHandler'
TBLPROPERTIES (
  "mapred.jdbc.driver.class"="com.mysql.jdbc.Driver",
  "mapred.jdbc.url"="jdbc:mysql://localhost:3306/rstore",
  "mapred.jdbc.username"="root",
  "mapred.jdbc.input.table.name"="JDBCTable",
  "mapred.jdbc.output.table.name"="JDBCTable",
  "mapred.jdbc.password"="",
  "mapred.jdbc.hive.lazy.split"= "false"
);

这篇关于如何在不使用Sqoop的情况下自动将Hive外部表与MySQL表同步?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！