从MySQL连续数据迁移到Hbase [英] Continuous data migration from mysql to Hbase

问题描述

我已经安装了hadoop和hbase以进行实时分析。我面临的问题是将数据从mysql迁移到Hbase。

sqoop工具对于批量数据迁移很有用，是否有任何方法可以将mysql中的数据传输到HBase上（当然还有插入/更新/删除发生）。因此可以实现实时分析。不是近实时的。

请在这方面帮助我。

解决方案

我认为您需要在不同的DBMS之间设置复制任务。原生复制机制无用时就是这种情况。
最简单的解决方案是在要复制的表上创建一组触发器，并将要复制的数据写入一些附加表中。然后，您可以设置监控此表并将更改应用于HBase。

更健壮但更复杂的解决方案是分析用于其本机复制机制的MySQL日志，并将更改应用于HBase。

在同一时间，我不清楚锄头HBase会给你实时分析。我在这里写下了关于这个问题的技巧：
HBase集团

I have installed hadoop and hbase for real time analytics purpose. The proble I face is to migrate data on line from mysql to Hbase.

The sqoop tool is useful to do bulk data migrations, is there any way that the data from mysql can be transfered to HBase on line (then and there when an insert/update/delete happens). So that real time analytics can be achieved. Not near-real-time.

Please help me on this regards.

解决方案

I think you faced task of setting up replication between different DBMS. It is a case when native replication mechanism is not useful. Simplest solution would be create set of triggers over tables you want to replicate - and write the data to be replicated into some additional table. Then you can set up monitoring this table and apply changes to the HBase.
More robust, but more complicated solution would be to analyze MySQL log used for its native replication mechanism and apply the changes to the HBase.
In the same time it is not clear for me hoe HBase will give you real-time analytics. I wrote abit about this issue here: Group by In HBase

这篇关于从MySQL连续数据迁移到Hbase的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！