如何将mysql数据迁移到ElasticSearch实时 [英] how to migrate mysql data to ElasticSearch realtime

问题描述

我有一个mysql数据库与几个表，我想将mysql数据迁移到ElasticSearch。通过批处理作业将整个数据库迁移到ES很容易。但是我应该如何从mysql实时更新ES。即如果在mysql中有更新操作，那么我应该在ES中执行相同的操作。
我研究了可以反映mysql中任何更改的mysql binLog。但是我必须将binLog解析为ES语法，我觉得这真的很痛苦。谢谢！
（与Solr相同的情况）

解决方案

有一个现有的项目需要您的binlog，转换和运送您可以通过以下方式查看： https://github.com/siddontang/go- mysql-elasticsearch

另一个是这样的： https://github.com/noplay/python-mysql-replication 。

然而，请注意，无论选择哪一个，在索引binlog之前预先创建索引和映射是一个很好的做法。这样可以更好地控制您的数据。

I have a mysql database with couple tables, I wanna migrate the mysql data to ElasticSearch. It's easy to migrate the whole database to ES via a batch job. But how should I update ES from mysql realtime. i.e if there was a update operation in mysql then I should do the same operation in ES. I researched mysql binLog which can reflect any changes from mysql. But I have to parse binLog to ES syntax, I think it's really painful. Thanks! (the same case with Solr)

解决方案

There is an existing project which takes your binlog, transforms it and ships it to Elasticsearch, You can check it out at: https://github.com/siddontang/go-mysql-elasticsearch

Another one would be this one: https://github.com/noplay/python-mysql-replication.

Note, however, that whichever you pick, it's a good practice to pre-create your index and mappings before indexing your binlog. That gives you more control over your data.

这篇关于如何将mysql数据迁移到ElasticSearch实时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！