如何将mysql数据实时迁移到ElasticSearch [英] how to migrate mysql data to ElasticSearch realtime

问题描述

我有一个包含几个表的 mysql 数据库，我想将 mysql 数据迁移到 ElasticSearch.通过批处理作业将整个数据库迁移到 ES 很容易.但是我应该如何从 mysql 实时更新 ES.即如果 mysql 中有更新操作，那么我应该在 ES 中执行相同的操作.我研究了 mysql binLog，它可以反映 mysql 的任何更改.但是我要把binLog解析成ES语法，我觉得真的很痛苦.谢谢！(与 Solr 相同的情况)

I have a mysql database with couple tables, I wanna migrate the mysql data to ElasticSearch. It's easy to migrate the whole database to ES via a batch job. But how should I update ES from mysql realtime. i.e if there was a update operation in mysql then I should do the same operation in ES. I researched mysql binLog which can reflect any changes from mysql. But I have to parse binLog to ES syntax, I think it's really painful. Thanks! (the same case with Solr)

推荐答案

有一个现有项目可以获取您的 binlog，将其转换并将其发送到 Elasticsearch，您可以在以下位置查看:https://github.com/siddontang/go-mysql-elasticsearch

There is an existing project which takes your binlog, transforms it and ships it to Elasticsearch, You can check it out at: https://github.com/siddontang/go-mysql-elasticsearch

另一个是这个:https://github.com/noplay/python-mysql-复制.

但是请注意，无论您选择哪种方式，在索引 binlog 之前预先创建索引和映射都是一个好习惯.这让您可以更好地控制自己的数据.

Note, however, that whichever you pick, it's a good practice to pre-create your index and mappings before indexing your binlog. That gives you more control over your data.

更新:

这是关于该主题的另一篇有趣的博客文章:如何使用 Logstash 保持 Elasticsearch 与关系数据库同步

Here is another interesting blog article on the subject: How to keep Elasticsearch synchronized with a relational database using Logstash

这篇关于如何将mysql数据实时迁移到ElasticSearch的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！