从MySQL到弹性搜索的数据实时迁移？ [英] Real time migration of data from MySQL to elasticsearch?

问题描述

我在MySQL中存在大量数据，不同的数据库和各自的表格。他们都相互关联。但是当我必须在数据中进行分析时，我必须创建不同的脚本，结合数据，合并并显示给我，但是这也需要很多时间和精力。我喜欢弹性搜索的速度和可视化的数据通过kibana，因此我决定将我的整个MySQL数据实时移动到弹性搜索，保持数据在MySQL也。但是我想要一个可扩展的策略，并且将数据迁移到弹性搜索的过程。

建议最好的工具或方法来完成这项工作。

谢谢。

解决方案

1. 在弹性搜索2.x之前，您可以编写自己的Elasticsearch _river插件，您可以将其安装到弹性搜索。您可以控制您希望通过_river将您所编写的脚本的数据加入您的脚本的频率（注意：这不是真正的推荐）。




2. 您还可以使用您最喜欢的排队消息代理工具，例如 ActiveMQ 将您的数据推送到弹性搜索




3. 如果要完全控制以满足实时迁移数据的需要，您还可以编写一个简单的应用程序，利用弹性搜索REST终点，只需写入它通过REST。您甚至可以批量POST




4. 使用任何弹性搜索工具，如 beat ， logstash ，这些都很出色几乎任何类型的数据进入弹性搜索

对于其他替代方案，将您的数据转移到平面文件，或者如果你想维持关系见此贴 here

I have tons of data present in MySQL in form of different database, and their respective tables. They all are related to each other. But when I have to do analysis in data, I have to create different scripts, that combine data, merge it and show me as a result, but this takes a lot of time, and effort too. I love elasticsearch for its speed and visualization of data via kibana, therefore I have decided to move my entire MySQL data in real time to elasticsearch, keeping data in MySQL too. But I want a scalable strategy, and process that migrates that data to elasticsearch.

Suggest the best tool, or methods to do the job.

Thank you.

解决方案

1. Prior to Elasticsearch 2.x you could write your own Elasticsearch _river plugin that you can install into elasticsearch. You can control how often you want this said data you've munged with your scripts to be pulled in by the _river (Note: this is not truly recommended).

2. You may also use your favourite Queuing Message Broker tool such as ActiveMQ to push your data into elasticsearch

3. If you want full control to meet your need for real time migration of data you may also write a simple app that makes use of elasticsearch REST end point, by simply writing to it via REST. You can even do bulk POST

4. Make use of any of the elasticsearch tools such as beat, logstash that are great at shipping almost any type of data into elasticsearch

For other alternatives of munging your data to a flat file, or if you want to maintain relationships see this post here

这篇关于从MySQL到弹性搜索的数据实时迁移？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！