将MongoDB与ElasticSearch同步 [英] Sync MongoDB with ElasticSearch

问题描述

我想将我的MongoDB数据同步到ElasticSearch，我读了很多关于Elasticsearch River插件和mongo连接器的文章，但是所有这些都已在mongo 4和elasticsearch 7中弃用了！

I want to sync my MongoDB data to ElasticSearch, I read a lot of posts talking about elasticsearch river plugin and mongo connector, but all of them are deprecated for mongo 4 and elasticsearch 7!

由于logstash是专有软件，我想使用它来同步两者...任何人都知道如何做到这一点?

As logstash is a proprietary software I would like to use it to sync both... Anyone knows how it's possible to do it?

推荐答案

您可以将MongoDB和Elasticsearch与Logstash同步；实际上，同步是Logstash的主要应用之一. 安装Logstash 之后，您需要做的所有事情是为您的用例指定一个管道:一个或多个输入源(在您的情况下为MongoDB)和一个或多个输出接收器(在您的情况下为Elasticsearch)，作为配置文件放置(在下面的示例中) Logstash的 pipeline 目录； Logstash负责其余的工作.

You may sync MongoDB and Elasticsearch with Logstash; syncing is, in fact, one of the major applications of Logstash. After installing Logstash, all that you need to do is specify a pipeline for your use case: one or more input sources (MongoDB in your case) and one or more output sinks (Elasticsearch in your case), put as a config file (example follows) inside Logstash's pipeline directory; Logstash takes care of the rest.

Logstash正式提供了许多常用数据源和接收器的插件；这些插件使您仅需少量配置即可从各种来源读取数据或将数据写入各种来源.您只需要找到正确 插件， Elasticsearch 的官方输出插件.其配置非常直观.但是，Logstash不为MongoDB提供任何输入插件.您需要找到第三方. 这个似乎很有希望.

Logstash officially provides plugins for a lot of commonly used data sources and sinks; those plugins let you read data from and write data to various sources with just a few configuration. You just need to find the right plugin, install it, and configure it for your scenario. Logstash has an official output plugin for Elasticsearch and its configuration is pretty intuitive. Logstash, however, doesn't provide any input plugin for MongoDB. You need to find a third party one; this one seems to be pretty promising.

最后，您的管道可能类似于以下内容:

In the end, your pipeline may look something like the following:

input {
  mongodb {
    uri => 'mongodb://10.0.0.30/my-logs?ssl=true'
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'events_'
    batch_size => 5000
  }
}
output {
  stdout {
    codec => rubydebug #outputs the same thing as elasticsearch in stdout to facilitate debugging
  }
  elasticsearch {
    hosts => "localhost:9200"
    index => "target_index"
    document_type => "document_type"
    document_id => "%{id}"
  }
}

这篇关于将MongoDB与ElasticSearch同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！