将MongoDb同步到ElasticSearch [英] Sync MongoDb to ElasticSearch

问题描述

我正在寻找一种在MongoDB中将集合与Elastic Search(ES)同步的方法.目标是将MongoDB用作主要数据源，并将MongoDB用作全文本搜索引擎.(我项目的业务逻辑是用python编写的.)

I am looking for a way to sync collections in MongoDB with Elastic Search (ES). The goal is to have MongoDB as a primary data source and use MongoDB as a full text search engine. (The business logic of my project is written in python).

几种方法可以在线获得.

Several approaches are online available.

• Mongo-connect
• 河插件
• logstash-input-mongodb(logstash插件)看到类似的问题
• 运输车

但是，大多数建议都存在数年之久，我找不到支持当前版本的ES(ES 7.4.0)的任何解决方案.有人在使用这样的构造吗?你有什么建议吗?

However, most of the suggestions are several years old and I could not find any solution that supports the current version of ES (ES 7.4.0). Is anyone using such a construct? Do you have any suggestions?

我考虑过放弃MongoDB作为主要数据源，而只是使用ES进行存储和搜索.尽管我读过ES不应用作主要数据源.

I thought about dropping MongoDB as primary data source and just using ES for storing and searching. Though I have read that ES should not be used as a primary data source.

修改

谢谢@ gurdeep.sabarwal.我遵循了你的方法.但是，我无法将mongodb同步到ES.我的配置如下:

Thank you @gurdeep.sabarwal. I followed your approach. However, I do not manage to sync the mongodb to ES. My configuration looks like this:

input {
    jdbc {
#        jdbc_driver_library => "/usr/share/logstash/mongodb-driver-3.11.0-source.jar"
        jdbc_driver_library => "/usr/share/logstash/mongojdbc1.5.jar"
#        jdbc_driver_library => "/usr/share/logstash/mongodb-driver-3.11.1.jar"

#        jdbc_driver_class => "mongodb.jdbc.MongoDriver"
#        jdbc_driver_class => "Java::com.mongodb.MongoClient"
        jdbc_driver_class => "Java::com.dbschema.MongoJdbcDriver"
        jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
#        jdbc_driver_class => ""

        jdbc_connection_string => "jdbc:mongodb://<myserver>:27017/<mydb>"
        jdbc_user => "user"
        jdbc_password => "pw"
        statement => "db.getCollection('mycollection').find({})"
    }
}

output {
    elasticsearch {
        hosts => ["http://localhost:9200/"]
        index => "myindex"
    }
}

这使我更接近目标.但是，出现以下错误:

This brings me a bit closer to my goal. However, I get the following error:

错误:未加载Java :: com.dbschema.MongoJdbcDriver.您确定已在:jdbc_driver_library中包含正确的jdbc驱动程序吗?
例外:LogStash :: ConfigurationError`

Error: Java::com.dbschema.MongoJdbcDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
Exception: LogStash::ConfigurationError`

因为它不起作用，所以我也尝试了带注释的版本，但没有成功.

Since, it did not work, I tried also the commented version but did not succeed.

推荐答案

1. 下载 https://dbschema.com/jdbc-drivers/MongoDbJdbcDriver.zip
2. 解压缩所有文件并将其复制到路径(〜/logstash-7.4.2/logstash-core/lib/jars/)
3. 修改以下配置文件(mongo-logstash.conf):
4. 运行:〜/logstash-7.4.2/bin/logstash -f mongo-logstash.conf
5. 成功，请尝试！

ps:这是我在stackoverflow中的第一个答案:-)

ps: this is my first answer in stackoverflow :-)

input {
  jdbc{
    # NOT THIS # jdbc_driver_class => "Java::mongodb.jdbc.MongoDriver"
    jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
    jdbc_driver_library => "mongojdbc1.5.jar"
    jdbc_user => "" #no user and pwd
    jdbc_password => ""
    jdbc_connection_string => "jdbc:mongodb://127.0.0.1:27017/db1"
    statement => "db.t1.find()"
  }
}

output {
    #stdout { codec => dots }
    stdout { }
}

这篇关于将MongoDb同步到ElasticSearch的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！