MongoDB 4.x实时同步到ElasticSearch 6.x + [英] MongoDB 4.x Real Time Sync to ElasticSearch 6.x +
问题描述
我正在尝试找到一种简单的方法来将mongoDB 4.x中的数据同步到elasticsearch6.x.我的用例是用于Elasticsearch支持的部分文本搜索，但mongodb不支持.MongoDB是我的应用程序的主要数据库.
我发现的所有解决方案似乎都已过时，仅支持旧版本的mongoDB/elasticsearch.其中包括mongodb-connector，mongodb river
什么是最好的工具，以便对mongoDB中的数据所做的任何更改(CRUD)自动同步到elasticsearch?
有一个名为Monstache的工具，用于将MOngoDB数据实时迁移到ElasticSearch.该工具支持最新的MongoDB版本.
其用Go编写的同步守护程序，该守护程序连续将您的MongoDB集合索引到Elasticsearch中.Monstache使您能够使用Elasticsearch对MongoDB数据进行复杂的搜索和聚合，并轻松构建实时Kibana可视化效果和仪表板.
它是一个go守护进程，可将mongodb实时同步到elasticsearch.它是Monstache.可在以下网站上找到它:
I'm trying to find an easy way to sync data in mongoDB 4.x, to elasticsearch 6.x . My use case is for partial text search that is supported by elasticsearch but no supported by mongodb. MongoDB is the primary database for my applications.
All solutions i found seem outdated and only support older version of mongoDB / elasticsearch. These include mongodb-connector, mongodb river
What is the best tool to use so that any changes (CRUD) to data in mongoDB is automatically synced to elasticsearch?
There is tool called Monstache for migrating teh MOngoDB data to ElasticSearch in real time. This tool supports the latest MongoDB Version.
Its sync daemon written in Go that continously indexes your MongoDB collections into Elasticsearch. Monstache gives you the ability to use Elasticsearch to do complex searches and aggregations of your MongoDB data and easily build realtime Kibana visualizations and dashboards.
Its a go daemon that syncs mongodb to elasticsearch in realtime. Its the Monstache. Its available at : Monstache The monstache required mongodb to run in replication mode.
Below the initial setup to configure and use it.
Step 1:
C:\Program Files\MongoDB\Server\4.0\bin>mongod --smallfiles --oplogSize 50 --replSet test
Step 2 :
C:\Program Files\MongoDB\Server\4.0\bin>mongo
C:\Program Files\MongoDB\Server\4.0\bin>mongo
MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 4.0.2
Server has startup warnings:
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten]
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten]
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** Start the server with --bind_ip <address> to specify which IP
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten]
MongoDB Enterprise test:PRIMARY>
Step 3 : Verify the replication.
MongoDB Enterprise test:PRIMARY> rs.status();
{
"set" : "test",
"date" : ISODate("2019-01-18T11:39:00.380Z"),
"myState" : 1,
"term" : NumberLong(2),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"readConcernMajorityOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"appliedOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"durableOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
}
},
"lastStableCheckpointTimestamp" : Timestamp(1547811517, 1),
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 736,
"optime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2019-01-18T11:38:57Z"),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1547810805, 1),
"electionDate" : ISODate("2019-01-18T11:26:45Z"),
"configVersion" : 1,
"self" : true,
"lastHeartbeatMessage" : ""
}
],
"ok" : 1,
"operationTime" : Timestamp(1547811537, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1547811537, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
MongoDB Enterprise test:PRIMARY>
Step 4.
Download the "https://github.com/rwynn/monstache/releases".
Unzip the download and adjust your PATH variable to include the path to the folder for your platform.
GO to cmd and type "monstache -v"
# 4.13.1
Monstache uses the TOML format for its configuration. Configure the file for migration named config.toml
Step 5.
My config.toml -->
mongo-url = "mongodb://127.0.0.1:27017/?replicaSet=test"
elasticsearch-urls = ["http://localhost:9200"]
direct-read-namespaces = [ "admin.users" ]
gzip = true
stats = true
index-stats = true
elasticsearch-max-conns = 4
elasticsearch-max-seconds = 5
elasticsearch-max-bytes = 8000000
dropped-collections = false
dropped-databases = false
resume = true
resume-write-unsafe = true
resume-name = "default"
index-files = false
file-highlighting = false
verbose = true
exit-after-direct-reads = false
index-as-update=true
index-oplog-time=true
Step 6.
D:\15-1-19>monstache -f config.toml
这篇关于MongoDB 4.x实时同步到ElasticSearch 6.x +的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
