将具有大量数据的 mongo 集合复制到另一个集合? [英] Copying a mongo collection with huge data to another collection?
问题描述
我有一个包含 250 万条数据的 mongo 集合，并且可能会增长到 300 万条.我正在使用 spring 批处理，并试图将该集合复制到另一个集合.我使用的方法如下:
I have a mongo collection(s) with 2.5 million data and that may grow upto 3 million. I am using spring batch and am trying to copy that collection to another collection. Approaches I have used are as follows :
在 tasklet 中，我创建了一个 ProcessBuilder 对象并调用了一个执行 mongo 查询的 shell 脚本.shell脚本内容如下:
Inside a tasklet, I have Created a ProcessBuilder object and called a shell script which executes a mongo query. Content of shell script is as follows :
> mongo $serverURL/$dbName js-file-to-execute.js
// js file contains copy command (db.collection.copyto('newCollection'))
对于较少的数据(<200 k)，它工作正常，但对于 200 万数据，它挂起 mongo 服务器并且作业因套接字异常而失败
For less data (< 200 k) it works fine but for 2 million data it hangs the mongo server and the job got failed with Socket Exception
- 使用 mongo 模板并执行查询
dbMongoTemplate.getDb().getCollection("collection").aggregate(Arrays.asList((DBObject) new BasicDBObject("$out","newCollection")));
这将执行一个 mongo 聚合查询 db.collection.aggregate({$out : "newCollection"})
This executes a mongo aggregate query db.collection.aggregate({$out : "newCollection"})
这也适用于数据较少的集合，但对于较大的数据集，它会一直运行，直到发生套接字超时并最终导致作业失败.
This also worked for collections with less data but for larger data set it keeps running until socket time out occurs and fails the job at the end.
请提出有效的数据复制方法?
Please suggest efficient way to copy data?
推荐答案
要复制这个集合，我建议使用 mongodump/mongoexport
To copy this collection I will sugest using mongodump/mongoexport
mongodump --db databaseName --collection collectionName --out directory-path
mongodump --db databaseName --collection collectionName --out directory-path
然后复制目录directory-path，然后使用
then copy directory directory-path and then restore on target machine using
mongorestore --db databaseName --collection collectionName directory-path
mongorestore --db databaseName --collection collectionName directory-path
这篇关于将具有大量数据的 mongo 集合复制到另一个集合?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
