使用 mongoose 在 MongoDB 中批量插入 [英] Bulk insert in MongoDB using mongoose

问题描述

我目前在 Mongodb 中有一个集合，比如Collection1".我有以下需要插入到 MongoDB 中的对象数组.我正在使用猫鼬 API.现在，我正在遍历数组并将它们中的每一个插入到 mongo 中.这暂时没问题，但是当数据太大时会出现问题.我需要一种将数据批量插入 MongoDB 而不重复的方法.我不知道该怎么做.我在 Mongoose 中找不到批量选项.

I currently have a collection in Mongodb say "Collection1". I have the following array of objects that need to be into inserted into MongoDB. I am using Mongoose API. For now, I am iterating through the array and inserting each of them into mongo. This is ok for now, but will be a problem when the data is too big. I need a way of inserting the data in bulk into MongoDB without repetition. I am not sure how to do this. I could not find a bulk option in Mongoose.

我的代码如下

myData = [Obj1,Obj2,Obj3.......]

myData.forEach(function(ele){
      //console.log(ele)
     saveToMongo(ele);
    });
function saveToMongo(obj){
    (new Collection1(obj)).save(function (err, response) {
          if (err) {
             // console.log('Error while inserting: ' + obj.name + " " +err);
          } else {
            // console.log('Data successfully inserted');
          }
      });

      return Collection1(obj);
  }

推荐答案

您可能想要使用 insertMany() 方法，如果您使用的是最新的 Mongoose 版本 4.4.X 及更高版本，它基本上使用Model.collection.insertMany() 在幕后，驱动程序可能会为您处理并行化 >= 1000 文档.

You might want to use the insertMany() method here if you're using the latest Mongoose version 4.4.X and greater, which essentially uses Model.collection.insertMany() under the hood and the driver might handle parallelizing >= 1000 docs for you.

myData = [Obj1, Obj2, Obj3.......];
Collection1.insertMany(myData, function(error, docs) {});

或者使用 Promises 来更好地处理错误

or using Promises for better error handling

Collection1.insertMany(myData)
    .then(function(docs) {
         // do something with docs
    })
    .catch(function(err) {
        // error handling here
    });

它的工作原理是创建一堆文档，对它们并行调用.validate()，然后调用底层驱动程序的insertMany() 在 的结果上toObject({ virtuals: false }); 每个文档.虽然 insertMany() 不触发预保存钩子，它具有更好的性能，因为它只对服务器进行 1 次往返，而不是对每个文档进行 1 次往返.

It works by creating a bunch of documents, calls .validate() on them in parallel, and then calls the underlying driver's insertMany() on the result of toObject({ virtuals: false }); of each doc. Although insertMany() doesn't trigger pre-save hooks, it has better performance because it only makes 1 round-trip to the server rather than 1 for each document.

对于支持 MongoDB Server >=2.6.x 的 Mongoose 版本 ~3.8.8, ~3.8.22, 4.x，您可以使用 Bulk API 如下

For Mongoose versions ~3.8.8, ~3.8.22, 4.x which support MongoDB Server >=2.6.x, you could use the Bulk API as follows

var bulk = Collection1.collection.initializeOrderedBulkOp(),
    counter = 0;

myData.forEach(function(doc) {
    bulk.insert(doc);

    counter++;
    if (counter % 500 == 0) {
        bulk.execute(function(err, r) {
           // do something with the result
           bulk = Collection1.collection.initializeOrderedBulkOp();
           counter = 0;
        });
    }
});

// Catch any docs in the queue under or over the 500's
if (counter > 0) {
    bulk.execute(function(err,result) {
       // do something with the result here
    });
}

这篇关于使用 mongoose 在 MongoDB 中批量插入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！