使用mongoose在MongoDB中批量插入 [英] Bulk insert in MongoDB using mongoose

问题描述

我目前在Mongodb中有一个集合说Collection1。
我有以下需要插入MongoDB的对象数组。我正在使用Mongoose API。现在，我正在迭代数组并将它们中的每一个插入到mongo中。
现在可以，但是当数据太大时会出现问题。
我需要一种将数据批量插入MongoDB而不重复的方法。
我不知道该怎么做。我在Mongoose找不到批量选项。

I currently have a collection in Mongodb say "Collection1". I have the following array of objects that need to be into inserted into MongoDB. I am using Mongoose API. For now, I am iterating through the array and inserting each of them into mongo. This is ok for now, but will be a problem when the data is too big. I need a way of inserting the data in bulk into MongoDB without repetition. I am not sure how to do this. I could not find a bulk option in Mongoose.

我的代码在下面

myData = [Obj1,Obj2,Obj3.......]

myData.forEach(function(ele){
      //console.log(ele)
     saveToMongo(ele);
    });
function saveToMongo(obj){
    (new Collection1(obj)).save(function (err, response) {
          if (err) {
             // console.log('Error while inserting: ' + obj.name + " " +err);
          } else {
            // console.log('Data successfully inserted');
          }
      });

      return Collection1(obj);
  }

推荐答案

您可能想要使用 insertMany（） 4.4.X 和更高版本，这里使用 Model.collection.insertMany（） 引擎盖下，驱动程序可能会为您处理并行化> = 1000 docs。

You might want to use the insertMany() method here if you're using the latest Mongoose version 4.4.X and greater, which essentially uses Model.collection.insertMany() under the hood and the driver might handle parallelizing >= 1000 docs for you.

myData = [Obj1, Obj2, Obj3.......];
Collection1.insertMany(myData, function(error, docs) {});

或使用Promises更好地处理错误

or using Promises for better error handling

Collection1.insertMany(myData)
    .then(function(docs) {
         // do something with docs
    })
    .catch(function(err) {
        // error handling here
    });

它的工作原理是创建一堆文件，调用 .validate（）并行调用它们，然后调用底层驱动程序的 toObject（{virtuals：false}）的结果，rel =noreferrer> insertMany（） ;每个文档的。
虽然 insertMany（） 不触发预保存挂钩，它具有更好的性能，因为它只对服务器进行1次往返，而不是每个文档1次。

It works by creating a bunch of documents, calls .validate() on them in parallel, and then calls the underlying driver's insertMany() on the result of toObject({ virtuals: false }); of each doc. Although insertMany() doesn't trigger pre-save hooks, it has better performance because it only makes 1 round-trip to the server rather than 1 for each document.

对于支持MongoDB的Mongoose版本 ~3.8.8，~3.8.22,4.x 服务器> = 2.6.x ，您可以使用 批量API ，如下所示

For Mongoose versions ~3.8.8, ~3.8.22, 4.x which support MongoDB Server >=2.6.x, you could use the Bulk API as follows

var bulk = Collection1.collection.initializeOrderedBulkOp(),
    counter = 0;

myData.forEach(function(doc) {
    bulk.insert(doc);

    counter++;
    if (counter % 500 == 0) {
        bulk.execute(function(err, r) {
           // do something with the result
           bulk = Collection1.collection.initializeOrderedBulkOp();
           counter = 0;
        });
    }
});

// Catch any docs in the queue under or over the 500's
if (counter > 0) {
    bulk.execute(function(err,result) {
       // do something with the result here
    });
}

这篇关于使用mongoose在MongoDB中批量插入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！