如何使用NodeJS在mongodb中执行批量插入 [英] How to perform mass inserts into mongodb using NodeJS

问题描述

我必须使用nodejs在mongodb中插入大约10,00000个文档.

I Have to Insert about 10,00000 documents in mongodb using nodejs.

在最终将它们插入到mongodb中之前，我正在使用for循环将这些文档存储到数组中.

I'm generating these documents using a for loop storing them into an array before finally inserting them into mongodb.

var codeArray = new Array();
for (var i = 0; i<1000000; i++){
    var token = strNpm.generate();
    var now = moment().format('YYYYMMDD hhmmss');
    var doc1 = {id:token,
        Discount_strId:"pending",
        Promotion_strCode:token,
        Promotion_strStatus:"I",
        Promotion_dtmGeneratedDate:now,
        User_strLogin:"test",
        Promotion_strMode:"S",
        Promotion_dtmValidFrom:"pending",
        Promotion_dtmValidTill:"pending",
        LastModified_dtmStamp:now
    };
    codeArray.push(doc1);
    db.collection('ClPromoCodeMaster').insert(codeArray, function (err, result) {
    if (err){
        console.log(err);
    }else{
        console.log('Inserted Records - ', result.ops.length);
    }
});

我面临的问题是mongo的插入限制为16mb，因此我无法一次插入整个数组. 请提出最佳解决方案.

The problem I'm facing is mongo has an inserting limit of 16mb, so I can't insert the entire array at once. Please suggest most optimum solutions.

推荐答案

主要问题在于请求的大小，而不是文档的大小，但这是相同的限制. 批量操作和带有

The main problem is in the request size and not the document size, but it amounts to the same limitation. Bulk operations and the async library with async.whilst will handle this:

var bulk = db.collection('ClPromoCodeMaster').initializeOrderedBulkOp(),
    i = 0;

async.whilst(
  function() { return i < 1000000; },
  function(callback) {
    var token = strNpm.generate();
    var now = moment().format('YYYYMMDD hhmmss');
    var doc = {
      id:token,
      Discount_strId:"pending",
      Promotion_strCode:token,
      Promotion_strStatus:"I",
      Promotion_dtmGeneratedDate:now,
      User_strLogin:"test",
      Promotion_strMode:"S",
      Promotion_dtmValidFrom:"pending",
      Promotion_dtmValidTill:"pending",
      LastModified_dtmStamp:now
    };

    bulk.insert(doc);
    i++;

    // Drain every 1000
    if ( i % 1000 == 0 ) {
      bulk.execute(function(err,response){
        bulk = db.collection('ClPromoCodeMaster').initializeOrderedBulkOp();
        callback(err);
      });
    } else {
        callback();
    }

  },
  function(err) {
    if (err) throw err;
    console.log("done");
  }
);

我应该注意，无论对批量操作有内部限制，每批只能进行1000次操作.您可以提交更大的文件，但是驱动程序将分解这些文件，并且仍以1000为批次进行提交.

I should note that regardless there is an internal limit on bulk operations to 1000 operations per batch. You can submit in larger sizes, but the driver is just going to break these up and still submit in batches of 1000.

1000是一个不错的数字，因为它已经与请求的处理方式保持一致，并且在耗尽请求队列并将其发送到请求之前已保留在内存中的合理数量.服务器.

The 1000 is a good number to stay at though, since it is already in line with how the request will be handled, as well as being a reasonable number of things to hold in memory before draining the request queue and sending to the server.

这篇关于如何使用NodeJS在mongodb中执行批量插入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！