使用Node.js将许多记录插入Mongodb的正确方法 [英] Correct way to insert many records into Mongodb with Node.js

问题描述

我想知道使用Node.js对Mongodb(尽管可能是任何其他数据库)进行大容量插入的正确方法是什么

I was wondering what is the correct way to do bulk inserts into Mongodb (although could be any other database) with Node.js

我已经编写了以下代码作为示例，尽管我相信它是作为底稿的，因为db.close()可能在所有异步collection.insert调用完成之前运行.

I have written the following code as an example, although I believe it is floored as db.close() may be run before all the asynchronous collection.insert calls have completed.

MongoClient.connect('mongodb://127.0.0.1:27017/test', function (err, db) {
    var i, collection;
    if (err) {
        throw err;
    }
    collection = db.collection('entries');
    for (i = 0; i < entries.length; i++) {
        collection.insert(entries[i].entry);
    }
    db.close();
});

推荐答案

如果您的MongoDB服务器是2.6或更高版本，则最好使用写命令

If your MongoDB server is 2.6 or newer, it would be better to take advantage of using a write commands Bulk API that allow for the execution of bulk insert operations which are simply abstractions on top of the server to make it easy to build bulk operations and thus get perfomance gains with your update over large collections.

批量发送批量插入操作将减少服务器的通信量，从而通过不发送所有单独语句中的所有内容，而是将其分解为可管理的大块来进行服务器承诺，从而执行有效的电汇交易.通过这种方法，在回调中等待响应的时间也更少了.

Sending the bulk insert operations in batches results in less traffic to the server and thus performs efficient wire transactions by not sending everything all in individual statements, but rather breaking up into manageable chunks for server commitment. There is also less time waiting for the response in the callback with this approach.

这些批量操作主要有两种:

These bulk operations come mainly in two flavours:

• 订购的批量操作.这些操作按顺序执行所有操作，并在第一个写入错误时出错.
• 无序批量操作.这些操作并行执行所有操作，并汇总所有错误.无序批量操作不能保证执行顺序.

• Ordered bulk operations. These operations execute all the operation in order and error out on the first write error.
• Unordered bulk operations. These operations execute all the operations in parallel and aggregates up all the errors. Unordered bulk operations do not guarantee order of execution.

请注意，对于版本低于2.6的旧服务器，API将下转换操作.但是，不可能下变频100％，因此在某些极端情况下，它无法正确报告正确的数字.

Note, for older servers than 2.6 the API will downconvert the operations. However it's not possible to downconvert 100% so there might be some edge cases where it cannot correctly report the right numbers.

在您的情况下，您可以像这样批量执行1000次批量API插入操作:

In your case, you could implement the Bulk API insert operation in batches of 1000 like this:

对于MongoDB 3.2 + ，使用 bulkWrite

var MongoClient = require('mongodb').MongoClient;
var url = 'mongodb://localhost:27017/test';
var entries = [ ... ] // a huge array containing the entry objects

var createNewEntries = function(db, entries, callback) {

    // Get the collection and bulk api artefacts
    var collection = db.collection('entries'),          
        bulkUpdateOps = [];    

    entries.forEach(function(doc) {
        bulkUpdateOps.push({ "insertOne": { "document": doc } });

        if (bulkUpdateOps.length === 1000) {
            collection.bulkWrite(bulkUpdateOps).then(function(r) {
                // do something with result
            });
            bulkUpdateOps = [];
        }
    })

    if (bulkUpdateOps.length > 0) {
        collection.bulkWrite(bulkUpdateOps).then(function(r) {
            // do something with result
        });
    }
};

对于MongoDB< 3.2

var MongoClient = require('mongodb').MongoClient;
var url = 'mongodb://localhost:27017/test';
var entries = [ ... ] // a huge array containing the entry objects

var createNewEntries = function(db, entries, callback) {

    // Get the collection and bulk api artefacts
    var collection = db.collection('entries'),          
        bulk = collection.initializeOrderedBulkOp(), // Initialize the Ordered Batch
        counter = 0;    

    // Execute the forEach method, triggers for each entry in the array
    entries.forEach(function(obj) {         

        bulk.insert(obj);           
        counter++;

        if (counter % 1000 == 0 ) {
            // Execute the operation
            bulk.execute(function(err, result) {  
                // re-initialise batch operation           
                bulk = collection.initializeOrderedBulkOp();
                callback();
            });
        }
    });             

    if (counter % 1000 != 0 ){
        bulk.execute(function(err, result) {
            // do something with result 
            callback();             
        }); 
    } 
};

调用 createNewEntries() 函数.

Call the createNewEntries() function.

MongoClient.connect(url, function(err, db) {
    createNewEntries(db, entries, function() {
        db.close();
    });
});

这篇关于使用Node.js将许多记录插入Mongodb的正确方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！