如何在Mongoskin中使用无序批量插入? [英] How to use unordered bulk inserting with Mongoskin?

问题描述

我无法使用Mongoskin在Node上执行批量插入(MongoDB 2.6 +).

var dbURI = urigoeshere;
var db = mongo.db(dbURI, {safe:true});
var bulk = db.collection('collection').initializeUnorderedBulkOp();

for (var i = 0; i < 200000; i++) {
    bulk.insert({number: i}, function() {
        console.log('bulk inserting: ', i);
    });
}

bulk.execute(function(err, result) {
    res.json('send response statement');
}); 

上面的代码给出了以下警告/错误:

(node) warning: possible EventEmitter memory leak detected. 51 listeners added. Use emitter.setMaxListeners() to increase limit.
TypeError: Object #<SkinClass> has no method 'execute'
(node) warning: possible EventEmitter memory leak detected. 51 listeners added. Use emitter.setMaxListeners() to increase limit.
TypeError: Object #<SkinClass> has no method 'execute'

是否可以使用Mongoskin执行无序的批量操作?如果是这样，我在做什么错了?

解决方案

您可以执行此操作，但是您需要更改调用约定以执行此操作，因为只有回调"形式实际上会返回一个集合对象，可以从中调用.initializeUnorderedBulkOp()方法.您认为这种用法的使用方式也存在一些差异:

 var dbURI = urigoeshere;
var db = mongo.db(dbURI, {safe:true});
db.collection('collection',function(err,collection) {
    var bulk = collection.initializeUnorderedBulkOp();
    count = 0;

    for (var i = 0; i < 200000; i++) {
        bulk.insert({number: i});
        count++;

        if ( count % 1000 == 0 )
            bulk.execute(function(err,result) {
               // maybe do something with results
               bulk = collection.initializeUnorderedBulkOp(); // reset after execute
            });      

    });

    // If your loop was not a round divisor of 1000
    if ( count % 1000 != 0 )
        bulk.execute(function(err,result) {
          // maybe do something here
        });
});
 

因此，实际的批量"方法本身不需要回调并且可以完全按照文档.理解是.execute()，它实际上将语句发送到服务器.

虽然驱动程序会为您解决一些问题，但在调用execute之前先排队太多操作可能不是一个好主意.这基本上是在内存中建立的，尽管驱动程序一次只能发送1000个批次(这是服务器的限制，并且整个批次都在16MB以下)，但您可能希望在此处进行更多控制，至少限制内存使用.

这就是所示模数测试的重点，但是，如果用于构建操作的内存和可能非常大的响应对象对您来说不是问题，那么您可以继续排队操作并调用.execute(). /p>

响应"的格式与 BulkWriteResult .

I'm having trouble using Mongoskin to perform bulk inserting (MongoDB 2.6+) on Node.

var dbURI = urigoeshere;
var db = mongo.db(dbURI, {safe:true});
var bulk = db.collection('collection').initializeUnorderedBulkOp();

for (var i = 0; i < 200000; i++) {
    bulk.insert({number: i}, function() {
        console.log('bulk inserting: ', i);
    });
}

bulk.execute(function(err, result) {
    res.json('send response statement');
}); 

The above code gives the following warnings/errors:

(node) warning: possible EventEmitter memory leak detected. 51 listeners added. Use emitter.setMaxListeners() to increase limit.
TypeError: Object #<SkinClass> has no method 'execute'
(node) warning: possible EventEmitter memory leak detected. 51 listeners added. Use emitter.setMaxListeners() to increase limit.
TypeError: Object #<SkinClass> has no method 'execute'

Is it possible to use Mongoskin to perform unordered bulk operations? If so, what am I doing wrong?

解决方案

You can do it but you need to change your calling conventions to do this as only the "callback" form will actually return a collection object from which the .initializeUnorderedBulkOp() method can be called. There are also some usage differences to how you think this works:

var dbURI = urigoeshere;
var db = mongo.db(dbURI, {safe:true});
db.collection('collection',function(err,collection) {
    var bulk = collection.initializeUnorderedBulkOp();
    count = 0;

    for (var i = 0; i < 200000; i++) {
        bulk.insert({number: i});
        count++;

        if ( count % 1000 == 0 )
            bulk.execute(function(err,result) {
               // maybe do something with results
               bulk = collection.initializeUnorderedBulkOp(); // reset after execute
            });      

    });

    // If your loop was not a round divisor of 1000
    if ( count % 1000 != 0 )
        bulk.execute(function(err,result) {
          // maybe do something here
        });
});

So the actual "Bulk" methods themselves don't require callbacks and work exactly as shown in the documentation. The exeception is .execute() which actually sends the statements to the server.

While the driver will sort this out for you somewhat, it probably is not a great idea to queue up too many operations before calling execute. This basically builds up in memory, and though the driver will only send in batches of 1000 at a time ( this is a server limit as well as the complete batch being under 16MB ), you probably want a little more control here, at least to limit memory usage.

That is the point of the modulo tests as shown, but if memory for building the operations and a possibly really large response object are not a problem for you then you can just keep queuing up operations and call .execute() once.

The "response" is in the same format as given in the documentation for BulkWriteResult.

这篇关于如何在Mongoskin中使用无序批量插入?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！