处理Mongodb连接的正确方法是什么? [英] What is the right way to deal with Mongodb connections?

问题描述

我尝试使用10gen的本机node.js驱动器将mongodb(2.2.2)与node.js一起使用.

I try node.js with mongodb (2.2.2) together using the native node.js drive by 10gen.

起初一切都很顺利.但是当涉及到并发基准测试部分时，会发生很多错误.频繁进行1000次并发连接/关闭可能会导致mongodb拒绝任何进一步的请求，并显示以下错误:

At first everything went well. But when coming to the concurrency benchmarking part, a lot of errors occured. Frequent connect/close with 1000 concurrencies may cause mongodb reject any further requests with error like:

Error: failed to connect to [localhost:27017]

Error: Could not locate any valid servers in initial seed list

Error: no primary server found in set

此外，如果许多客户端在没有显式关闭的情况下关闭，则mongodb将花费几分钟的时间来检测并关闭它们.这也将导致类似的连接问题. (使用/var/log/mongodb/mongodb.log检查连接状态)

Also, if a lot of clients shutdown without explicit close, it'll take mongodb minutes to detect and close them. Which will also cause similar connection problems. (Using /var/log/mongodb/mongodb.log to check the connection status)

我已经尝试了很多.根据手册，mongodb没有连接限制，但是 poolSize 选项似乎对我没有影响.

I have tried a lot. According to the manual, mongodb don't have connection limitation, but poolSize option seems to have no effects to me.

由于我只在node-mongodb-native模块中使用过它，所以我不太确定最终导致问题的原因.其他语言和驱动程序的性能如何?

As I have only worked with it in node-mongodb-native module, I'm not very sure what eventually caused the problem. What about the performance in other other languages and drivers?

PS:目前，我想出了使用自我维护池的唯一解决方案，但是使用它不能解决副本集的问题.根据我的测试，副本集似乎比独立的mongodb需要更少的连接.但是不知道为什么会这样.

PS: Currently, using self maintained pool is the only solution I figured out, but using it can not can not solve the problem with replica set. According to my test, replica set seems take much less connections then standalone mongodb. But have no idea why this happens.

并发测试代码:

var MongoClient = require('mongodb').MongoClient;

var uri = "mongodb://192.168.0.123:27017,192.168.0.124:27017/test";

for (var i = 0; i < 1000; i++) {
    MongoClient.connect(uri, {
        server: {
            socketOptions: {
                connectTimeoutMS: 3000
            }
        },
    }, function (err, db) {
        if (err) {
            console.log('error: ', err);
        } else {
            var col = db.collection('test');
            col.insert({abc:1}, function (err, result) {
                if (err) {
                    console.log('insert error: ', err);
                } else {
                    console.log('success: ', result);
                }
                db.close()
            })
        }
    })
}

通用池解决方案:

var MongoClient = require('mongodb').MongoClient;
var poolModule = require('generic-pool');

var uri = "mongodb://localhost/test";

var read_pool = poolModule.Pool({
    name     : 'redis_offer_payment_reader',
    create   : function(callback) {
        MongoClient.connect(uri, {}, function (err, db) {
            if (err) {
                callback(err);
            } else {
                callback(null, db);
            }
        });
    },
    destroy  : function(client) { client.close(); },
    max      : 400,
    // optional. if you set this, make sure to drain() (see step 3)
    min      : 200, 
    // specifies how long a resource can stay idle in pool before being removed
    idleTimeoutMillis : 30000,
    // if true, logs via console.log - can also be a function
    log : false 
});


var size = [];
for (var i = 0; i < 100000; i++) {
    size.push(i);
}

size.forEach(function () {
    read_pool.acquire(function (err, db) {
        if (err) {
            console.log('error: ', err);
        } else {
            var col = db.collection('test');
            col.insert({abc:1}, function (err, result) {
                if (err) {
                    console.log('insert error: ', err);
                } else {
                    //console.log('success: ', result);
                }
                read_pool.release(db);
            })
        }
    })
})

推荐答案

由于Node.js是单线程的，因此您不应在每个请求上打开和关闭连接(就像在其他多线程环境中那样).

Since Node.js is single threaded you shouldn't be opening and closing the connection on each request (like you would do in other multi-threaded environments.)

这是写MongoDB node.js客户端模块的人的话:

This is a quote from the person that wrote the MongoDB node.js client module:

当您的应用程序启动并重新使用时，您只需打开一次do MongoClient.connect db对象.每个.connect都不是一个单例连接池. 创建一个新的连接池.因此，请一次打开并在所有范围内重用 要求." -christkv https://groups.google.com/forum/# ！msg/node-mongodb-native/mSGnnuG8C1o/Hiaqvdu1bWoJ

"You open do MongoClient.connect once when your app boots up and reuse the db object. It's not a singleton connection pool each .connect creates a new connection pool. So open it once an[d] reuse across all requests." - christkv https://groups.google.com/forum/#!msg/node-mongodb-native/mSGnnuG8C1o/Hiaqvdu1bWoJ

这篇关于处理Mongodb连接的正确方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！