在插入大量数据的同时关闭node.js中的mongodb连接 [英] closing mongodb connection in node.js while inserting lot of data

问题描述

我正在尝试编写一个程序来解析iis日志数据并将其插入到mongodb中.文件不是很大，大约有600行.与.net和sql server相比，尝试说服我的管理nodejs和mongodb更好:).
看看下面的nodejs代码.逻辑:我解析每一行并将其转换为json，然后将保存内容插入db中.我正在使用Mongonative驱动程序.
问题:即使在将所有行插入到Db中之前，db连接也会关闭.
我看到日志文件有6000行，但是db中的记录数仅为arnd4000.我知道这是nodejs的异步特性，以此方式(确定是否已插入所有行之后)如何以更确定的方式关闭连接?

I am trying write a program to parse and insert iis logs data in to mongodb. The files aren't that huge it's around 600 lines. Trying to convince my management nodejs and mongodb is better for this compared to .net and sql server :).
Have a look at the below code in nodejs. Logic: I parse every line and convert into json and insert the save in db. i am using mongonative driver.
Issue : The db connection gets closed even before all lines are inserted into the Db.
I see the log file has 6000 lines, but num of records in db is only arnd 4000. I understand it's nodejs's async characteristic, in this how can i close the connection in more deterministic way (after checking if all lines got inserted)?

var MongoClient = require('mongodb').MongoClient;
var mongoServer = require('mongodb').Server;
var serverOptions = {
    'auto_reconnect': true,
    'poolSize': 5
};

var fs = require('fs');
var readline = require('readline');

var rd = readline.createInterface({
    input: fs.createReadStream('C:/logs/Advisor_Metrics/UI/P20VMADVSRUI01/u_ex130904.log'),
    output: process.stdout,
    terminal: false
});
var mongoClient = new MongoClient(new mongoServer('localhost', 27017, serverOptions));
var db = mongoClient.db('test');
var collection = db.collection('new_file_test');
var cntr = 0;
mongoClient.open(function (err, mongoClient) {
    console.log(err);
    if (mongoClient)
    {        
        rd.on('line', function (line) {
            if (line.indexOf('#') == -1) {
                var lineSplit = line.split(' ');
                var data =
                {
                    d: lineSplit[0],
                    t: lineSplit[1],
                    sip: lineSplit[2],
                    met: lineSplit[3],
                    uri: lineSplit[4],
                    cip: lineSplit[8],
                    cua: lineSplit[9],
                    stat: lineSplit[10],
                    tt: lineSplit[13]
                };

                collection.insert(data, function (err, docs) {
                    console.log('closing connection');
                    //db.close();
                });
            }
        });
    }
})
rd.on('close', function () {
    db.close();
});

Sol 1:一种解决方案是解析json对象并添加到数组中，然后将该数组添加到mongodb中.我不想这样做，因为那样想将整个巨大的日志文件解析到内存中！，还有其他解决方案吗?

推荐答案

我100％确信，但是据我所知，您正在同步插入数据.我的意思是，一旦您获得一行，就尝试将其插入，而不必等待结果.尝试使用另一种方法:

I'm 100% sure but as far as I can see you are inserting data synchronous. I mean once you get a line you try to insert it and don't wait for the result. Try using another approach:

• 读取所有行并将其存储在数组中
• 开始异步逐行插入数据

类似的东西:

var lines = [];
var readAllLines = function(callback) {
    // store every line inside lines array
    // and call the callback at the end
    callback();
}
var storeInDb = function(callback) {
    if(lines.length === 0) {
        callback();
        return;
    }
    var line = lines.shift();
    collection.insert(line, function (err, docs) {
        storeInDb(callback);
    });
}

mongoClient.open(function (err, mongoClient) {
    console.log(err);
    if (mongoClient) {
        readAllLines(function() {
            storeInDb(function() {
                // lines are inserted
                // close the db connection
            })
        });
    }
});

这篇关于在插入大量数据的同时关闭node.js中的mongodb连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！