插入10000个文档时超出最大调用堆栈大小 [英] Maximum call stack size exceeded on insert 10000 documents
问题描述
这是正在运行的代码，该代码返回超出范围最大调用堆栈大小"错误. //使用node.js在mongodb上插入10000个值
This is the code am running which returns the Range Maximum call stack size exceeded error. // to insert 10000 values on to mongodb using node.js
var MongoClient = require('mongodb').MongoClient;
var mongoServer = require('mongodb').Server;
var serverOptions = {
'auto_reconnect': true,
'poolSize': 100
};
var i=0;
var async =require('async');
var mongoClient = new MongoClient(new mongoServer('localhost', 27017, serverOptions));
var db = mongoClient.db('test');
var collection = db.collection('new_file_test');
mongoClient.open(function (err, mongoClient)
{
if(err){console.log(err)};
function start(i,call)
{
if(i<10000) {
call(start);
}
}
function pass(callback)
{
Insert(save);
i++;
callback(i,pass);
}
start(i,pass);
});
function Insert(callback) {
console.log("Inserting" );
var doc={
'trip_paramid':i,
'tripid':'116',
'lattitude':'12.8929183',
'longitude':'77.63627',
'speed':'2',
'heading':'0',
'altitude':'80469',
'address':'qwertyasdfgxcvbn',
'engine_status':'Normal',
'oil_pressure': '83.12',
'water_temp': '28',
'fuel_content':'0',
'brake':'Normal',
'creation_time':'2013-08-31 23:22:17',
'brakelight_status':'Normal',
'battery_status':'12.68',
'event_code':'8',
'dbinsert_time':'2013-08-31 23:24:59',
'gsm_status':'-51',
'cell_id':'45',
'vehicle_id':'123456',
'distance':'0'}
callback(doc);
}
function save(doc)
{
collection.insert(doc, function(err)
{
if (err)
{
console.log('Error occured');
}
else
console.log("Saved");
});
}
如果条件是插入1000行，则可以正常工作，并且仅当条件超过10000时，才会引发错误.
If the condition is to insert 1000 rows it works fine and the error throws only when the condition goes beyond 10000.
推荐答案
问题来自您创建的递归循环:
function start(i, call) {
if (i < 10000) {
call(start);
}
}
function pass(callback) {
Insert(save);
i++;
callback(i, pass);
}
start(i, pass);
您应该将其更改为类似的内容:
for (var i = 0; i < 10000; i++) {
Insert(save);
}
简化代码，您会得到以下提示:
Simplifying your code you have this:
var i = 0;
function pass() {
if (i < 10000) {
Insert(save);
pass(i);
}
i++;
}
pass();
问题出在您递归调用此函数的部分，并且由于javascript没有尾部递归消除，因此
The problem comes from the part that you are calling this function recursively, and since javascript doesn't have tail recursion elimination, the callstack keeps growing. V8(nodejs javascript engine) has it's limits, the callstack once reached to the maximum defined size the error will be thrown.
您也可以查看以下问题以获取更多信息:
You can also have look at the following questions for more information:
- Maximum call stack size exceeded error
- JavaScript recursion: Maximum call stack size exceeded
这都是关于解决Maximum call stack size exceeded错误的问题.但是10000看起来是一个巨大的数字.我只是运行它，在我的机器上花了大约3秒钟，使用monk完成了循环.使用mongo shell花费了大约1秒钟.如果您正在运行服务器，则在循环运行时，您的应用程序将无响应.
This is all about fixing Maximum call stack size exceeded error. But 10000 looks like a huge number. I just ran that and it took about 3 seconds on my machine, to finish the loop using monk. Using mongo shell it took about 1 second. If you are running a server, when the loop is running your application is unresponsive.
我建议改为批量插入，然后使用节点的setImmediate函数进行计划在待处理的I/O事件(例如处理新的Web请求)之后运行的下一批:
I suggest instead, insert in batches, and use node's setImmediate function to schedule the next batch to be run after pending I/O events(like handling new web requests):
function insert10000(i) {
insert100();
i++;
if (i < 100) {
setImmidiate(insert10000, i);
}
}
function insert100() {
for (var i = 0; i < 100; i++) {
Insert(save);
}
}
由于我们涉及批量插入调用的主题，所以collection.insert方法支持一系列文档，而不仅仅是要插入的文档.
And since we came on the topic of batching insert calls, collection.insert method, supports an array of documents instead of just one to be inserted.
因此，当我们当前具有以下内容时:
So when we currently have something like following:
collection.insert(doc1);
collection.insert(doc2);
可以更改为此:
collection.insert([doc1, doc2]);
那实际上更快.因此，您可以将代码更改为此:
And that actually is faster. So you can change the code to this:
function insert10000(i) {
insert100(i);
i++;
if (i < 100) {
setImmediate(insert10000, i);
}
}
function insert100(i) {
var docs = [];
for (var l = i + 1000; i < l; i++) {
docs.push({
'trip_paramid':i,
'tripid':'116',
'lattitude':'12.8929183',
'longitude':'77.63627',
'speed':'2',
'heading':'0',
'altitude':'80469',
'address':'qwertyasdfgxcvbn',
'engine_status':'Normal',
'oil_pressure': '83.12',
'water_temp': '28',
'fuel_content':'0',
'brake':'Normal',
'creation_time':'2013-08-31 23:22:17',
'brakelight_status':'Normal',
'battery_status':'12.68',
'event_code':'8',
'dbinsert_time':'2013-08-31 23:24:59',
'gsm_status':'-51',
'cell_id':'45',
'vehicle_id':'123456',
'distance':'0'
});
}
collection.insert(docs, function(err) {
if (err) {
console.log('Error occurred', err);
}
});
}
我对此进行了测量，它比原始情况要快两倍.
I measured this, it was faster twice faster than the original case.
这篇关于插入10000个文档时超出最大调用堆栈大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
