将数据导入 mongodb 时如何修复 JavaScript 堆内存不足错误? [英] How to fix JavaScript heap out of memory error when importing data to mongodb?

查看：26
本文介绍了将数据导入 mongodb 时如何修复 JavaScript 堆内存不足错误?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

谁能指出正确的方向，为什么我无法将数据导入 mongodb?当我尝试仅导入总文件的前 100 行时

Anyone who can point me in right direction why I cant import the data to mongodb? When I try to import only the first 100 lines of the total file I get

database-operations git:(master) ✗ 节点 import_acparts_to_mongdb.js(节点:10216)警告:检测到可能的 EventEmitter 内存泄漏.11添加了密切的听众.使用emitter.setMaxListeners()增加e限制 ➜ 数据库操作 git:(master) ✗

database-operations git:(master) ✗ node import_acparts_to_mongdb.js (node:10216) Warning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increas e limit ➜ database-operations git:(master) ✗

我尝试从同一个文件中导入 600.000 行，这是一个具有以下结构的 csv 文件:

I try to import 600.000 lines from the same file, a csv file that has following structure:

facility;item_number;part_name;part_description;net_weight;customs_statisticalPBL;5535210444;封面;封面;0;84314980D37;5535211545;支架;支架-消防补给坦克 A101-20;2,939;72169110PBL;5535211234;支架;支架-消防补给坦克 A101-20;2,939;84314300PBL;5535212478;RING-SNAP;RING-SNAP;0,045;84314980............

facility;item_number;part_name;part_description;net_weight;customs_statistical PBL;5535210444;COVER;COVER;0;84314980 D37;5535211545;BRACKET;BRACKET-FIRE SUPP TANK A101-20;2,939;72169110 PBL;5535211234;BRACKET;BRACKET-FIRE SUPP TANK A101-20;2,939;84314300 PBL;5535212478;RING-SNAP;RING-SNAP;0,045;84314980 ....... .......

➜ 数据库操作 git:(master) ✗ 节点 import_acparts_to_mongdb.js

➜ database-operations git:(master) ✗ node import_acparts_to_mongdb.js

<--- 最后几次 GC --->

<--- Last few GCs --->

38787 毫秒:标记扫描 1384.9 (1436.8) -> 1384.8 (1436.8) MB，1181.9/0.0 ms [分配失败] [旧空间中的 GC 请求].39964毫秒:标记扫描 1384.8 (1436.8) -> 1384.8 (1436.8) MB，1177.7/0.0 毫秒[分配失败] [旧空间中的 GC 请求].41199 毫秒:标记扫描 1384.8 (1436.8) -> 1385.8 (1420.8) MB，1234.0/0.0 毫秒[最后一招gc].42429 毫秒:标记扫描 1385.8 (1420.8) -> 1386.9(1420.8) MB, 1229.8/0.0 ms [最后手段 gc].

38787 ms: Mark-sweep 1384.9 (1436.8) -> 1384.8 (1436.8) MB, 1181.9 / 0.0 ms [allocation failure] [GC in old space requested]. 39964 ms: Mark-sweep 1384.8 (1436.8) -> 1384.8 (1436.8) MB, 1177.7 / 0.0 ms [allocation failure] [GC in old space requested]. 41199 ms: Mark-sweep 1384.8 (1436.8) -> 1385.8 (1420.8) MB, 1234.0 / 0.0 ms [last resort gc]. 42429 ms: Mark-sweep 1385.8 (1420.8) -> 1386.9 (1420.8) MB, 1229.8 / 0.0 ms [last resort gc].

<--- JS 堆栈跟踪 --->

<--- JS stacktrace --->

==== JS 堆栈跟踪 =========================================

==== JS stack trace =========================================

安全上下文:0x4962c9cfb391:$__validate [/Users/isaklafleur/Dropbox/Isak/Coding/Other/autoMDM/node_modules/mongoose/lib/document.js:~1404][pc=0xe52ebc4f d97] (this=0x383867c1f221,callback=0x383867c201e1)2:验证[/Users/isaklafleur/Dropbox/Isak/Coding/Other/autoMDM/node_modules/mongoose/lib/document.js:~1324][pc=0x...

Security context: 0x4962c9cfb39 1: $__validate [/Users/isaklafleur/Dropbox/Isak/Coding/Other/autoMDM/node_modules/mongoose/lib/document.js:~1404] [pc=0xe52ebc4f d97] (this=0x383867c1f221 ,callback=0x383867c201e1 ) 2: validate [/Users/isaklafleur/Dropbox/Isak/Coding/Other/autoMDM/node_modules/mongoose/lib/document.js:~1324] [pc=0x...

致命错误:CALL_AND_RETRY_LAST 分配失败 - JavaScript 堆内存不足 1: node::Abort() [/usr/local/bin/node] 2:node::FatalException(v8::Isolate*, v8::Local,v8::Local) [/usr/local/bin/node] 3:v8::internal::V8::FatalProcessOutOfMemory(char const*, bool)[/usr/local/bin/node] 4: v8::internal::Factory::NewFillerObject(int,bool, v8::internal::AllocationSpace) [/usr/local/bin/node] 5:v8::internal::Runtime_AllocateInTargetSpace(int,v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]6: 0xe52eb8079a7 [1] 10085 中止节点import_acparts_to_mongdb.js ➜ 数据库操作 git:(master) ✗

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [/usr/local/bin/node] 2: node::FatalException(v8::Isolate*, v8::Local, v8::Local) [/usr/local/bin/node] 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node] 4: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node] 5: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node] 6: 0xe52eb8079a7 [1] 10085 abort node import_acparts_to_mongdb.js ➜ database-operations git:(master) ✗

<小时>

const mongoose  = require('mongoose'),
    parse       = require('csv-parse'),
    path        = require('path'),
    fs          = require('fs'),
    ACpart      = require('./models/acparts');

mongoose.Promise = require('bluebird');

mongoose.connect('mongodb://localhost/automdm_test');

const db = mongoose.connection;

db.on('error', console.error.bind(console, 'connection error:'));

db.once('open', function() {
    // we're connected!

    const p = path.join(__dirname, '/../', 'file-operations', 'csv-files');
    //console.log(p);

    const parser = parse({delimiter: ';'}, function(err, data){
        //console.log(data);
        const facility = data.map((item,i) => data[i][0]);
        const item_number = data.map((item,i) => data[i][1]);
        const part_name = data.map((item,i) => data[i][2]);
        const part_description = data.map((item,i) => data[i][3]);
        const net_weight = data.map((item,i) => data[i][4]);
        const customs_statistical = data.map((item,i) => data[i][5]);

        // Looping and storing the data into mongodb
        for (let i = 1; i < data.length; i++) {

            const newACpart = new ACpart();
            newACpart.facility = facility[i]
            newACpart.item_number = item_number[i];
            newACpart.part_name = part_name[i];
            newACpart.part_description = part_description[i];
            newACpart.net_weight = net_weight[i];
            newACpart.customs_statistical = customs_statistical[i];
            newACpart.save()
            .then(function() {
                mongoose.disconnect();
            })
            .catch(function(err) {
                console.log('There was an error', err);
            });
        }
    });
    fs.createReadStream(p + '/mrsparts.csv').pipe(parser);
});

推荐答案

如果所有东西都比堆大，你就不能把它放进内存.使用流式 CSV 解析器，例如其中之一:

You will not be able to put everything in memory if it's larger than you heap. Use a streaming CSV parser, like one of those:

然后将其分批发送到数据库，而不是一次发送.

And then send it to the database in batches instead of all at once.

这篇关于将数据导入 mongodb 时如何修复 JavaScript 堆内存不足错误?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
前端开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆