Javascript读取大文件失败 [英] Javascript Read large files failed

问题描述

JSON文件为6 GB。使用以下代码阅读时，

A JSON file is 6 GB. When reading it with the following code,

var fs = require('fs');
var contents = fs.readFileSync('large_file.txt').toString();

出现以下错误：

buffer.js:182
    throw err;
    ^

RangeError: "size" argument must not be larger than 2147483647
    at Function.Buffer.allocUnsafe (buffer.js:209:3)
    at tryCreateBuffer (fs.js:530:21)
    at Object.fs.readFileSync (fs.js:569:14)
    at Object.<anonymous> (/home/readHugeFile.js:4:19)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)

有人可以帮忙吗？

推荐答案

缓冲区的最大尺寸 ，这是 readFileSync（）在内部使用来保存文件数据，大约是2GB（来源： https://nodejs.org/api/buffer.html#buffer_buffer_kmaxlength ）。

The maximum size for a Buffer, which is what readFileSync() uses internally to hold the file data, is about 2GB (source: https://nodejs.org/api/buffer.html#buffer_buffer_kmaxlength).

你可能需要一个流式JSON解析器，如 JSONStream ，处理你的文件：

You probably need a streaming JSON parser, like JSONStream, to process your file:

const JSONStream = require('JSONStream');
const fs         = require('fs');

fs.createReadStream('large_file.json')
  .pipe(JSONStream.parse('*'))
  .on('data', entry => {
    console.log('entry', entry);
  });

这篇关于Javascript读取大文件失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！