node.js：将文本文件读入数组。 （每行都是数组中的一个项目。） [英] node.js: read a text file into an array. (Each line an item in the array.)

问题描述

我想将一个非常非常大的文件读入node.js中的JavaScript数组。

I would like to read a very, very large file into a JavaScript array in node.js.

所以，如果文件是这样的：

So, if the file is like this:

first line
two 
three
...
...

我有数组：

['first line','two','three', ... , ... ] 

该函数如下所示：

var array = load(filename); 

因此，将它全部作为字符串加载然后拆分它的想法是不可接受的。

Therefore the idea of loading it all as a string and then splitting it is not acceptable.

推荐答案

如果您可以将最终数据放入数组中，那么您也不能将它放入字符串中并将其拆分，有人建议吗？
在任何情况下，如果您想一次处理一行文件，您也可以尝试这样的事情：

If you can fit the final data into an array then wouldn't you also be able to fit it in a string and split it, as has been suggested? In any case if you would like to process the file one line at a time you can also try something like this:

var fs = require('fs');

function readLines(input, func) {
  var remaining = '';

  input.on('data', function(data) {
    remaining += data;
    var index = remaining.indexOf('\n');
    while (index > -1) {
      var line = remaining.substring(0, index);
      remaining = remaining.substring(index + 1);
      func(line);
      index = remaining.indexOf('\n');
    }
  });

  input.on('end', function() {
    if (remaining.length > 0) {
      func(remaining);
    }
  });
}

function func(data) {
  console.log('Line: ' + data);
}

var input = fs.createReadStream('lines.txt');
readLines(input, func);

编辑：（以回应 phopkins的评论）我认为（至少在较新版本中）substring不会复制数据，而是创建一个特殊的SlicedString对象（快速浏览一下v8源代码）。在任何情况下，这里都有一个修改，避免提到的子字符串（在一个文件上测试几兆字节的所有工作，没有游戏让杰克变成一个沉闷的男孩）：

(in response to comment by phopkins) I think (at least in newer versions) substring does not copy data but creates a special SlicedString object (from a quick glance at the v8 source code). In any case here is a modification that avoids the mentioned substring (tested on a file several megabytes worth of "All work and no play makes Jack a dull boy"):

function readLines(input, func) {
  var remaining = '';

  input.on('data', function(data) {
    remaining += data;
    var index = remaining.indexOf('\n');
    var last  = 0;
    while (index > -1) {
      var line = remaining.substring(last, index);
      last = index + 1;
      func(line);
      index = remaining.indexOf('\n', last);
    }

    remaining = remaining.substring(last);
  });

  input.on('end', function() {
    if (remaining.length > 0) {
      func(remaining);
    }
  });
}

这篇关于node.js：将文本文件读入数组。 （每行都是数组中的一个项目。）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！