Node.js使用fs.readFileSync读取大文件 [英] Node.js read big file with fs.readFileSync
问题描述
我尝试通过具有96GB RAM的服务器上的 fs.readFileSync 将大文件（〜6Gb）加载到内存中。
I try to load big file (~6Gb) into memory with fs.readFileSync on the server with 96GB RAM.
问题是否失败并显示以下错误消息
The problem is it fails with the following error message
RangeError：尝试分配大于最大大小的缓冲区：0x3fffffff字节
RangeError: Attempt to allocate Buffer larger than maximum size: 0x3fffffff bytes
不幸的是，我没有发现增加Buffer的可能性，这似乎是一个常数。
Unfortunately I didn't find how it is possible to increase Buffer, it seems like it's a constant.
如何克服这个问题并使用Node.js加载大文件？
How I can overcome this problem and load a big file with Node.js?
谢谢！
推荐答案
从喜悦常见问题解答：
节点进程的内存限制是多少？
What is the memory limit on a node process?
当前，默认情况下，v8在32位
系统上的内存限制为512mb，在64位系统上为1gb。可以通过将
--max_old_space_size设置为最大〜1024（〜1 GiB）（32位）和〜1741（〜1.7GiB）（64位）来提高限制，但是建议您如果您要达到内存限制，则将单个
进程拆分为多个工作程序。
Currently, by default v8 has a memory limit of 512mb on 32-bit systems, and 1gb on 64-bit systems. The limit can be raised by setting --max_old_space_size to a maximum of ~1024 (~1 GiB) (32-bit) and ~1741 (~1.7GiB) (64-bit), but it is recommended that you split your single process into several workers if you are hitting memory limits.
如果您显示有关文件以及您正在使用的文件，我们可能会提供一些有关如何分块使用它的想法。如果是纯数据，则您可能希望使用数据库，并让数据库根据需要从磁盘中获取内容并管理内存。
If you show more detail about what's in the file and what you're doing with it, we can probably offer some ideas on how to work with it in chunks. If it's pure data, then you probably want to be using a database and let the database handle getting things from disk as needed and manage the memory.
这里是最近的讨论问题的原因： https://code.google.com/p/v8 / issues / detail？id = 847
Here's a fairly recent discussion of the issue: https://code.google.com/p/v8/issues/detail?id=847
然后，这是博客文章，其中声称您可以编辑V8源代码并重建节点以删除内存限制。自行决定尝试此操作。
And, here's a blog post that claims you can edit the V8 source code and rebuilt node to remove the memory limit. Try this at your own discretion.
这篇关于Node.js使用fs.readFileSync读取大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
