节点以指定的块大小读取文件 [英] Node reading file in specified chunk size

问题描述

目标：将大型文件上传到AWS Glacier而不将整个文件保存在内存中。

The goal: Upload large files to AWS Glacier without holding the whole file in memory.

我目前正在使用fs.readFileSync（）和一切正常。但是，我需要处理大于4GB的文件，并且希望并行上传多个块。这意味着转移到分段上传。
我可以选择块的大小，但是冰川需要每个块都具有相同的大小（最后一个除外）

I'm currently uploading to glacier now using fs.readFileSync() and things are working. But, I need to handle files larger than 4GB and I'd like to upload multiple chunks in parallel. This means moving to multipart uploads. I can choose the chunk size but then glacier needs every chunk to be the same size (except the last)

此线程建议我可以在读取流上设置块大小，但实际上并不能保证获得它。

This thread suggests that I can set a chunk size on a read stream but that I'm not actually guaranteed to get it.

关于如何获取一致部分而无需将整个文件读入内存并手动拆分的任何信息？

Any info on how I can get consistent parts without reading the whole file into memory and splitting it up manually?

假设我可以做到这一点，我将使用集群，并使用一些进程以最快的速度将其拉出流，然后将它们上传到AWS。
如果这似乎是使工作并行化的错误方法，我会在那儿建议。

Assuming I can get to that point I was just going to use cluster with a few processes pulling off the stream as fast as they can upload to AWS. If that seems like the wrong way to parallelize the work I'd love suggestions there.

推荐答案

您可以只使用 fs.open（） ， fs.read （） 和 fs.close（） 手动。示例：

If nothing else you can just use fs.open(), fs.read(), and fs.close() manually. Example:

var CHUNK_SIZE = 10 * 1024 * 1024, // 10MB
    buffer = Buffer.alloc(CHUNK_SIZE),
    filePath = '/tmp/foo';

fs.open(filePath, 'r', function(err, fd) {
  if (err) throw err;
  function readNextChunk() {
    fs.read(fd, buffer, 0, CHUNK_SIZE, null, function(err, nread) {
      if (err) throw err;

      if (nread === 0) {
        // done reading file, do any necessary finalization steps

        fs.close(fd, function(err) {
          if (err) throw err;
        });
        return;
      }

      var data;
      if (nread < CHUNK_SIZE)
        data = buffer.slice(0, nread);
      else
        data = buffer;

      // do something with `data`, then call `readNextChunk();`
    });
  }
  readNextChunk();
});

这篇关于节点以指定的块大小读取文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！