节点读取指定块大小的文件 [英] Node reading file in specified chunk size

问题描述

目标:将大文件上传到 AWS Glacier，而不会将整个文件保存在内存中.

The goal: Upload large files to AWS Glacier without holding the whole file in memory.

我目前正在使用 fs.readFileSync() 上传到冰川，一切正常.但是，我需要处理大于 4GB 的文件，并且我想并行上传多个块.这意味着转向分段上传.我可以选择块大小，但冰川需要每个块的大小相同(最后一个除外)

I'm currently uploading to glacier now using fs.readFileSync() and things are working. But, I need to handle files larger than 4GB and I'd like to upload multiple chunks in parallel. This means moving to multipart uploads. I can choose the chunk size but then glacier needs every chunk to be the same size (except the last)

这个线程建议我可以在阅读流，但我实际上并不能保证得到它.

This thread suggests that I can set a chunk size on a read stream but that I'm not actually guaranteed to get it.

关于如何在不将整个文件读入内存并手动拆分的情况下获得一致的部分的任何信息?

Any info on how I can get consistent parts without reading the whole file into memory and splitting it up manually?

假设我能做到这一点，我只是要使用集群，其中有几个进程以尽可能快的速度拉出流，因为它们可以上传到 AWS.如果这似乎是并行工作的错误方式，我会喜欢那里的建议.

Assuming I can get to that point I was just going to use cluster with a few processes pulling off the stream as fast as they can upload to AWS. If that seems like the wrong way to parallelize the work I'd love suggestions there.

推荐答案

如果没有别的，你可以使用 fs.open(), fs.read() 和 fs.close() 手动.示例:

If nothing else you can just use fs.open(), fs.read(), and fs.close() manually. Example:

var CHUNK_SIZE = 10 * 1024 * 1024, // 10MB
    buffer = Buffer.alloc(CHUNK_SIZE),
    filePath = '/tmp/foo';

fs.open(filePath, 'r', function(err, fd) {
  if (err) throw err;
  function readNextChunk() {
    fs.read(fd, buffer, 0, CHUNK_SIZE, null, function(err, nread) {
      if (err) throw err;

      if (nread === 0) {
        // done reading file, do any necessary finalization steps

        fs.close(fd, function(err) {
          if (err) throw err;
        });
        return;
      }

      var data;
      if (nread < CHUNK_SIZE)
        data = buffer.slice(0, nread);
      else
        data = buffer;

      // do something with `data`, then call `readNextChunk();`
    });
  }
  readNextChunk();
});

这篇关于节点读取指定块大小的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！