将流通过管道传输到s3.upload() [英] Pipe a stream to s3.upload()

问题描述

我目前正在使用名为 s3-upload-stream 的node.js插件来非常流式传输大文件到Amazon S3.它使用了由多个部分组成的API，并且在大多数情况下效果都很好.

I'm currently making use of a node.js plugin called s3-upload-stream to stream very large files to Amazon S3. It uses the multipart API and for the most part it works very well.

但是，此模块正在显示其年龄，我已经不得不对其进行修改(作者也已弃用它).今天，我遇到了另一个与亚马逊有关的问题，我真的很想接受作者的建议，并开始使用官方的aws-sdk完成上传.

However, this module is showing its age and I've already had to make modifications to it (the author has deprecated it as well). Today I ran into another issue with Amazon, and I would really like to take the author's recommendation and start using the official aws-sdk to accomplish my uploads.

但是.

官方SDK似乎不支持管道到s3.upload(). s3.upload的本质是，您必须将可读流作为参数传递给S3构造函数.

The official SDK does not seem to support piping to s3.upload(). The nature of s3.upload is that you have to pass the readable stream as an argument to the S3 constructor.

我大约有120多个用于执行各种文件处理的用户代码模块，它们与输出的最终目标无关.引擎将它们传递给可管道传输的可写输出流，并通过管道传递给它.我不能给他们一个AWS.S3对象并要求他们在不向所有模块添加代码的情况下调用upload()对象.我使用s3-upload-stream的原因是因为它支持管道.

I have roughly 120+ user code modules that do various file processing, and they are agnostic to the final destination of their output. The engine hands them a pipeable writeable output stream, and they pipe to it. I cannot hand them an AWS.S3 object and ask them to call upload() on it without adding code to all the modules. The reason I used s3-upload-stream was because it supported piping.

有没有一种方法可以使aws-sdk s3.upload()可以通过流传输到其中?

Is there a way to make aws-sdk s3.upload() something I can pipe the stream to?

推荐答案

用node.js stream.PassThrough()流包装S3 upload()函数.

Wrap the S3 upload() function with the node.js stream.PassThrough() stream.

这是一个例子:

inputStream
  .pipe(uploadFromStream(s3));

function uploadFromStream(s3) {
  var pass = new stream.PassThrough();

  var params = {Bucket: BUCKET, Key: KEY, Body: pass};
  s3.upload(params, function(err, data) {
    console.log(err, data);
  });

  return pass;
}

这篇关于将流通过管道传输到s3.upload()的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！