将流通过管道传输到 s3.upload() [英] Pipe a stream to s3.upload()

问题描述

我目前正在使用一个名为 s3-upload-stream 的 node.js 插件来流式传输大文件到 Amazon S3.它使用多部分 API，并且在大多数情况下运行良好.

I'm currently making use of a node.js plugin called s3-upload-stream to stream very large files to Amazon S3. It uses the multipart API and for the most part it works very well.

然而，这个模块显示了它的年龄，我已经不得不对其进行修改(作者也弃用了它).今天又遇到了Amazon的问题，很想采纳作者的建议，开始使用官方的aws-sdk来完成上传.

However, this module is showing its age and I've already had to make modifications to it (the author has deprecated it as well). Today I ran into another issue with Amazon, and I would really like to take the author's recommendation and start using the official aws-sdk to accomplish my uploads.

但是.

官方 SDK 似乎不支持管道到 s3.upload().s3.upload 的本质是您必须将可读流作为参数传递给 S3 构造函数.

The official SDK does not seem to support piping to s3.upload(). The nature of s3.upload is that you have to pass the readable stream as an argument to the S3 constructor.

我有大约 120 多个用户代码模块可以进行各种文件处理，并且它们与输出的最终目的地无关.引擎交给他们一个可管道化的可写输出流，然后他们通过管道传递给它.我不能给他们一个 AWS.S3 对象并要求他们在不向所有模块添加代码的情况下调用 upload() .我使用 s3-upload-stream 的原因是因为它支持管道.

I have roughly 120+ user code modules that do various file processing, and they are agnostic to the final destination of their output. The engine hands them a pipeable writeable output stream, and they pipe to it. I cannot hand them an AWS.S3 object and ask them to call upload() on it without adding code to all the modules. The reason I used s3-upload-stream was because it supported piping.

有没有办法让 aws-sdk s3.upload() 一些我可以通过管道传输到的东西?

Is there a way to make aws-sdk s3.upload() something I can pipe the stream to?

推荐答案

用 node.js stream.PassThrough() 流包裹 S3 upload() 函数.

Wrap the S3 upload() function with the node.js stream.PassThrough() stream.

这是一个例子:

inputStream
  .pipe(uploadFromStream(s3));

function uploadFromStream(s3) {
  var pass = new stream.PassThrough();

  var params = {Bucket: BUCKET, Key: KEY, Body: pass};
  s3.upload(params, function(err, data) {
    console.log(err, data);
  });

  return pass;
}

这篇关于将流通过管道传输到 s3.upload()的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！