使用强大的和(knox 或 aws-sdk)将文件流式上传到 Node.js 上的 S3 [英] Stream uploading file to S3 on Node.js using formidable and (knox or aws-sdk)
问题描述
我正在尝试使用 aws-sdk 或 knox.表单处理是通过 formidable 完成的.
I'm trying to stream upload a file submitted via a form directly to an Amazon S3 bucket, using aws-sdk or knox. Form handling is done with formidable.
我的问题是:如何使用这些库的每个最新功能来处理流，我如何正确地将 formidable 与 aws-sdk(或 knox)一起使用?
My question is: how do I properly use formidable with aws-sdk (or knox) using each of these libraries' latest features for handling streams?
我知道这里已经有人以不同的方式询问过这个话题，即:
I'm aware that this topic has already been asked here in different flavors, ie:
- 如何使用 node.js 强大的库接收上传的文件并使用 knox 将其保存到 Amazon S3?
- node应用流文件直接上传到amazon s3
- 从一个访问原始文件流节点强大的文件上传(以及它关于覆盖 form.onPart() 的非常有用的公认答案)
- How to receive an uploaded file using node.js formidable library and save it to Amazon S3 using knox?
- node application stream file upload directly to amazon s3
- Accessing the raw file stream from a node-formidable file upload (and its very useful accepted answer on overiding form.onPart())
但是，我认为这些答案有点过时和/或偏离主题(即 CORS 支持，由于各种原因我现在不希望使用它)和/或最重要的是，没有提及来自 aws-sdk 的最新功能(请参阅:https://github.com/aws/aws-sdk-js/issues/13#issuecomment-16085442) 或 knox(特别是 putStream() 或其 readableStream.pipe(req) 变体，两者都在文档中进行了解释).
However, I believe the answers are a bit outdated and/or off topic (ie. CORS support, which I don't wish to use for now for various reasons) and/or, most importantly, make no reference to the latest features from either aws-sdk (see: https://github.com/aws/aws-sdk-js/issues/13#issuecomment-16085442) or knox (notably putStream() or its readableStream.pipe(req) variant, both explained in the doc).
经过数小时的挣扎，我得出的结论是我需要一些帮助(免责声明:我是流的新手).
After hours of struggling, I came to the conclusion that I needed some help (disclaimer: I'm quite a newbie with streams).
HTML 表单:
<form action="/uploadPicture" method="post" enctype="multipart/form-data">
<input name="picture" type="file" accept="image/*">
<input type="submit">
</form>
Express bodyParser 中间件是这样配置的:
Express bodyParser middleware is configured this way:
app.use(express.bodyParser({defer: true}))
POST 请求处理程序:
POST request handler:
uploadPicture = (req, res, next) ->
form = new formidable.IncomingForm()
form.parse(req)
form.onPart = (part) ->
if not part.filename
# Let formidable handle all non-file parts (fields)
form.handlePart(part)
else
handlePart(part, form.bytesExpected)
handlePart = (part, fileSize) ->
# aws-sdk version
params =
Bucket: "mybucket"
Key: part.filename
ContentLength: fileSize
Body: part # passing stream object as body parameter
awsS3client.putObject(params, (err, data) ->
if err
console.log err
else
console.log data
)
但是，我收到以下错误:
However, I'm getting the following error:
{ [RequestTimeout: 您与服务器的套接字连接在超时期限内未被读取或写入.空闲连接将被关闭.]
{ [RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.]
消息:'您与服务器的套接字连接在超时期限内未被读取或写入.空闲连接将被关闭.',代码:'请求超时'，name: '请求超时',状态代码:400，可重试:假}
message: 'Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.', code: 'RequestTimeout', name: 'RequestTimeout', statusCode: 400, retryable: false }
以这种方式定制的 Knox 版本的 handlePart() 函数也失败了:
A knox version of handlePart() function tailored this way also miserably fails:
handlePart = (part, fileSize) ->
headers =
"Content-Length": fileSize
"Content-Type": part.mime
knoxS3client.putStream(part, part.filename, headers, (err, res) ->
if err
console.log err
else
console.log res
)
我还在某处得到了一个带有 400 statusCode 的大 res 对象.
I also get a big res object with a 400 statusCode somewhere.
Region 在这两种情况下都配置为 eu-west-1.
Region is configured to eu-west-1 in both case.
附加说明:
节点 0.10.12
来自 npm (1.0.14) 的最新强大
latest formidable from npm (1.0.14)
来自 npm (1.3.1) 的最新 aws-sdk
latest aws-sdk from npm (1.3.1)
来自 npm (0.8.3) 的最新 Knox
latest knox from npm (0.8.3)
推荐答案
好吧，根据Formidable 的创造者，直接流式传输到 Amazon S3 是不可能的:
Well, according to the creator of Formidable, direct streaming to Amazon S3 is impossible :
S3 API 要求您在创建新文件时提供新文件的大小.在完全收到多部分/表单数据文件之前，此信息不可用.这意味着流式传输是不可能的.
The S3 API requires you to provide the size of new files when creating them. This information is not available for multipart/form-data files until they have been fully received. This means streaming is impossible.
确实，form.bytesExpected指的是整个表单的大小，而不是单个文件的大小.
Indeed, form.bytesExpected refers to the size of the whole form, and not the size of the single file.
因此，数据在上传到 S3 之前必须首先命中服务器上的内存或磁盘.
The data must therefore either hit the memory or the disk on the server first before being uploaded to S3.
这篇关于使用强大的和(knox 或 aws-sdk)将文件流式上传到 Node.js 上的 S3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
