到/来自S3的AWS Lambda和分段上传 [英] AWS Lambda and Multipart Upload to/from S3

问题描述

使用Lambda将文件从S3移至我们的Redshift.

Using Lambda to move files from an S3 to our Redshift.

直接从数据提供商的Redshift使用UNLOAD命令将数据放置在S3中.它包含10个不同的部分，由于并行运行，有时会在不同的时间完成.

The data is placed in the S3 using an UNLOAD command directly from the data provider's Redshift. It comes in 10 different parts that, due to running in parallel, sometimes complete at different times.

我希望Lambda触发器要等到所有数据都完全上传后，再触发触发器将数据导入到我的Redshift中.

I want the Lambda trigger to wait until all the data is completely uploaded before firing the trigger to import the data to my Redshift.

Lambda中有一个名为完全分段上传"的事件选项. UNLOAD函数是否在Lambda中计为分段上传?还是在提供者完全上传了所有部分之前，不会触发简单的"POST"事件?

There is an event option in Lambda called "Complete Multipart Upload." Does the UNLOAD function count as a multipart upload within Lambda? Or would the simple "POST" event not fire until all the parts are completely uploaded by the provider?

没有明确的文档来确认Redshift的UNLOAD命令算作分段上传，也没有任何确认触发器在数据提供者的整个上传完成之前不会触发的信息.

There is no explicit documentation confirming that Redshift's UNLOAD command counts as a Multipart upload, or any confirming that the trigger will not fire until the data provider's entire upload is complete.

推荐答案

对于Amazon S3，分段上传是单个文件，可以分段上传到S3.当所有部分都已上传后，客户端将调用CompleteMultipartUpload.仅在客户端调用CompleteMultipartUpload之后，文件才会出现在S3中.

For Amazon S3, a multi-part upload is a single file, uploaded to S3 in multiple parts. When all parts have been uploaded, the client calls CompleteMultipartUpload. Only after the client calls CompleteMultipartUpload will the file appear in S3.

只有在文件完成后，才会触发Lambda函数.您不会为每个零件获得Lambda触发器.

And only after the file is complete will the Lambda function be triggered. You will not get a Lambda trigger for each part.

如果您的UNLOAD操作正在S3中生成多个对象/文件，则它不是S3的分段上传".

If your UNLOAD operation is generating multiple objects/files in S3, then it is NOT an S3 "multi-part upload".

这篇关于到/来自S3的AWS Lambda和分段上传的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！