上传大文件到S3与恢复支持 [英] Upload large files to S3 with resume support

问题描述

（我是新来的亚马逊AWS / S3，所以请大家多多包涵）

(I'm new to Amazon AWS/S3, so please bear with me)

我的最终目标是让使用他们的网络浏览器我的用户将文件上传到S3，我的要求是：

My ultimate goal is to allow my users to upload files to S3 using their web browser, my requirements are:

1. 在我必须处理大文件（2GB +）
2. 我必须支持暂停/使用进度指示器恢复
3. （可选，但希望！）有能力重新上传，如果连接暂时跌出

我的两个部分的问题是：

• 在我读过有关S3的多部分上传的，但现在还不清楚哪能 实现暂停/恢复的web浏览器为基础的上传。

• I've read about the S3 multipart upload but it's not clear how can I implement the pause/resume for webbrowser-based uploads.

它甚至有可能做到这一点的大文件？如果是的话怎么样？

Is it even possible to do this for large files? If so how?

• 我应该将文件上传到EC2然后将它们移到S3，一旦我做了什么？能够 我（安全）上传文件直接使用一个临时的S3代替。 Web服务器？

• Should I upload files to EC2 then move them to S3 once I'm done? Can I (securely) upload files directly to S3 instead of using a temp. webserver?

如果有可能直接上传到S3，我该怎么处理暂停/恢复？

If it's possible to upload directly to S3, how can I handle pause/resume?

PS。我使用PHP 5.2 +

推荐答案

的同时提供 AWS SDK对JavaScript（在浏览器中）支持Amazon S3的，包括一类 ManagedUpload 支持用例在手的多部分上传方面（见更多关于这个preceding更新）。现在它可能是您的情况最好的解决方案。因此，如见<一href="http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-examples.html#Uploading_a_local_file_using_the_File_API"相对=nofollow>使用File API 一个简明的例子，它使用的 HTML5文件API 反过来 - 介绍博客文章<一个href="http://blogs.aws.amazon.com/javascript/post/Tx3EQZP53BODXWF/Announcing-the-Amazon-S3-Managed-Uploader-in-the-AWS-SDK-for-JavaScript"相对=nofollow>宣布亚马逊S3管理上载的AWS SDK为JavaScript 提供有关此SDK功能的更多细节。

Update 20150527

The meanwhile available AWS SDK for JavaScript (in the Browser) supports Amazon S3, including a class ManagedUpload to support the multipart upload aspects of the use case at hand (see preceding update for more on this). It might now be the best solution for your scenario accordingly, see e.g. Uploading a local file using the File API for a concise example that uses the HTML5 File API in turn - the introductory blog post Announcing the Amazon S3 Managed Uploader in the AWS SDK for JavaScript provides more details about this SDK feature.

我最初的回答显然错过了主攻点，所以要澄清：

My initial answer apparently missed the main point, so to clarify:

如果你想要做的基于浏览器的通过简单的HTML表单上传，您被限制使用的 POST对象操作，其中的使用HTML表单添加一个对象到指定的斗的：

If you want to do browser based upload via simple HTML forms, you are constrained to using the POST Object operation, which adds an object to a specified bucket using HTML forms:

POST放的另一种形式，它使基于浏览器的上传，   一种方法把对象的桶。被传递给put参数   通过HTTP头，而不是传递表单字段中为POST   的multipart / form-data的EN codeD邮件正文。 [...]

POST is an alternate form of PUT that enables browser-based uploads as a way of putting objects in buckets. Parameters that are passed to PUT via HTTP Headers are instead passed as form fields to POST in the multipart/form-data encoded message body. [...]

上传在这里单个操作处理，因此不支持暂停/恢复和限制您的5千兆字节（GB）或原始的最大对象大小的少的。

The upload is handled in a single operation here, thus doesn't support pause/resume and limits you to the original maximum object size of 5 gigabytes (GB) or less.

您只能克服了两者的限制由使用多部分上传的REST API，而不是< /一>，后者又使用像 AWS SDK软件开发工具包为PHP 实现此功能。

You can only overcome both limitations by Using the REST API for Multipart Upload instead, which is in turn used by SDKs like the AWS SDK for PHP to implement this functionality.

这显然需要一台服务器（例如，在EC2）来处理通过浏览器发起的操作（它允许您方便的 S3斗政策和/或 IAM政策访问控制容易为好）。

This obviously requires a server (e.g. on EC2) to handle the operation initiated via the browser (which allows you to facilitate S3 Bucket Policies and/or IAM Policies for access control easily as well).

在一个选择可能是使用JavaScript库，并执行该客户端，如见 jQuery的上传进度和AJAX文件上传一个初步的指针。 <击>不幸的是没有规范的JavaScript SDK可用的AWS（ AWS-lib中令人惊讶的竟然不支持S3还） - 诺克斯增加了多部分上传显然有些叉，如见 slakis的叉，我还没有使用其中任一对用例在眼前，虽然。

The one alternative might be using a JavaScript library and performing this client side, see e.g. jQuery Upload Progress and AJAX file upload for an initial pointer. Unfortunately there is no canonical JavaScript SDK for AWS available (aws-lib surprisingly doesn't even support S3 yet) - apparently some forks of knox have added multipart upload, see e.g. slakis's fork, I haven't used either of these for the use case at hand though.

如果有可能上载[大文件]直接到S3，我该怎么处理   暂停/恢复？

If it's possible to upload [large files] directly to S3, how can I handle pause/resume?

借助 AWS SDK的PHP 支持按的方式上传大文件到Amazon S3的对于多上传低级别的PHP API：

The AWS SDK for PHP supports uploading large files to Amazon S3 by means of the Low-Level PHP API for Multipart Upload:

在AWS SDK的PHP暴露了低级别的API，它非常类似于   对于多部分上传的Amazon S3 REST API（请参阅使用的REST API   多部分上传）。 使用低级别的API时，你需要停下来   继续多部分上传，上传过程中改变零件尺寸，或不   知道数据提前尺寸。使用高层次的API（见   使用多部分上传的高级PHP API），只要你不   有这些要求。 [重点煤矿] 的

The AWS SDK for PHP exposes a low-level API that closely resembles the Amazon S3 REST API for multipart upload (see Using the REST API for Multipart Upload ). Use the low-level API when you need to pause and resume multipart uploads, vary part sizes during the upload, or do not know the size of the data in advance. Use the high-level API (see Using the High-Level PHP API for Multipart Upload) whenever you don't have these requirements. [emphasis mine]

亚马逊S3可以处理的对象的从1个字节一路5千兆字节（TB）的，请参见相应的介绍后的亚马逊S3 - 对象大小的限制，现在5 TB ：

Amazon S3 can handle objects from 1 byte all the way to 5 terabytes (TB), see the respective introductory post Amazon S3 - Object Size Limit Now 5 TB:

[...]现在，用户可以存储非常   大文件作为单个对象，从而大大简化了它们的存储   经验。亚马逊S3做幕后的簿记我们   客户，所以现在就可以拿到大的物体就像你   任何其他Amazon S3对象。

[...] Now customers can store extremely large files as single objects, which greatly simplifies their storage experience. Amazon S3 does the bookkeeping behind the scenes for our customers, so you can now GET that large object just like you would any other Amazon S3 object.

为了存储更大的对象，你会使用新的多部分上传API 我的博客上讲述上个月上载的对象   部分。 [...]

In order to store larger objects you would use the new Multipart Upload API that I blogged about last month to upload the object in parts. [...]

这篇关于上传大文件到S3与恢复支持的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！