直接上传到S3使用Python / Boto / Django来构建策略 [英] Direct Upload to S3 Using Python/Boto/Django to Construct Policy

问题描述

我已经经历了很多次这个问题的迭代，搜索了很多不同的例子，并且都是通过这些文档。

I have been through many iterations of this problem so far, searched out many different examples, and have been all through the documentation.

我正在尝试结合Plupload （ http://www.plupload.com/ ）使用AWS S3直接方式（ http://aws.amazon.com/articles/1434 ）。不过，我相信我正在制定我的政策和传输签名方式有问题。当我提交表单时，我没有得到服务器的响应，但是我重新连接到服务器。

I am trying to combine Plupload (http://www.plupload.com/) with the AWS S3 direct post method (http://aws.amazon.com/articles/1434). However, I believe there's something wrong with the way I am constructing my policy and signature for transmission. When I submit the form, I don't get a response from the server, but rather my connection to the server is reset.

我尝试使用python代码示例：

I have attempted using the python code in the example:

import base64
import hmac, sha

policy = base64.b64encode(policy_document)

signature = base64.b64encode(
hmac.new(aws_secret_key, policy, sha).digest())

我还试图在python中使用更多最新的hashlib库。无论我用什么方法来构建我的策略和签名，我总是得到不同于在这里生成的值：

I have also tried to use the more up-to-date hashlib library in python. Whatever method I use to construct my policy and signature, I always get different values than those generated here:

http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html

我已经阅读了这个问题：

I have read through this question:

如何使Plupload直接上传到Amazon S3？

但是我发现这些例子过于复杂，无法准确地实现。

But I found the examples provided to be overly complicated and wasn't able to accurately implement them.

我最近的尝试是使用boto库：

My most recent attempts have been to use portions of the boto library:

http://boto.cloudhackers.com/ref/s3.html#module-boto.s3.connection

但是使用S3Commection .build_po st_form_args方法对我来说还没有工作。

But using the S3Commection.build_post_form_args method has not worked for me either.

如果有人可以提供一个正确的例子，说明如何使用python创建post表单，我将非常感谢。甚至一些简单的见解，为什么连接总是重置将是很好的。

If anyone could provide a proper example of how to create the post form using python, I would very much appreciate it. Even some simple insights on why the connection is always reset would be nice.

一些注意事项：

我会喜欢使用hashlib，如果可能的话。
我想从亚马逊获取XML响应（大概success_action_status ='201）这样做）
我需要能够上传大型文件，最大大小〜2GB。

I would like to use hashlib if possible. I want to get an XML response from Amazon (presumably "success_action_status = '201'" does this) I need to be able to upload largish type files, max size ~2GB.

最后一个注意事项是，当我在Chrome中运行时，它提供了上传进度，上传通常在37％左右。

One final note, when I run this in Chrome, it provides upload progress, and the upload usually fails around 37%.

推荐答案

Nathan的回答帮助我开始了。我已经包括两个目前为我工作的解决方案。

Nathan's answer helped get me started. I've included two solutions that are currently working for me.

第一个解决方案使用纯Python。第二个使用boto。

The first solution uses plain Python. The second uses boto.

我试图让boto先工作，但不断收到错误。所以我回到了亚马逊的ruby文档，并让S3接受文件使用python没有boto。 （使用HTML POST浏览器上传到S3）

I tried to get boto working first, but kept getting errors. So I went back to the Amazon ruby documentation and got S3 to accept files using python without boto. (Browser Uploads to S3 using HTML POST)

在了解发生了什么后，我能够修复我的错误并使用boto，这是一个更简单的解决方案。

After understanding what was going on, I was able to fix my errors and use boto, which is a simpler solution.

我正在包括解决方案1，因为它明确显示如何使用python设置策略文档和签名。

I'm including solution 1 because it shows explicitly how to setup the policy document and signature using python.

我的目标是将html上传页面创建为动态页面，以及用户在成功上传后看到的成功页面。解决方案1显示了表单上传页面的动态创建，而解决方案2显示了上传表单页面和成功页面的创建。

My goal was to create the html upload page as a dynamic page, along with the "success" page the user sees after a successful upload. Solution 1 shows the dynamic creation of the form upload page, while solution 2 shows the creation of both the upload form page and the success page.

解决方案1：

import base64
import hmac, hashlib

###### EDIT ONLY THE FOLLOWING ITEMS ######

DEBUG = 1
AWS_SECRET_KEY = "MySecretKey"
AWS_ACCESS_KEY = "MyAccessKey"
HTML_NAME = "S3PostForm.html"
EXPIRE_DATE = "2015-01-01T00:00:00Z" # Jan 1, 2015 gmt
FILE_TO_UPLOAD = "${filename}"
BUCKET = "media.mysite.com"
KEY = ""
ACL = "public-read" # or "private"
SUCCESS = "http://media.mysite.com/success.html"
CONTENT_TYPE = ""
CONTENT_LENGTH = 1024**3 # One gigabyte
HTTP_OR_HTTPS = "http" # Or "https" for better security
PAGE_TITLE = "My Html Upload to S3 Form"
ACTION = "%s://%s.s3.amazonaws.com/" % (HTTP_OR_HTTPS, BUCKET)

###### DON'T EDIT FROM HERE ON DOWN ######

policy_document_data = {
"expire": EXPIRE_DATE,
"bucket_name": BUCKET,
"key_name": KEY,
"acl_name": ACL,
"success_redirect": SUCCESS,
"content_name": CONTENT_TYPE,
"content_length": CONTENT_LENGTH,
}

policy_document = """
{"expiration": "%(expire)s",
  "conditions": [ 
    {"bucket": "%(bucket_name)s"}, 
    ["starts-with", "$key", "%(key_name)s"],
    {"acl": "%(acl_name)s"},
    {"success_action_redirect": "%(success_redirect)s"},
    ["starts-with", "$Content-Type", "%(content_name)s"],
    ["content-length-range", 0, %(content_length)d]
  ]
}
""" % policy_document_data

policy = base64.b64encode(policy_document)
signature = base64.b64encode(hmac.new(AWS_SECRET_KEY, policy, hashlib.sha1).digest())

html_page_data = {
"page_title": PAGE_TITLE,
"action_name": ACTION,
"filename": FILE_TO_UPLOAD,
"access_name": AWS_ACCESS_KEY,
"acl_name": ACL,
"redirect_name": SUCCESS,
"policy_name": policy,
"sig_name": signature,
"content_name": CONTENT_TYPE,
}

html_page = """
<html> 
 <head>
  <title>%(page_title)s</title> 
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 </head>
<body>
 <form action="%(action_name)s" method="post" enctype="multipart/form-data">
  <input type="hidden" name="key" value="%(filename)s">
  <input type="hidden" name="AWSAccessKeyId" value="%(access_name)s">
  <input type="hidden" name="acl" value="%(acl_name)s">
  <input type="hidden" name="success_action_redirect" value="%(redirect_name)s">
  <input type="hidden" name="policy" value="%(policy_name)s">
  <input type="hidden" name="signature" value="%(sig_name)s">
  <input type="hidden" name="Content-Type" value="%(content_name)s">

  <!-- Include any additional input fields here -->

  Browse to locate the file to upload:<br \> <br \>

  <input name="file" type="file"><br> <br \>
  <input type="submit" value="Upload File to S3"> 
 </form> 
</body>
</html>
""" % html_page_data

with open(HTML_NAME, "wb") as f:
    f.write(html_page)

###### Dump output if testing ######
if DEBUG:

    if 1: # Set true if not using the LEO editor
        class G:
            def es(self, data):print(data)
        g = G()

    items = [
    "",
    "",
    "policy_document: %s" % policy_document,
    "ploicy: %s" % policy,
    "signature: %s" % signature,
    "",
    "",
    ]
    for item in items:
        g.es(item)

解决方案2：

from boto.s3 import connection

###### EDIT ONLY THE FOLLOWING ITEMS ######

DEBUG = 1
AWS_SECRET_KEY = "MySecretKey"
AWS_ACCESS_KEY = "MyAccessKey"
HTML_NAME = "S3PostForm.html"
SUCCESS_NAME = "success.html"
EXPIRES = 60*60*24*356 # seconds = 1 year
BUCKET = "media.mysite.com"
KEY = "${filename}" # will match file entered by user
ACL = "public-read" # or "private"
SUCCESS = "http://media.mysite.com/success.html"
CONTENT_TYPE = "" # seems to work this way
CONTENT_LENGTH = 1024**3 # One gigabyte
HTTP_OR_HTTPS = "http" # Or https for better security
PAGE_TITLE = "My Html Upload to S3 Form"

###### DON'T EDIT FROM HERE ON DOWN ######

conn = connection.S3Connection(AWS_ACCESS_KEY,AWS_SECRET_KEY)
args = conn.build_post_form_args(
    BUCKET,
    KEY,
    expires_in=EXPIRES,
    acl=ACL,
    success_action_redirect=SUCCESS,
    max_content_length=CONTENT_LENGTH,
    http_method=HTTP_OR_HTTPS,
    fields=None,
    conditions=None,
    storage_class='STANDARD',
    server_side_encryption=None,
    )

form_fields = ""
line = '  <input type="hidden" name="%s" value="%s" >\n'
for item in args['fields']:
    new_line = line % (item["name"], item["value"])
    form_fields += new_line

html_page_data = {
"page_title": PAGE_TITLE,
"action": args["action"],
"input_fields": form_fields,
}

html_page = """
<html> 
 <head>
  <title>%(page_title)s</title> 
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 </head>
<body>
 <form action="%(action)s" method="post" enctype="multipart/form-data" >
%(input_fields)s
  <!-- Include any additional input fields here -->

  Browse to locate the file to upload:<br \> <br \>

  <input name="file" type="file"><br> <br \>
  <input type="submit" value="Upload File to S3"> 
 </form> 
</body>
</html>
""" % html_page_data

with open(HTML_NAME, "wb") as f:
    f.write(html_page)

success_page = """
<html>
  <head>
    <title>S3 POST Success Page</title>
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
      <script src="jquery.js"></script>
      <script src="purl.js"></script>
<!--

    Amazon S3 passes three data items in the url of this page if
        the upload was successful:
        bucket = bucket name
        key = file name upload to the bucket
        etag = hash of file

    The following script parses these values and puts them in
    the page to be displayed.

-->

<script type="text/javascript">
var pname,url,val,params=["bucket","key","etag"];
$(document).ready(function()
{
  url = $.url();
  for (param in params)
  {
    pname = params[param];
    val = url.param(pname);
    if(typeof val != 'undefined')
      document.getElementById(pname).value = val;
  }
});
</script>

  </head>
  <body>
      <div style="margin:0 auto;text-align:center;">
      <p>Congratulations!</p>
      <p>You have successfully uploaded the file.</p>
        <form action="#" method="get"
          >Location:
        <br />
          <input type="text" name="bucket" id="bucket" />
        <br />File Name:
        <br />
          <input type="text" name="key" id="key" />
        <br />Hash:
        <br />
          <input type="text" name="etag" id="etag" />
      </form>
    </div>
  </body>
</html>
"""

with open(SUCCESS_NAME, "wb") as f:
    f.write(success_page)

###### Dump output if testing ######
if DEBUG:

    if 1: # Set true if not using the LEO editor
        class G:
            def es(self, data):print(data)
        g = G()

    g.es("conn = %s" % conn)
    for key in args.keys():
        if key is not "fields":
            g.es("%s: %s" % (key, args[key]))
            continue
        for item in args['fields']:
            g.es(item)

这篇关于直接上传到S3使用Python / Boto / Django来构建策略的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！