直接用户上传到GCS与给定的名称和参数 [英] Direct user upload to GCS with given name and parameters

问题描述

我希望启用我的Python GAE应用程序的客户端，使用我设置的特定对象名称和GCS参数（如缓存控制和ACL）将文件直接上传到GCS，并在上传完成后执行一些操作。

我目前使用 blobstore.create_upload_url 来做这件事。然而，由于我不能提供任何参数或对象名称，我需要将整个文件从'上传'回调处理程序复制到GCS上的新位置。这是时间和（计算）资源消耗，听起来像它可以避免。

有什么方法可以优化这个过程？

• 我可以创建一个上传URL，立即上传一个具有正确名称和正确ACL +缓存控制参数的对象吗？ （我简单地看过签名的URL，但这些似乎不支持像blobstore API那样的回调处理程序）


• 如果没有，是否有办法不必复制文件逐字节地址到新位置并在完成后删除它？ （能够移动文件并更改其属性会很好）

更新：我刚刚注意到GCS python客户端库现在有 copy2 。

https://github.com/voscausa/appengine-gcs-upload =nofollow>这是我的Python，webapp2，jinja代码，用于使用带签名的表单直接上传（发布）到GCS一份政策文件：

  def gcs_upload（acl ='bucket-owner-read'）：
jinja2上传表单上下文
 
 default_bucket = app_identity.get_default_gcs_bucket_name（）
 google_access_id = app_identity.get_service_account_name（）
 succes_redirect = webapp2.uri_for（'_ gcs_upload_ok'，_full = True） 
 expiration_dt = datetime.now（）+ timedelta（seconds = 300）
 
 policy_string =
 {expiration：％s，
条件：[
 [starts-with，$ key，]，
 {acl：％s}，
 {success_action_redirect： s}，
 {success_action_status：201}，
]}％（expiration_dt.replace（微秒= 0）.isoformat（）+'Z'，acl，succes_redirect）
 
 policy = base64.b64encode（policy_string）
_，signature_bytes = app_identity.sign_blob（policy）
 signature = base64.b64encode（signature_bytes）
 
 return dict（form_bucket = default_bucket，form_access_id = google_access_id，form_policy = policy，form_signature =签名，
 form_succes_redirect = succes_redirect，form_acl = acl）
 
 
 class GcsUpload（BaseHandler）：
 
 def get（self）：
 
 self.render_template（'gcs_upload.html'，** gcs_upload（））
  

$ b $

 < form method =postaction =https： //storage.googleapis.com / {{form_bucket}}enctype =multipart / form-data> 
< input type =hiddenid =keyname =keyvalue =images / test.png> 
< input type =hiddenname =GoogleAccessIdvalue ={{form_access_id}}> 
< input type =hiddenname =aclvalue ={{form_acl}}> 
< input type =hiddenname =success_action_redirectvalue ={{form_succes_redirect}}> 
< input type =hiddenname =success_action_statusvalue =201> 
< input type =hiddenname =policyvalue ={{form_policy}}> 
< input type =hiddenname =signaturevalue ={{form_signature}}> 
< input name =filetype =file> 
< input type =submitvalue =上传> 
< / form> 
  

Google文件使用表单这里

I want to enable clients of my Python GAE app to upload files directly to GCS using a specific object name and GCS parameters I set up front (such as cache control and ACL), and perform some actions after the upload has finished.

I currently use the blobstore.create_upload_url for this. However, since I cannot give any parameters or object name up front, I need to copy the entire file to a new location on GCS from within the 'uploaded' callback handler. This is time and (compute) resource consuming, and sounds like it could be avoided.

Is there any way to optimize this process?

• Can I create an upload URL that will immediately upload an object with the right name and right ACL + Cache control parameters? (I briefly looked at signed URLs, but those don't seem to support a callback handler like the blobstore API does)
• If not, is there a way to not have to copy the file byte by byte to the new location and delete it when i'm done? (being able to move a file and change its attributes would be nice)

Update: I just noticed the GCS python client library has copy2 now. I could have sworn this wasn't there before, but it sounds like it answers the second question.

解决方案

Here is my Python, webapp2, jinja code for a direct upload (post) to GCS using a signed form with a policy document:

def gcs_upload(acl='bucket-owner-read'):
    """ jinja2 upload form context """

    default_bucket = app_identity.get_default_gcs_bucket_name()
    google_access_id = app_identity.get_service_account_name()
    succes_redirect = webapp2.uri_for('_gcs_upload_ok', _full=True)
    expiration_dt = datetime.now() + timedelta(seconds=300)

    policy_string = """
    {"expiration": "%s",
              "conditions": [
                  ["starts-with", "$key", ""],
                  {"acl": "%s"},
                  {"success_action_redirect": "%s"},
                  {"success_action_status": "201"},
              ]}""" % (expiration_dt.replace(microsecond=0).isoformat() + 'Z', acl, succes_redirect)

    policy = base64.b64encode(policy_string)
    _, signature_bytes = app_identity.sign_blob(policy)
    signature = base64.b64encode(signature_bytes)

    return dict(form_bucket=default_bucket, form_access_id=google_access_id, form_policy=policy, form_signature=signature,
            form_succes_redirect=succes_redirect, form_acl=acl)


class GcsUpload(BaseHandler):

    def get(self):

        self.render_template('gcs_upload.html', **gcs_upload())

And the form fields:

<form method="post" action="https://storage.googleapis.com/{{ form_bucket }}" enctype="multipart/form-data">
    <input type="hidden" id="key" name="key" value="images/test.png">
    <input type="hidden" name="GoogleAccessId" value="{{ form_access_id }}">
    <input type="hidden" name="acl" value="{{ form_acl }}">
    <input type="hidden" name="success_action_redirect" value="{{ form_succes_redirect }}">
    <input type="hidden" name="success_action_status" value="201">
    <input type="hidden" name="policy" value="{{ form_policy }}">
    <input type="hidden" name="signature" value="{{ form_signature }}">
    <input name="file" type="file">
    <input type="submit" value="Upload">
</form>

Docs GCS upload using a form here

这篇关于直接用户上传到GCS与给定的名称和参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！