通过API发送大文件到GAE Blob存储（从S3） [英] Send large files to GAE Blobstore via API (from S3)

问题描述

我试图发送大文件（50MB-2GB）采用filepicker.io到谷歌的App Engine Blob存储（在Python），我已存储在S3。我想这样的事情发生，而无需通过客户端浏览器的一种形式，因为它违背了项目需求（而且往往挂有非常大的文件）。

I'm trying to send large files (50MB-2GB) that I have stored in S3 using filepicker.io to the Google App Engine Blobstore (in Python). I would like this to happen without going through a form on the client browser as it defeats the project requirements (and often hangs with very large files).

我试过了多种解决方案，包括：

I tried multiple solutions, including:

• 尝试<一href="http://stackoverflow.com/questions/12149315/how-do-i-integrate-filepicker-io-with-google-app-engine-blobstore">load文件到GAE与urlfetch （但<一个href="http://stackoverflow.com/questions/6898736/upload-file-bigger-than-40mb-to-google-app-engine/6899157#6899157">GAE拥有32MB的限制的请求/响应）
• 在Python中构建一个多部分的形式，并把它发送到 blobstore.create_upload_url（）（不能传输通过URL的文件，无法加载它因为32MB限制脚本）
• 使用<一个href="http://stackoverflow.com/questions/3948391/is-is-possible-to-read-a-file-from-s3-in-google-app-engine-using-boto">boto读取文件直入Blob存储（连接超时，日志显示遇到HTTPException异常从博托触发 CancelledError：API调用logservice.Flush（）被明确取消了。从GAE的崩溃过程。

• trying to load the file into GAE with urlfetch (but GAE has a 32MB limit for requests/responses)
• constructing a multi-part form in python and sending it to blobstore.create_upload_url() (can't transfer the file via url, and can't load it in the script because of the 32MB limit)
• using boto to read the file straight into the blobstore (connection times out and logs show encountered HTTPException exception from boto that triggers CancelledError: The API call logservice.Flush() was explicitly cancelled. from GAE that crashes the process.

我在努力寻找一个有效的解决方案。我如何能执行此转让，或如何从S3作为一种形式的附件传递文件，而不先加载它在Python（即只指定其URL）的任何暗示将非常AP preciated。

I am struggling to find a working solution. Any hints on how I could perform this transfer, or how to pass the file from s3 as a form attachment without loading it in python first (ie. just specifying its url) would be very much appreciated.

推荐答案

该BlobstoreUploadHandler不受制于32MB的限制：的https://developers.google.com/appengine/docs/python/tools/webapp/blobstorehandlers.但是，我不知道这是如何融入你的应用程序。如果你能发布该文件由BlobstoreUploadHandler处理的终点，那么你应该是好去。

The BlobstoreUploadHandler isn't constrained by a 32MB limit: https://developers.google.com/appengine/docs/python/tools/webapp/blobstorehandlers. However, I'm not sure how this might fit into your app. If you can post the file to an endpoint handled by a BlobstoreUploadHandler, then you should be good to go.

这篇关于通过API发送大文件到GAE Blob存储（从S3）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！