使用瓶子（或烧瓶或类似物）流式传输文件 [英] Streaming file upload using bottle (or flask or similar)

问题描述

我有一个使用Python / Bottle编写的REST前端，它处理文件上传，通常是大文件上传。这个API是以这样一种方式进行的：

客户端发送PUT文件作为有效载荷。除此之外，它还发送日期和授权标头。这是一个针对重播攻击的安全措施 - 请求是用临时密钥，使用目标url，日期和其他一些东西来烧录的。

现在出现这个问题。如果提供的日期在15分钟的给定日期时间窗口中，则服务器接受请求。如果上传时间足够长，则会比允许的时间增量长。现在，请求授权处理是使用装饰者在瓶子视图方法上完成的。然而，瓶子不会启动调度过程，除非上传完成，所以验证失败的更长的上传。

我的问题是：有没有办法解释瓶或WSGI立即处理请求和流传输上传？由于其他原因，这对我也很有用。还是其他解决方案？在写这篇文章的时候，我想到了WSGI中间件，但是仍然需要外部的洞察力。

我愿意切换到Flask，甚至是其他Python框架，因为REST前端是相当轻量级的。

谢谢

解决方案

我建议将传入的文件分割成前端的较小的块。我这样做是为了在Flask应用程序中实现大文件上传的暂停/恢复功能。

使用 maxChunkSize 来实现分块（chunking） ，如：

  $（'＃file-select'）。fileupload（{
 url：'/ uploads /'，
 sequentialUploads：true，
 done：function（e，data）{
 console.log（uploaded：+ data.files [0] .name）
 }，
 maxChunkSize：1000000 // 1 MB 
}）; 
  

现在客户端在上传大文件时会发送多个请求。而您的服务器端代码可以使用 Content-Range 标题将原始大文件修补到一起。对于Flask应用程序，视图可能如下所示：

 ＃上传文件
 @ app.route（'/ uploads /'，methods = ['POST']）
 def results（）：
 
 files = request.files 
 
＃假设只有一个文件被传入请求
 key = files.keys（）[0] 
 value = files [key]＃这是一个Werkzeug FileStorage对象
 filename = value.filename 
 
如果request.headers中的'Content-Range'：
＃从Content-Range头字符串中提取起始字节
 range_str = request.headers ['Content-Range'] 
 start_bytes = int（range_str .blit（''）[1] .split（' - '）[0]）
 
将块添加到磁盘上的文件，或者创建新的
打开（文件名， a'）作为f：
 f.seek（start_bytes）
 f.write（value.stream.read（））
 
 else：
＃这不是一个分块的请求，所以只需保存整个文件
 value.save （文件名）
 $ b $发送适当的MIME类型头的响应
返回jsonify（{name：value.filename，
size：os.path.getsize（filename ），
url：'uploads /'+ value.filename，
thumbnail_url：None，
delete_url：None，
delete_type：None，} ）
  

对于特定的应用程序，您只需确保仍然发送了正确的auth头文件与每个请求。

希望这有助于！我一直在努力解决这个问题;）

I have a REST frontend written using Python/Bottle which handles file uploads, usually large ones. The API is wirtten in such a way that:

The client sends PUT with the file as a payload. Among other things, it sends Date and Authorization headers. This is a security measure against replay attacks -- the request is singed with a temporary key, using target url, the date and several other things

Now the problem. The server accepts the request if the supplied date is in given datetime window of 15 minutes. If the upload takes long enough time, it will be longer than the allowed time delta. Now, the request authorization handling is done using decorator on bottle view method. However, bottle won't start the dispatch process unless the upload is finished, so the validation fails on longer uploads.

My question is: is there a way to explain to bottle or WSGI to handle the request immediately and stream the upload as it goes? This would be useful for me for other reasons as well. Or any other solutions? As I am writing this, WSGI middleware comes to mind, but still, I'd like external insight.

I would be willing to switch to Flask, or even other Python frameworks, as the REST frontend is quite lightweight.

Thank you

解决方案

I recommend splitting the incoming file into smaller-sized chunks on the frontend. I'm doing this to implement a pause/resume function for large file uploads in a Flask application.

Using Sebastian Tschan's jquery plugin, you can implement chunking by specifying a maxChunkSize when initializing the plugin, as in:

$('#file-select').fileupload({
    url: '/uploads/',
    sequentialUploads: true,
    done: function (e, data) {
        console.log("uploaded: " + data.files[0].name)
    },
    maxChunkSize: 1000000 // 1 MB
});

Now the client will send multiple requests when uploading large files. And your server-side code can use the Content-Range header to patch the original large file back together. For a Flask application, the view might look something like:

# Upload files
@app.route('/uploads/', methods=['POST'])
def results():

    files = request.files

    # assuming only one file is passed in the request
    key = files.keys()[0]
    value = files[key] # this is a Werkzeug FileStorage object
    filename = value.filename

    if 'Content-Range' in request.headers:
        # extract starting byte from Content-Range header string
        range_str = request.headers['Content-Range']
        start_bytes = int(range_str.split(' ')[1].split('-')[0])

        # append chunk to the file on disk, or create new
        with open(filename, 'a') as f:
            f.seek(start_bytes)
            f.write(value.stream.read())

    else:
        # this is not a chunked request, so just save the whole file
        value.save(filename)

    # send response with appropriate mime type header
    return jsonify({"name": value.filename,
                    "size": os.path.getsize(filename),
                    "url": 'uploads/' + value.filename,
                    "thumbnail_url": None,
                    "delete_url": None,
                    "delete_type": None,})

For your particular application, you will just have to make sure that the correct auth headers are still sent with each request.

Hope this helps! I was struggling with this problem for a while ;)

这篇关于使用瓶子（或烧瓶或类似物）流式传输文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！