大量文件下载在cherrypy [英] Large file downloads in cherrypy
问题描述
cherrypy.config.update（ {'tools.sessions.timeout'：1000000}）但是这并不能真正解决问题，因为它是一个糟糕的恶作剧解决方案，并没有真正的工作。它主要只是导致另一个问题，使超时非常大。此外，浏览器无法估计需要非常准确的时间，最终会挂起一段时间（阅读：硬连线上的5分钟以上），然后快速开始下载。 起始于
然后转到
我的下载代码很简单，只是由这一行。
return cherrypy.lib.static.serve_file（path，application / x-download，os.path.basename（path））
我以前的下载代码并没有很好的解决。
f = file（path）
cherrypy.response.headers ['Content-Type'] = getType（path）[0]
return f
有没有办法解决这个问题？
一般考虑
首先，我不得不说这是一个堆叠的配置， - > uWSGI - > Nginx ，这样一个有限的环境。 根据作者，可以自行使用CherryPy 来进行小尺寸应用程序，当没有特殊要求的时候。在前面添加Nginx 增加了很多灵活性，所以通常是有益的，但只要CherryPy 的默认部署是标准HTTP，我强烈建议留下两个（并忘记了WSGI）。
其次，考虑到您尝试的解决方法，您可能已经知道您的问题可能与会话相关。有关流媒体响应的文档中的报价身体哪个文件下载是。
一般来说，流输出更安全，更容易。因此，默认情况下，流输出为
。流输出和使用会话需要一个很好的理解
的会话锁如何工作。
它是什么意思是手动会话锁管理。了解您的应用程序的工作原理应该导致您进行适当的锁定设计。
第三。通常有一种方法可以将处理文件下载的义务转移到Web服务器，基本上是通过从代理的应用程序发送带有文件名的适当的头文件。如果是 nginx ，则称为 X-accel code> 。所以你可以避免锁管理的麻烦，还有会话限制下载。
实验
做了一个简单的CherrPy 应用程序与两个下载选项，并把它放在了Nginx 后面。我在Firefox 和 Chromium 之间的本地Linux 机器上播放了1.3GiB视频文件。有三种方式：
- 从CherryPy（
http://127.0.0.1:8080/ native / video.mp4）， - 从CherryPy通过Nginx进行代理下载（
http：//test/native/video.mp4）， - 从CherryPy通过Nginx下载X-accel（
http：//test/nginx/video.mp4 上运行几天，我不断有〜5MiB / s的下载速度和一个满载的CPU内核。在新鲜的 Firefox 中，没有这样的行为。 （2）在Chromium上导致了一些未完成的中断下载（大约1GiB）。但是一般来说，两台浏览器都能显示出50-70MiB / s的HDD物理性能。
（3）我没有问题，同样的50-70MiB / s吞吐量，所以在我的小实验中，它最终成为最稳定的方式。
设置
app.py
＃！/ usr / bin / env python
＃ - * - 编码： utf-8 - * -
import os
import cherrypy
DownloadPath ='/ home / user /视频'
config = {
'global'：{
'server.socket_host'：'127.0.0.1'，
'server.socket_port'：8080，
'server.thread_pool'：8
}
}
类App：
@ cherrypy.expose
def index（self）：
return'下载测试'
@ cherrypy.expose
def native（self，name）：
basename = os.path .basename（name）
filename = os.path.join（DownloadPath，basename）
mime ='application / octet-strea m'
return cherrypy.lib.static.serve_file（filename，mime，basename）
@ cherrypy.expose
def nginx（self，name）：
basename = os.path.basename（name）
cherrypy.response.headers.update（{
'X-Accel-Redirect'：'/download/{0}'.format(basename），
'Content-Disposition'：'附件; filename = {0}'。format（basename），
'Content-Type'：'application / octet-stream'
}）
如果__name__ = ='__main__'：
cherrypy.quickstart（App（），'/'，config）
app.conf
server {
listen 80;
server_name test;
root / var / www / test / public;
位置/资源{
＃静态文件像图像，css，js等
access_log off;
}
位置/ {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $ host;
proxy_set_header X-Real-IP $ remote_addr;
proxy_set_header X-Forwarded-For $ proxy_add_x_forwarded_for;
}
位置/下载{
内部;
alias / home / user / Videos;
}
}
I'm hosting a file access type website using Cherrypy, through uwsgi and nginx on a Raspberry Pi. One thing I've noticed is that if the file is rather large (let's say, about a gigabyte), uwsgi says it was killed by signal 9. This was remedied by putting a
cherrypy.config.update({'tools.sessions.timeout': 1000000})but this doesn't really solve the problem, as much as it is a bad hacky workaround that doesn't really work. It mainly just causes another problem by making the timeout very large. In addition, the browser cannot estimate how long it will take very accurately, and will end up hanging for a while (Read: 5 or so mins on a hardwired connection), and then rapidly starts downloading.It starts as
Then goes to
My download code is very simple, just consisting of this single line.
return cherrypy.lib.static.serve_file(path,"application/x-download",os.path.basename(path))My previous download code didn't quite work out well.
f = file(path) cherrypy.response.headers['Content-Type'] = getType(path)[0] return fIs there a way to remedy this?解决方案General consideration
First, of all I have to say it's such a piled up configuration, CherryPy -> uWSGI -> Nginx, for such a constrained environment. According to the author, it's safe to use CherryPy on its own for small-size applications, when there's no special requirement. Adding Nginx in front adds a lot of flexibility, so it's usually beneficial, but as long as CherryPy's default deployment is standard HTTP, I strongly suggest to stay with the two (and forget about WSGI altogether).
Second, you probably already know that your problem is likely session-related, considering the workaround you've tried. Here's the quote from documentation about streaming response body which file download is.
In general, it is safer and easier to not stream output. Therefore, streaming output is off by default. Streaming output and also using sessions requires a good understanding of how session locks work.
What it suggests is manual session lock management. Knowing how your application works should lead you to appropriate lock design.
And third. There's usually a way to shift the duty of handling a file download to a web-server, basically by sending appropriate header with filename from the proxied application. In case on nginx it's called
X-accel. So you can avoid the hassle of lock management, still having session restricted downloads.Experiment
I've made a simple CherrPy app with two download options and putted it behind Nginx. I played with 1.3GiB video file on local Linux machine in Firefox and Chromium. There were three ways:
- Un-proxied download from CherryPy (
http://127.0.0.1:8080/native/video.mp4), - Proxied download from CherryPy via Nginx (
http://test/native/video.mp4), - X-accel download from CherryPy via Nginx (
http://test/nginx/video.mp4).
With (1) and (2) I had minor strange behaviour in both Firefox and Chromium. (1) on Firefox with uptime of several days I constantly had ~5MiB/s download speed and one full-loaded CPU core. On fresh Firefox there was no such behaviour. (2) on Chromium resulted in a couple of unfinished interrupted downloads (all times around 1GiB). But in general both browsers showed around HDD physical performance of 50-70MiB/s.
With (3) I had no issue in both, same 50-70MiB/s throughput, so somehow in my small experiment it ended up as the most stable way.
Setup
app.py
#!/usr/bin/env python # -*- coding: utf-8 -*- import os import cherrypy DownloadPath = '/home/user/Videos' config = { 'global' : { 'server.socket_host' : '127.0.0.1', 'server.socket_port' : 8080, 'server.thread_pool' : 8 } } class App: @cherrypy.expose def index(self): return 'Download test' @cherrypy.expose def native(self, name): basename = os.path.basename(name) filename = os.path.join(DownloadPath, basename) mime = 'application/octet-stream' return cherrypy.lib.static.serve_file(filename, mime, basename) @cherrypy.expose def nginx(self, name): basename = os.path.basename(name) cherrypy.response.headers.update({ 'X-Accel-Redirect' : '/download/{0}'.format(basename), 'Content-Disposition' : 'attachment; filename={0}'.format(basename), 'Content-Type' : 'application/octet-stream' }) if __name__ == '__main__': cherrypy.quickstart(App(), '/', config)app.conf
server { listen 80; server_name test; root /var/www/test/public; location /resource { # static files like images, css, js, etc. access_log off; } location / { proxy_pass http://127.0.0.1:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location /download { internal; alias /home/user/Videos; } }这篇关于大量文件下载在cherrypy的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
- Un-proxied download from CherryPy (
