Django Filewrapper 内存错误服务大文件，如何流式传输 [英] Django Filewrapper memory error serving big files, how to stream

问题描述

我有这样的代码:

@login_required定义下载文件(请求):content_type = "应用程序/八位字节流"下载名称 = os.path.join(DATA_ROOT, "video.avi")使用 open(download_name, "rb") 作为 f:包装器 = FileWrapper(f, 8192)响应 = HttpResponse(包装器，content_type=content_type)response['Content-Disposition'] = '附件；文件名=blabla.avi'响应['内容长度'] = os.path.getsize(download_name)# response['Content-Length'] = _file.size返回响应

看来是可行的.但是，如果我下载更大的文件(例如~600MB)，我的内存消耗会增加 600MB.经过几次这样的下载后，我的服务器抛出:

<块引用>

内部服务器错误:/download/Traceback(最近一次调用最后一次):
文件"/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/exception.py",第 35 行，在内部response = get_response(request) File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/base.py",第 128 行，在 _get_response 中response = self.process_exception_by_middleware(e, request) File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/base.py",第 126 行，在 _get_response 中响应=wrapped_callback(request, *callback_args, **callback_kwargs) 文件/home/matous/.local/lib/python3.5/site-packages/django/contrib/auth/decorators.py"，第 21 行，在 _wrapped_view 中返回 view_func(request, *args, **kwargs) 文件/media/matous/89104d3d-fa52-4b14-9c5d-9ec54ceebebb/home/matous/phd/emoapp/emoapp/mainapp/views.py"，第 118 行，在下载文件中response = HttpResponse(wrapper, content_type=content_type) 文件/home/matous/.local/lib/python3.5/site-packages/django/http/response.py"，第 285 行，在 init 中self.content = content File "/home/matous/.local/lib/python3.5/site-packages/django/http/response.py",第 308 行，在内容中content = b''.join(self.make_bytes(chunk) for chunk in value) MemoryError

我做错了什么?是否可以以某种方式对其进行配置，以便在没有这种疯狂的内存存储的情况下从硬盘驱动器逐个流式传输它?

注意:我知道 Django 不应该提供大文件，但我正在寻找一种简单的方法来验证用户对任何提供的文件的访问权限.

解决方案

尝试使用 StreamingHttpResponse 代替，这会有所帮助，这正是您正在寻找的.

是否有可能以某种方式对其进行配置，以便在没有这种疯狂的内存存储的情况下从硬盘驱动器逐块流式传输?

导入操作系统从 django.http 导入 StreamingHttpResponse从 django.core.servers.basehttp 导入 FileWrapper #django <=1.8从 wsgiref.util 导入 FileWrapper #django >1.8@需要登录定义下载文件(请求):file_path = os.path.join(DATA_ROOT, "video.avi")文件名 = os.path.basename(file_path)块大小 = 8192响应 = StreamingHttpResponse(FileWrapper(open(file_path, 'rb'), chunk_size),content_type="应用程序/八位字节流")响应['内容长度'] = os.path.getsize(file_path)response['Content-Disposition'] = "附件；文件名=%s" % 文件名返回响应

这将分块流式传输您的文件，而无需将其加载到内存中；或者，您可以使用 FileResponse,

<块引用>

它是针对二进制优化的 StreamingHttpResponse 的子类文件.

I have code like this:

@login_required
def download_file(request):
    content_type = "application/octet-stream"
    download_name = os.path.join(DATA_ROOT, "video.avi")

    with open(download_name, "rb") as f:
        wrapper = FileWrapper(f, 8192)
        response = HttpResponse(wrapper, content_type=content_type)
    response['Content-Disposition'] = 'attachment; filename=blabla.avi'
    response['Content-Length'] = os.path.getsize(download_name)
    # response['Content-Length'] = _file.size
    return response

It seems that it works. However, If I download bigger file (~600MB for example) my memory consumption increase by this 600MB. After few such a downloads my server throws:

Internal Server Error: /download/ Traceback (most recent call last):
File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 35, in inner response = get_response(request) File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/base.py", line 128, in _get_response response = self.process_exception_by_middleware(e, request) File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/base.py", line 126, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/home/matous/.local/lib/python3.5/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view return view_func(request, *args, **kwargs) File "/media/matous/89104d3d-fa52-4b14-9c5d-9ec54ceebebb/home/matous/phd/emoapp/emoapp/mainapp/views.py", line 118, in download_file response = HttpResponse(wrapper, content_type=content_type) File "/home/matous/.local/lib/python3.5/site-packages/django/http/response.py", line 285, in init self.content = content File "/home/matous/.local/lib/python3.5/site-packages/django/http/response.py", line 308, in content content = b''.join(self.make_bytes(chunk) for chunk in value) MemoryError

What I am doing wrong? Is it possible to configure it somehow to stream it the piece by piece from hard-drive without this insane memory storage?

Note: I know that big files should not be served by Django, but I am looking for simple approach that allows to verify user access rights for any served file.

解决方案

Try to use StreamingHttpResponse instead, that will help, it is exactly what you are looking for.

Is it possible to configure it somehow to stream it the piece by piece from hard-drive without this insane memory storage?

import os
from django.http import StreamingHttpResponse
from django.core.servers.basehttp import FileWrapper #django <=1.8
from wsgiref.util import FileWrapper #django >1.8

@login_required
def download_file(request):
   file_path = os.path.join(DATA_ROOT, "video.avi")
   filename = os.path.basename(file_path)
   chunk_size = 8192
   response = StreamingHttpResponse(
       FileWrapper(open(file_path, 'rb'), chunk_size),
       content_type="application/octet-stream"
   )
   response['Content-Length'] = os.path.getsize(file_path)    
   response['Content-Disposition'] = "attachment; filename=%s" % filename
   return response

This will stream your file in chunks without loading it in memory; alternatively, you can use FileResponse,

which is a subclass of StreamingHttpResponse optimized for binary files.

这篇关于Django Filewrapper 内存错误服务大文件，如何流式传输的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！