大文件上传失败 [英] Large file upload fails

问题描述

我正在编写一个 python 模块来将文件 POST 到服务器，我可以上传最大 500MB 的文件但是当我尝试上传 1gb 文件时上传失败，如果我要使用类似的东西卷曲它不会失败.我在谷歌搜索如何使用 python 上传 multipart formdata 后得到了代码，代码可以在 这里.我刚刚编译并运行了该代码，我得到的错误是这个

I'm in the process of writing a python module to POST files to a server , I can upload files of size of upto 500MB but when I tried to upload a 1gb file the upload failed, If I were to use something like cURL it won't fail. I got the code after googling how to upload multipart formdata using python , the code can be found here. I just compiled and ran that code , the error I'm getting is this

Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    opener.open("http://127.0.0.1/test_server/upload",params)
  File "C:\Python27\lib\urllib2.py", line 392, in open
    req = meth(req)
  File "C:\Python27\MultipartPostHandler.py", line 35, in http_request
    boundary, data = self.multipart_encode(v_vars, v_files)
  File "C:\Python27\MultipartPostHandler.py", line 63, in multipart_encode
    buffer += '\r\n' + fd.read() + '\r\n'  
MemoryError

我是 Python 新手，很难掌握它.我还遇到了另一个程序 here ，老实说我不知道​​如何运行.我尝试通过根据函数名称猜测来运行它，但这不起作用.

I'm new to python and having a hard time grasping it. I also came across another program here , I'll be honest I don't know how to run it. I tried running it by guessing based on the function name , but that didn't work.

推荐答案

有问题的脚本不是很聪明，而是在内存中构建 POST 主体.

The script in question isn't very smart and builds the POST body in memory.

因此，要 POST 一个 1GB 的文件，您需要 1GB 的内存来保存该数据，以及 HTTP 标头、边界、python 和代码本身.

Thus, to POST a 1GB file, you'll need 1GB of memory just to hold that data, plus the HTTP headers, boundaries, and python and the code itself.

您必须重新编写脚本以改用 mmap，首先在临时文件中构造整个主体，然后将该文件包裹在 mmap.mmap 值中，然后将其传递给 <代码>request.add_data.

You'd have to rework the script to use mmap instead, where you first construct the whole body in a temp file before handing that file wrapped in a mmap.mmap value to passing it to request.add_data.

请参阅 Python:HTTP 使用流式传输大文件 有关如何实现这一目标的提示.

See Python: HTTP Post a large file with streaming for hints on how to achieve that.

这篇关于大文件上传失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！