python请求上传带有附加数据的大文件 [英] python requests upload large file with additional data

问题描述

我一直在寻找上传带有附加数据的大文件的方法，但似乎没有任何解决方案.要上传文件，我一直在使用此代码并且它在小文件上运行良好:

I've been looking around for ways to upload large file with additional data, but there doesn't seem to be any solution. To upload file, I've been using this code and it's been working fine with small file:

with open("my_file.csv", "rb") as f:
    files = {"documents": ("my_file.csv", f, "application/octet-stream")}
    data = {"composite": "NONE"}
    headers = {"Prefer": "respond-async"}
    resp = session.post("my/url", headers=headers, data=data, files=files)

问题是代码在发送之前加载了整个文件，上传大文件时会遇到MemoryError.环顾四周，流式传输数据的方式是设置

The problem is that the code loads the whole file up before sending, and I would run into MemoryError when uploading large files. I've looked around, and the way to stream data is to set

resp = session.post("my/url", headers=headers, data=f)

但我需要将 {"composite": "NONE"} 添加到数据中.否则，服务器将无法识别该文件.

but I need to add {"composite": "NONE"} to the data. If not, the server wouldn't recognize the file.

推荐答案

您可以使用 requests-toolbelt这样做:

import requests
from requests_toolbelt.multipart import encoder

session = requests.Session()
with open('my_file.csv', 'rb') as f:
    form = encoder.MultipartEncoder({
        "documents": ("my_file.csv", f, "application/octet-stream"),
        "composite": "NONE",
    })
    headers = {"Prefer": "respond-async", "Content-Type": form.content_type}
    resp = session.post(url, headers=headers, data=form)
session.close()

这将导致请求为您流式传输 multipart/form-data 上传.

This will cause requests to stream the multipart/form-data upload for you.

这篇关于python请求上传带有附加数据的大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！