如何将 POST 数据流式传输到 Python 请求中? [英] How to stream POST data into Python requests?

问题描述

我正在使用 Python requests 库来发送 POST 请求.生成 POST 数据的程序部分可以写入到任意的类文件对象(输出流)中.

I'm using the Python requests library to send a POST request. The part of the program that produces the POST data can write into an arbitrary file-like object (output stream).

我怎样才能让这两个部分适合?

How can I make these two parts fit?

我本来希望 requests 为这个用例提供一个流接口，但它似乎没有.它只接受一个类似文件的对象作为 data 参数，它读取.它不提供我可以写入的类似文件的对象.

I would have expected that requests provides a streaming interface for this use case, but it seems it doesn't. It only accepts as data argument a file-like object from which it reads. It doesn't provide a file-like object into which I can write.

这是 Python HTTP 库的基本问题吗?

Is this a fundamental issue with the Python HTTP libraries?

似乎最简单的解决方案是 fork() 并让请求库通过 管道 与 POST 数据生产者通信.

It seems that the simplest solution is to fork() and to let the requests library communicate with the POST data producer throgh a pipe.

有更好的方法吗?

或者，我可以尝试使 POST 数据生成器复杂化.然而，那是解析一个 XML 流(来自 stdin)并生成一个新的 XML 流以用作 POST 数据.然后我反过来遇到了同样的问题:XML 序列化程序库想要写入到一个类似文件的对象中，我不知道 XML 序列化程序提供一个类似文件的对象的任何可能性，从其他人可以阅读.

Alternatively, I could try to complicate the POST data producer. However, that one is parsing one XML stream (from stdin) and producing a new XML stream to used as POST data. Then I have the same problem in reverse: The XML serializer libraries want to write into a file-like object, I'm not aware of any possibility that an XML serializer provides a file-like object from which other can read.

我也知道最简洁、经典的解决方案是协程，它在 Python 中通过生成器(yield)有些可用.POST 数据可以通过 (yield) 而不是类似文件的对象进行流式传输，并使用 pull-parser.

I'm also aware that the cleanest, classic solution to this is coroutines, which are somewhat available in Python through generators (yield). The POST data could be streamed through (yield) instead of a file-like object and use a pull-parser.

但是，是否可以让 requests 接受 POST 数据的迭代器?是否有可以很容易地与 yield 结合使用的 XML 序列化程序?

However, is possible to make requests accept an iterator for POST data? And is there an XML serializer that can readily be used in combination with yield?

或者，是否有任何包装对象可以将写入文件类对象转换为生成器，和/或提供包装迭代器的类文件对象?

Or, are there any wrapper objects that turn writing into a file-like object into a generator, and/or provide a file-like object that wraps an iterator?

推荐答案

request 确实将迭代器或生成器作为 data 参数，详细信息在 块编码请求.在这种情况下，传输编码需要分块，因为事先不知道数据大小.

request does take an iterator or generator as data argument, the details are described in Chunk-Encoded Requests. The transfer encoding needs to be chunked in this case because the data size is not known beforehand.

这是一个使用 队列的非常简单的示例.队列，可以作为类似文件的对象进行写入:

Here is a very simle example that uses a queue.Queue and can be used as a file-like object for writing:

import requests
import queue
import threading

class WriteableQueue(queue.Queue):

    def write(self, data):
        # An empty string would be interpreted as EOF by the receiving server
        if data:
            self.put(data)

    def __iter__(self):
        return iter(self.get, None)

    def close(self):
        self.put(None)

# quesize can be limited in case producing is faster then streaming
q = WriteableQueue(100)

def post_request(iterable):
    r = requests.post("http://httpbin.org/post", data=iterable)
    print(r.text)

threading.Thread(target=post_request, args=(q,)).start()

# pass the queue to the serializer that writes to it ...    
q.write(b'1...')
q.write(b'2...')

# closing ends the request
q.close()

这篇关于如何将 POST 数据流式传输到 Python 请求中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！