如何将POST数据流式传输到Python请求中？ [英] How to stream POST data into Python requests?
问题描述
我正在使用Python requests 库来发送POST请求。生成POST数据的程序部分可以写入到任意类文件对象（输出流）中。
我该如何制作这两个部分适合吗？
我原本预计请求为这个用例提供了一个流接口，但它似乎没有。它只接受 data 参数作为读取的文件类对象。它没有提供类似文件的对象，我可以写。
这是Python HTTP库的一个基本问题吗？ / p>
到目前为止的想法：
似乎最简单的解决方案是 fork（） 并让请求库与POST数据生成器进行通信 pipe 。
有没有更好的方法？
或者，我可以尝试使POST数据生成器复杂化。但是，那个解析一个XML流（来自stdin）并生成一个新的XML流用作POST数据。然后我有相反的问题：XML序列化程序库想要写到类文件对象中，我不知道XML序列化程序提供类似文件的对象的可能性其他可以阅读。
我也知道最干净，最经典的解决方案是coroutines，它在Python中可以通过生成器获得（产量）。 POST数据可以通过（ yield ）而不是类似文件的对象流式传输，并使用拉式解析器。
但是，是否可以使请求接受POST数据的迭代器？是否有一个XML序列化器可以很容易地与 yield 组合使用？
或者，是否有任何包装器将写入文件的对象转换为生成器的对象，和/或提供包装迭代器的类文件对象？
request 确实将迭代器或生成器作为 data 参数，详细信息在 Chunk-Encoded Requests 。在这种情况下，传输编码需要分块，因为事先不知道数据大小。
这是一个非常简单的例子，使用 queue.Queue 可以使用作为一个类似文件的文件对象：
导入请求
导入队列
导入线程
class WriteableQueue（queue.Queue）：
def write（self，data）：
＃接收服务器将空字符串解释为EOF
if data：
self.put（data）
def __iter __（self）：
return iter（self.get，None）
def close （个体经营）：
self.put（无）
＃quesize可限制，以防生产速度快于流式传输
q = WriteableQueue（100）
def post_request（iterable）：
r = requests.post（http://httpbin.org/post，data = iterable）
print（r.text ）
threading.Thread（target = post_request，args =（q，））。start（）
＃将队列传递给写入它的串行器...
q.write（b'1 ...'）
q.write（b'2 ...'）
＃closing结束请求
q .close（）
I'm using the Python requests library to send a POST request. The part of the program that produces the POST data can write into an arbitrary file-like object (output stream).
How can I make these two parts fit?
I would have expected that requests provides a streaming interface for this use case, but it seems it doesn't. It only accepts as data argument a file-like object from which it reads. It doesn't provide a file-like object into which I can write.
Is this a fundamental issue with the Python HTTP libraries?
Ideas so far:
It seems that the simplest solution is to fork() and to let the requests library communicate with the POST data producer throgh a pipe.
Is there a better way?
Alternatively, I could try to complicate the POST data producer. However, that one is parsing one XML stream (from stdin) and producing a new XML stream to used as POST data. Then I have the same problem in reverse: The XML serializer libraries want to write into a file-like object, I'm not aware of any possibility that an XML serializer provides a file-like object from which other can read.
I'm also aware that the cleanest, classic solution to this is coroutines, which are somewhat available in Python through generators (yield). The POST data could be streamed through (yield) instead of a file-like object and use a pull-parser.
However, is possible to make requests accept an iterator for POST data? And is there an XML serializer that can readily be used in combination with yield?
Or, are there any wrapper objects that turn writing into a file-like object into a generator, and/or provide a file-like object that wraps an iterator?
request does take an iterator or generator as data argument, the details are described in Chunk-Encoded Requests. The transfer encoding needs to be chunked in this case because the data size is not known beforehand.
Here is a very simle example that uses a queue.Queue and can be used as a file-like object for writing:
import requests
import queue
import threading
class WriteableQueue(queue.Queue):
def write(self, data):
# An empty string would be interpreted as EOF by the receiving server
if data:
self.put(data)
def __iter__(self):
return iter(self.get, None)
def close(self):
self.put(None)
# quesize can be limited in case producing is faster then streaming
q = WriteableQueue(100)
def post_request(iterable):
r = requests.post("http://httpbin.org/post", data=iterable)
print(r.text)
threading.Thread(target=post_request, args=(q,)).start()
# pass the queue to the serializer that writes to it ...
q.write(b'1...')
q.write(b'2...')
# closing ends the request
q.close()
这篇关于如何将POST数据流式传输到Python请求中？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
