Python:多处理和请求 [英] Python: multiprocessing and requests

问题描述

以下是我运行以使用并行触发 HTTP 请求的多处理的代码片段.在控制台上运行后，它被挂在requests.get(url)"上，既不继续也不抛出错误.

Following is the snippet of code I am running to use multiprocessing which fires HTTP request in parallel. After the running on console it is getting hung at "requests.get(url)" and neither proceeding ahead nor throwing an error.

def echo_100(q):
    ...
    print "before"
    r = requests.get(url)
    print "after"
    ...
    q.put(r)

q = multiprocessing.Queue()
p = multiprocessing.Process(target=echo_100,args=(q))
p.start()
p.join()
resp = q.get()

推荐答案

在 Mac OS 上，似乎存在一些从操作系统读取代理设置的错误.我不知道确切的细节，但是在使用多处理时它有时会导致请求挂起.您可以尝试通过完全禁用操作系统代理来规避该问题，如下所示:

On Mac OS, there seem to be some bugs reading proxy settings from the operating system. I don't know the exact details, but it sometimes causes requests to hang when using multiprocessing. You could try to circumvent the problem by disabling OS proxies entirely, like this:

session = requests.Session()
session.trust_env = False  # Don't read proxy settings from OS
r = session.get(url)

那为我修好了.

这篇关于Python:多处理和请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！