urllib2和httplib线程安全吗? [英] Are urllib2 and httplib thread safe?

问题描述

我正在寻找有关urllib2和httplib的线程安全性的信息. 官方文档( http://docs.python.org/library/urllib2.html和 http://docs.python.org/library/httplib.html )缺乏有关此主题的任何信息；甚至没有提到 thread 这个词...

更新

好的，它们不是开箱即用的线程安全的. 使它们具有线程安全性需要什么，或者存在可以使它们成为线程安全性的情况? 我问是因为好像是这样

• 在每个线程中使用单独的 OpenerDirector
>
• 不共享HTTP连接 线程

足以在线程中安全地使用这些库.在问题 urllib2和cookielib线程安全性

中提出了类似的使用方案.

解决方案

httplib和urllib2是不是线程安全的.

urllib2不提供对全局(共享)的序列化访问 OpenerDirector对象，由urllib2.urlopen()使用.

类似地，httplib不提供对HTTPConnection对象的序列化访问(即，通过使用线程安全的连接池)，因此在线程之间共享HTTPConnection对象是不安全的.

我建议使用 httplib2 或http://docs.python.org/library/urllib2.html and http://docs.python.org/library/httplib.html) lacks any information on this subject; the word thread is not even mentioned there...

UPDATE

Ok, they are not thread-safe out of the box. What's required to make them thread-safe or is there a scenario in which they can be thread-safe? I'm asking because it's seems that

• using separate OpenerDirector in each thread
• not sharing HTTP connection among threads

would suffice to safely use these libs in threads. Similar usage scenario was proposed in question urllib2 and cookielib thread safety

解决方案

httplib and urllib2 are not thread-safe.

urllib2 does not provide serialized access to a global (shared) OpenerDirector object, which is used by urllib2.urlopen().

Similarly, httplib does not provide serialized access to HTTPConnection objects (i.e. by using a thread-safe connection pool), so sharing HTTPConnection objects between threads is not safe.

I suggest using httplib2 or urllib3 as an alternative if thread-safety is required.

Generally, if a module's documentation does not mention thread-safety, I would assume it is not thread-safe. You can look at the module's source code for verification.

When browsing the source code to determine whether a module is thread-safe, you can start by looking for uses of thread synchronization primitives from the threading or multiprocessing modules, or use of queue.Queue.

UPDATE

Here is a relevant source code snippet from urllib2.py (Python 2.7.2):

_opener = None
def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
    global _opener
    if _opener is None:
        _opener = build_opener()
    return _opener.open(url, data, timeout)

def install_opener(opener):
    global _opener
    _opener = opener

There is an obvious race condition when concurrent threads call install_opener() and urlopen().

Also, note that calling urlopen() with a Request object as the url parameter may mutate the Request object (see the source for OpenerDirector.open()), so it is not safe to concurrently call urlopen() with a shared Request object.

All told, urlopen() is thread-safe if the following conditions are met:

• install_opener() is not called from another thread.
• A non-shared Request object, or string is used as the url parameter.

这篇关于urllib2和httplib线程安全吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！