Python:类似`map`的东西可以在线程上工作 [英] Python: Something like `map` that works on threads

问题描述

我确定标准库中有类似的东西，但是看来我错了.

I was sure there was something like this in the standard library, but it seems I was wrong.

我有一堆要并行urlopen的URL.我想要类似内置的map函数，除了工作是由一堆线程并行完成的.

I have a bunch of urls that I want to urlopen in parallel. I want something like the builtin map function, except the work is done in parallel by a bunch of threads.

有一个好的模块可以做到这一点吗?

Is there a good module that does this?

推荐答案

有人建议我为此使用futures软件包.我尝试过，它似乎可以正常工作.

Someone recommended I use the futures package for this. I tried it and it seems to be working.

http://pypi.python.org/pypi/futures

这是一个例子:

"Download many URLs in parallel."

import functools
import urllib.request
import futures

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

def load_url(url, timeout):
    return urllib.request.urlopen(url, timeout=timeout).read()

with futures.ThreadPoolExecutor(50) as executor:
   future_list = executor.run_to_futures(
           [functools.partial(load_url, url, 30) for url in URLS])

这篇关于Python:类似`map`的东西可以在线程上工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！