python 是否有一个简单的基于进程的并行映射? [英] Is there a simple process-based parallel map for python?

问题描述

我正在为python寻找一个简单的基于进程的并行映射，即一个函数

I'm looking for a simple process-based parallel map for python, that is, a function

parmap(function,[data])

这将在不同进程的 [data] 的每个元素上运行函数(好吧，在不同的内核上，但是 AFAIK，在 python 中的不同内核上运行东西的唯一方法是启动多个解释器)，并返回一个结果列表.

that would run function on each element of [data] on a different process (well, on a different core, but AFAIK, the only way to run stuff on different cores in python is to start multiple interpreters), and return a list of results.

这样的东西存在吗?我想要一些简单，所以一个简单的模块会很好.当然，如果没有这样的东西存在，我会满足于一个大图书馆:-/

Does something like this exist? I would like something simple, so a simple module would be nice. Of course, if no such thing exists, I will settle for a big library :-/

推荐答案

我好像你需要的是 multiprocessing.Pool() 中的映射方法:

ma​​p(func, iterable[, chunksize])

A parallel equivalent of the map() built-in function (it supports only
one iterable argument though). It blocks till the result is ready.

This method chops the iterable into a number of chunks which it submits to the 
process pool as separate tasks. The (approximate) size of these chunks can be 
specified by setting chunksize to a positive integ

例如，如果你想映射这个函数:

For example, if you wanted to map this function:

def f(x):
    return x**2

到 range(10)，你可以使用内置的 map() 函数:

to range(10), you could do it using the built-in map() function:

map(f, range(10))

或使用 multiprocessing.Pool() 对象的方法 map():

or using a multiprocessing.Pool() object's method map():

import multiprocessing
pool = multiprocessing.Pool()
print pool.map(f, range(10))

这篇关于python 是否有一个简单的基于进程的并行映射?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！