在Python 3.2中并行执行for循环 [英] Perform a for-loop in parallel in Python 3.2

问题描述

可能重复:
如何并行化一个简单的python循环?

Possible Duplicate:
how do I parallelize a simple python loop?

我对Python(使用Python 3.2)很陌生，并且有一个关于并行化的问题.我有一个for循环，希望在Python 3.2中使用多重处理"并行执行:

I'm quite new to Python (using Python 3.2) and I have a question concerning parallelisation. I have a for-loop that I wish to execute in parallel using "multiprocessing" in Python 3.2:

def computation:    
    global output

    for x in range(i,j):
        localResult = ... #perform some computation as a function of i and j
        output.append(localResult)

总的来说，我想对i = 0到j = 100的范围执行此计算.因此，我想创建多个进程，每个进程都调用具有整个范围子域的函数计算".关于如何做到这一点的任何想法?有没有比使用多处理更好的方法?

In total, I want to perform this computation for a range of i=0 to j=100. Thus I want to create a number of processes that each call the function "computation" with a subdomain of the total range. Any ideas of how do to this? Is there a better way than using multiprocessing?

更具体地说，我想执行域分解，并且我有以下代码:

More specific, I want to perform a domain decomposition and I have the following code:

from multiprocessing import Pool

class testModule:

    def __init__(self):
        self

    def computation(self, args):
        start, end = args
        print('start: ', start, ' end: ', end)

testMod = testModule()
length = 100
np=4
p = Pool(processes=np)
p.map(yes tMod.computation, [(length, startPosition, length//np) for startPosition in    range(0, length, length//np)]) 

我收到一条提及PicklingError的错误消息.有什么想法可能是这里的问题吗?

I get an error message mentioning PicklingError. Any ideas what could be the problem here?

推荐答案

Joblib 专为为了简单的并行循环，环绕多处理.我建议您使用它，而不是直接与多重处理进行搏斗.

Joblib is designed specifically to wrap around multiprocessing for the purposes of simple parallel looping. I suggest using that instead of grappling with multiprocessing directly.

简单的情况如下:

from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(foo)(i**2) for i in range(10))  # n_jobs = number of processes

一旦您了解了语法，它就很简单.我们正在使用生成器语法，其中delayed用于调用函数foo，其参数包含在后面的括号中.

The syntax is simple once you understand it. We are using generator syntax in which delayed is used to call function foo with its arguments contained in the parentheses that follow.

在您的情况下，您应该使用生成器语法重写for循环，或者定义另一个函数(即"worker"函数)来执行单循环迭代的操作，并将其放入对Parallel的调用的生成器语法中

In your case, you should either rewrite your for loop with generator syntax, or define another function (i.e. 'worker' function) to perform the operations of a single loop iteration and place that into the generator syntax of a call to Parallel.

在后一种情况下，您将执行以下操作:

In the later case, you would do something like:

Parallel(n_jobs=2)(delayed(foo)(parameters) for x in range(i,j))

其中，foo是您定义的用于处理for循环主体的函数.请注意，由于Parallel仍会返回列表，因此您不想追加到列表.

where foo is a function you define to handle the body of your for loop. Note that you do not want to append to a list, since Parallel is returning a list anyway.

这篇关于在Python 3.2中并行执行for循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！