在python中并行化这些嵌套的for循环 [英] Parallelize these nested for loops in python

问题描述

我有一个多维数组(result)，应该由一些嵌套循环填充.函数fun()是一个复杂且耗时的函数.我想以并行方式填充数组元素，因此我可以使用系统的所有处理能力. 这是代码:

I have a multidimensional array (result) that should be filled by some nested loops. Function fun() is a complex and time-consuming function. I want to fill my array elements in a parallel manner, so I can use all my system's processing power. Here's the code:

import numpy as np


def fun(x, y, z):
    # time-consuming computation...
    # ...

    return output


dim1 = 10
dim2 = 20
dim3 = 30

result = np.zeros([dim1, dim2, dim3])

for i in xrange(dim1):
    for j in xrange(dim2):
        for k in xrange(dim3):
            result[i, j, k] = fun(i, j, k)

我的问题是我是否可以并行化此代码?如果可以，怎么做?"

My question is that "Can I parallelize this code or not? if yes, How?"

我正在使用Windows 10 64位和python 2.7.

I'm using Windows 10 64-bit and python 2.7.

如果可以的话，请通过更改我的代码来提供您的解决方案. 谢谢！

Please provide your solution by changing my code if you can. Thanks!

推荐答案

如果您想要一个更通用的解决方案，利用完全并行执行的优势，那为什么不使用这样的东西:

If you want a more general solution, taking advantage of fully parallel execution, then why not use something like this:

>>> import multiprocess as mp
>>> p = mp.Pool()
>>> 
>>> # a time consuming function taking x,y,z,...
>>> def fun(*args):
...   import time
...   time.sleep(.1)
...   return sum(*args)
... 
>>> dim1, dim2, dim3 = 10, 20, 30
>>> import itertools
>>> input = ((i,j,k) for i,j,k in itertools.combinations_with_replacement(xrange(dim3), 3) if i < dim1 and j < dim2)
>>> results = p.map(fun, input)
>>> p.close()
>>> p.join()
>>>
>>> results[:2]
[0, 1]
>>> results[-2:]
[56, 57]

请注意，我使用的是multiprocess而不是multiprocessing，但这只是为了能够在解释器中工作.

Note I'm using multiprocess instead of multiprocessing, but that's only to get the ability to work in the interpreter.

我没有使用numpy.array，但是如果您必须...可以将p.map的输出直接转储到numpy.array中，然后将shape属性修改为shape = (dim1, dim2, dim3) ，或者您可以执行以下操作:

I didn't use a numpy.array, but if you had to... you could just dump the output from p.map directly into a numpy.array and then modify the shape attribute to be shape = (dim1, dim2, dim3), or you could do something like this:

>>> input = ((i,j,k) for i,j,k in itertools.combinations_with_replacement(xrange(dim3), 3) if i < dim1 and j < dim2)
>>> import numpy as np
>>> results = np.empty(dim1*dim2*dim3)
>>> res = p.imap(fun, input)
>>> for i,r in enumerate(res):
...   results[i] = r
... 
>>> results.shape = (dim1,dim2,dim3)

这篇关于在python中并行化这些嵌套的for循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！