在IPython中并行嵌套for循环 [英] Parallelise nested for-loop in IPython

问题描述

我的python代码中有一个嵌套的for循环，如下所示：

I have a nested for loop in my python code that looks something like this:

results = []

for azimuth in azimuths:
    for zenith in zeniths:
        # Do various bits of stuff
        # Eventually get a result
        results.append(result)

我想在我的4核机器上并行化这个循环来加速它。看看IPython并行编程文档（http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#quick-and-easy-parallelism），似乎有一种简单的方法可以使用 map 以并行化迭代操作。

I'd like to parallelise this loop on my 4 core machine to speed it up. Looking at the IPython parallel programming documentation (http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#quick-and-easy-parallelism) it seems that there is an easy way to use map to parallelise iterative operations.

但是，要做到这一点，我需要将循环内的代码作为一个函数（这是容易做到），然后映射这个功能。我遇到的问题是我无法获得一个数组来映射这个函数。 itertools.product（）生成一个迭代器，我似乎无法使用map函数。

However, to do that I need to have the code inside the loop as a function (which is easy to do), and then map across this function. The problem I have is that I can't get an array to map this function across. itertools.product() produces an iterator which I can't seem to use the map function with.

Am我试图在这里使用地图咆哮错误的树？有没有更好的方法呢？或者是否有某种方法可以使用 itertools.product 然后使用映射到结果的函数进行并行执行？

Am I barking up the wrong tree by trying to use map here? Is there a better way to do it? Or is there some way to use itertools.product and then do parallel execution with a function mapped across the results?

推荐答案

要并行化每个调用，您只需要为每个参数获取一个列表。您可以使用 itertools.product + zip 来获取此信息：

To parallelize every call, you just need to get a list for each argument. You can use itertools.product + zip to get this:

allzeniths, allazimuths = zip(*itertools.product(zeniths, azimuths))

然后你可以使用map：

Then you can use map:

amr = dview.map(f, allzeniths, allazimuths)

为了更深入地了解这些步骤，这里有一个例子：

To go a bit deeper into the steps, here's an example:

zeniths = range(1,4)
azimuths = range(6,8)

product = list(itertools.product(zeniths, azimuths))
# [(1, 6), (1, 7), (2, 6), (2, 7), (3, 6), (3, 7)]

所以我们有一个对列表，但我们真正想要的是每个参数的单个列表，即一对列表。这正是稍微有点奇怪的 zip（* product）语法让我们：

So we have a "list of pairs", but what we really want is a single list for each argument, i.e. a "pair of lists". This is exactly what the slightly weird zip(*product) syntax gets us:

allzeniths, allazimuths = zip(*itertools.product(zeniths, azimuths))

print allzeniths
# (1, 1, 2, 2, 3, 3)
print allazimuths
# (6, 7, 6, 7, 6, 7)

现在我们只将我们的函数映射到这两个列表，以并行化嵌套for循环：

Now we just map our function onto those two lists, to parallelize nested for loops:

def f(z,a):
    return z*a

view.map(f, allzeniths, allazimuths)

并且没有什么特别的，只有两个 - 这个方法应该扩展到任意数量的嵌套循环。

And there's nothing special about there being only two - this method should extend to an arbitrary number of nested loops.

这篇关于在IPython中并行嵌套for循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！