在IPython中并行嵌套for循环 [英] Parallelise nested for-loop in IPython
问题描述
我的python代码中有一个嵌套的for循环,如下所示:
I have a nested for loop in my python code that looks something like this:
results = []
for azimuth in azimuths:
for zenith in zeniths:
# Do various bits of stuff
# Eventually get a result
results.append(result)
我想在我的4核机器上并行化这个循环来加速它。看看IPython并行编程文档(http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#quick-and-easy-parallelism),似乎有一种简单的方法可以使用 map
以并行化迭代操作。
I'd like to parallelise this loop on my 4 core machine to speed it up. Looking at the IPython parallel programming documentation (http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#quick-and-easy-parallelism) it seems that there is an easy way to use map
to parallelise iterative operations.
但是,要做到这一点,我需要将循环内的代码作为一个函数(这是容易做到),然后映射这个功能。我遇到的问题是我无法获得一个数组来映射这个函数。 itertools.product()
生成一个迭代器,我似乎无法使用map函数。
However, to do that I need to have the code inside the loop as a function (which is easy to do), and then map across this function. The problem I have is that I can't get an array to map this function across. itertools.product()
produces an iterator which I can't seem to use the map function with.
Am我试图在这里使用地图咆哮错误的树?有没有更好的方法呢?或者是否有某种方法可以使用 itertools.product
然后使用映射到结果的函数进行并行执行?
Am I barking up the wrong tree by trying to use map here? Is there a better way to do it? Or is there some way to use itertools.product
and then do parallel execution with a function mapped across the results?
推荐答案
要并行化每个调用,您只需要为每个参数获取一个列表。您可以使用 itertools.product
+ zip
来获取此信息:
To parallelize every call, you just need to get a list for each argument. You can use itertools.product
+ zip
to get this:
allzeniths, allazimuths = zip(*itertools.product(zeniths, azimuths))
然后你可以使用map:
Then you can use map:
amr = dview.map(f, allzeniths, allazimuths)
为了更深入地了解这些步骤,这里有一个例子:
To go a bit deeper into the steps, here's an example:
zeniths = range(1,4)
azimuths = range(6,8)
product = list(itertools.product(zeniths, azimuths))
# [(1, 6), (1, 7), (2, 6), (2, 7), (3, 6), (3, 7)]
所以我们有一个对列表,但我们真正想要的是每个参数的单个列表,即一对列表。这正是稍微有点奇怪的 zip(* product)
语法让我们:
So we have a "list of pairs", but what we really want is a single list for each argument, i.e. a "pair of lists". This is exactly what the slightly weird zip(*product)
syntax gets us:
allzeniths, allazimuths = zip(*itertools.product(zeniths, azimuths))
print allzeniths
# (1, 1, 2, 2, 3, 3)
print allazimuths
# (6, 7, 6, 7, 6, 7)
现在我们只将我们的函数映射到这两个列表,以并行化嵌套for循环:
Now we just map our function onto those two lists, to parallelize nested for loops:
def f(z,a):
return z*a
view.map(f, allzeniths, allazimuths)
并且没有什么特别的,只有两个 - 这个方法应该扩展到任意数量的嵌套循环。
And there's nothing special about there being only two - this method should extend to an arbitrary number of nested loops.
这篇关于在IPython中并行嵌套for循环的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!