如何并行处理这段代码? [英] How to parallelize this piece of code?

问题描述

我已经浏览了一段时间，但找不到我能理解的任何建设性答案.

I've been browsing for some time but couldn't find any constructive answer that I could comprehend.

我应该如何对以下代码进行Parallellize:

How should I paralellize the following code:

import random
import math
import numpy as np
import sys
import multiprocessing

boot = 20#number of iterations to be performed
def myscript(iteration_number):  
    #stuff that the code actually does


def main(unused_command_line_args):
    for i in xrange(boot):
        myscript(i)
    return 0

if __name__ == '__main__':
    sys.exit(main(sys.argv))

或者我在哪里可以读到它?我什至不知道该如何搜索.

or where can I read about it? I'm not really sure how to search for it even.

推荐答案

对于一系列尴尬的并行作业，从for循环到并行几乎是自然发展.

There's pretty much a natural progression from a for loop to parallel for a batch of embarrassingly parallel jobs.

>>> import multiprocess as mp
>>> # build a target function
>>> def doit(x):
...   return x**2 - 1
... 
>>> x = range(10)
>>> # the for loop
>>> y = []   
>>> for i in x:
...   y.append(doit(i))
... 
>>> y
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]

那么如何并行处理此功能?

So how to address this function in parallel?

>>> # convert the for loop to a map (still serial)
>>> y = map(doit, x)
>>> y
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>> 
>>> # build a worker pool for parallel tasks
>>> p = mp.Pool()
>>> # do blocking parallel
>>> y = p.map(doit, x)
>>> y
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>> 
>>> # use an iterator (non-blocking)
>>> y = p.imap(doit, x)
>>> y            
<multiprocess.pool.IMapIterator object at 0x10358d150>
>>> print list(y)
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>> # do asynchronous parallel
>>> y = p.map_async(doit, x)
>>> y
<multiprocess.pool.MapResult object at 0x10358d1d0>
>>> print y.get()
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>>
>>> # or if you like for loops, there's always this…
>>> y = p.imap_unordered(doit, x)
>>> z = []
>>> for i in iter(y):
...   z.append(i)
... 
>>> z
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]

最后一种形式是无序的迭代器，它通常是最快的……但是您不必担心结果返回的顺序-它们是无序的，并且没有保证返回按照提交顺序相同.

The last form is an unordered iterator, which tends to be the fastest… but you can't care about what order the results come back in -- they are unordered, and not guaranteed to return in the same order they were submitted.

还请注意，我使用了multiprocess(一个分叉)而不是multiprocessing…，但这纯粹是因为multiprocess在处理交互式定义的函数时更好.否则，上面的代码与multiprocessing相同.

Note also that I've used multiprocess (a fork) instead of multiprocessing… but purely because multiprocess is better when dealing with interactively defined functions. Otherwise the code above is the same for multiprocessing.

这篇关于如何并行处理这段代码?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！