如何并行化对象数组上的方法调用? [英] How can I parallelize method calls on an array of objects?

问题描述

我有一个包含对象列表的模拟.我想使用线程池在所有这些对象上并行调用一个方法，因为它们都不依赖于另一个.您无法腌制方法，因此我正在考虑使用具有副作用的包装器功能来执行以下操作:

I have a simulation that consists of a list of objects. I'd like to call a method on all of those objects in parallel, since none of them depends on the other, using a thread pool. You can't pickle a method, so I was thinking of using a wrapper function with a side effect to do something like the following:

from multiprocessing import Pool

class subcl:
    def __init__(self):
        self.counter=1
        return
    def increment(self):
        self.counter+=1
        return

def wrapper(targ):
    targ.increment()
    return

class sim:
    def __init__(self):
        self.world=[subcl(),subcl(),subcl(),subcl()]
    def run(self):
        if __name__=='__main__':
            p=Pool()
            p.map(wrapper,self.world)

a=sim()
a.run()
print a.world[1].counter #should be 2

但是，函数调用对数组中的实际对象没有预期的副作用.有没有一种方法可以通过线程池和映射来简单地处理此问题，还是我必须按照原始函数调用和元组/列表/字典进行所有操作(或者对多处理或其他并行库进行更详细的说明)?

However, the function call doesn't have the intended side effect on the actual objects in the array. Is there a way to handle this simply with a thread pool and map, or do I have to do everything in terms of raw function calls and tuples/lists/dicts (or get more elaborate with multiprocessing or some other parallelism library)?

推荐答案

混淆的主要根源是multiprocessing使用单独的进程而不是线程.这意味着父级不会自动看到子级对对象状态所做的任何更改.

The main source of confusion is that multiprocessing uses separate processes and not threads. This means that any changes to object state made by the children aren't automatically visible to the parent.

在您的示例中处理此问题的最简单方法是让wrapper返回新值，然后使用Pool.map的返回值:

The easiest way to handle this in your example is to have wrapper return the new value, and then use the return value of Pool.map:

from multiprocessing import Pool

class subcl:
    def __init__(self):
        self.counter=1
        return
    def increment(self):
        self.counter+=1
        return

def wrapper(targ):
    targ.increment()
    return targ                                        # <<<<< change #1

class sim:
    def __init__(self):
        self.world=[subcl(),subcl(),subcl(),subcl()]
    def run(self):
        if __name__=='__main__':
            p=Pool()
            self.world = p.map(wrapper,self.world)     # <<<<< change #2

a=sim()
a.run()
print a.world[1].counter # now prints 2

这篇关于如何并行化对象数组上的方法调用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！