pathos.multiprocessing 有星图吗? [英] Does pathos.multiprocessing have starmap?

问题描述

我在执行下面的代码时出错.问题似乎是 map 不支持接受多个输入的函数，就像在 python 内置的 multiprocessing 包中一样.但是在内置包中，有一个starmap可以解决这个问题.pathos.multiprocessing 是否相同?

import pathos.multiprocessing 作为 mp班级酒吧:def foo(self, name):返回 len(str(name))def boo(self, x, y, z):sum = self.foo(x)sum += self.foo(y)sum += self.foo(z)返还金额如果 __name__ == '__main__':b = 酒吧()池 = mp.ProcessingPool()结果 = pool.map(b.boo, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])打印(结果)

<块引用>

TypeError: boo() 缺少 2 个必需的位置参数:'y' 和 'z'

按照建议更新 lambda 表达式(无效):

如果 __name__ == '__main__':b = 酒吧()池 = mp.ProcessingPool()结果 = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])打印(结果)

<块引用>

multiprocess.pool.RemoteTraceback:

"""

回溯(最近一次调用最后一次):

文件"C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py",第 121 行，在工人中

result = (True, func(*args, **kwds))

文件"C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py",第 44 行，在地图星中

返回列表(map(*args))

文件"C:\Users\yg451\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py",第 15 行，在

func = lambda 参数:f(*args)

文件C:/Users/yg451/Code/foo/MachineLearning/xPype/test/scratch.py​​"，第 18 行，在

results = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])

NameError: name 'b' 未定义

"""

解决方案

我是 pathos 的作者.pathos 比 starmap 更旧，并不真正需要它.它以与内置 map 完全相同的方式解决池中的多个参数.

<预><代码>>>>导入 pathos.multiprocessing 作为 mp>>>班级酒吧:... def foo(self, name):... 返回 len(str(name))... def boo(self, x, y, z):... sum = self.foo(x)... sum += self.foo(y)... sum += self.foo(z)...返回总和...>>>b = 酒吧()>>>池 = mp.ProcessingPool()>>>f = lambda x: b.boo(*x)>>>结果 = pool.map(f, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])>>>结果[6, 4, 5]>>>结果 = pool.map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])>>>结果[6, 4, 5]>>>结果 = map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])>>>列表(结果)[6, 4, 5]>>>

所以，本质上，starmap 是不必要的.但是，由于它最近被添加到某些 Python 版本的 multiprocessing 中的标准 Pool 接口中，因此它可能应该在 pathos 中更加突出.请注意，如果您愿意，已经可以从 pathos 获得 starmap 的增强"版本.

<预><代码>>>>输入感伤>>>mp = pathos.helpers.mp>>>p = mp.Pool()>>>星图<0x1038684e0处<multiprocess.pool.Pool对象的绑定方法Pool.starmap>>>>

I got an error when executing code below. The problem seems to be map doesn't support functions taking multiple inputs, just as in the python built-in multiprocessing package. But in the built-in package, there is a starmap that solves this issue. Does pathos.multiprocessing have the same?

import pathos.multiprocessing as mp


class Bar:
    def foo(self, name):
        return len(str(name))

    def boo(self, x, y, z):
        sum = self.foo(x)
        sum += self.foo(y)
        sum += self.foo(z)
        return sum


if __name__ == '__main__':
    b = Bar()
    pool = mp.ProcessingPool()
    results = pool.map(b.boo, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
    print(results)

TypeError: boo() missing 2 required positional arguments: 'y' and 'z'

Update for lambda expression as suggested (didn't work):

if __name__ == '__main__':
    b = Bar()
    pool = mp.ProcessingPool()
    results = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
    print(results)

multiprocess.pool.RemoteTraceback:

"""

Traceback (most recent call last):

File "C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py", line 121, in worker

result = (True, func(*args, **kwds))

File "C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py", line 44, in mapstar

return list(map(*args))

File "C:\Users\yg451\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py", line 15, in

func = lambda args: f(*args)

File "C:/Users/yg451/Code/foo/Machine Learning/xPype/test/scratch.py", line 18, in

results = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])

NameError: name 'b' is not defined

"""

解决方案

I'm the pathos author. pathos is older than starmap, and doesn't really need it. It solved multiple arguments in a pool exactly the same way that the built-in map does.

>>> import pathos.multiprocessing as mp
>>> class Bar:
...     def foo(self, name):
...         return len(str(name))
...     def boo(self, x, y, z):
...         sum = self.foo(x)
...         sum += self.foo(y)
...         sum += self.foo(z)
...         return sum
... 
>>> b = Bar()
>>> pool = mp.ProcessingPool()
>>> f = lambda x: b.boo(*x)    
>>> results = pool.map(f, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
>>> results 
[6, 4, 5]
>>> results = pool.map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])
>>> results
[6, 4, 5]
>>> results = map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])
>>> list(results)
[6, 4, 5]
>>> 

So, essentially, starmap is unnecessary. However, as it's been recently added to the standard Pool interface in multiprocessing in certain versions of python, it probably should be more prominent in pathos. Note that it already is possible to get to an "augmented" version of starmap from pathos if you like.

>>> import pathos
>>> mp = pathos.helpers.mp
>>> p = mp.Pool()
>>> p.starmap
<bound method Pool.starmap of <multiprocess.pool.Pool object at 0x1038684e0>>
>>> 

这篇关于pathos.multiprocessing 有星图吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！