为什么不能在multiprocessing.Pool中使用operator.itemgetter? [英] Why can't I use operator.itemgetter in a multiprocessing.Pool?

问题描述

以下程序:

import multiprocessing,operator
f = operator.itemgetter(0)
# def f(*a): return operator.itemgetter(0)(*a)
if __name__ == '__main__':
    multiprocessing.Pool(1).map(f, ["ab"])

失败，并出现以下错误:

fails with the following error:

Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python3.2/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib/python3.2/multiprocessing/process.py", line 116, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.2/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python3.2/multiprocessing/queues.py", line 382, in get
    return recv()
TypeError: itemgetter expected 1 arguments, got 0

为什么会出现错误(在Linux x64上的cPython 2.7和3.2上)，如果我取消注释第三行，为什么它消失了?

Why do I get the error (on cPython 2.7 and 3.2 on Linux x64), and why does it vanish if I uncomment the third line?

推荐答案

这里的问题是，多处理模块通过复制将对象传递给其他进程(很明显)，并且itemgetter对象无法使用任何明显的方式进行复制:

The problem here is that the multiprocessing module passes objects by copy into the other processes (obviously), and itemgetter objects are not copyable using any of the obvious means:

In [10]: a = operator.itemgetter(0)
Out[10]: copy.copy(a)
TypeError: itemgetter expected 1 arguments, got 0

In [10]: a = operator.itemgetter(0)
Out[10]: copy.deepcopy(a)
TypeError: itemgetter expected 1 arguments, got 0

In [10]: a = operator.itemgetter(0)
Out[10]: pickle.dumps(a)
TypeError: can't pickle itemgetter objects

# etc.

问题甚至没有尝试在其他进程中调用f；首先尝试复制它. (如果您查看我在上面省略的堆栈跟踪，则会看到很多有关失败原因的信息.)

The problem isn't even attempting to call f inside the other processes; it's trying to copy it in the first place. (If you look at the stack traces, which I omitted above, you'll see a lot more information on why this fails.)

当然，通常这无关紧要，因为快速构造新的itemgetter和复制一个itemgetter几乎既简单又有效.这就是您替代的"f"函数正在执行的操作. (当然，复制动态创建itemgetter的函数不需要复制itemgetter.)

Of course usually this doesn't matter, because it's nearly as easy and efficient to construct a new itemgetter on the fly as to copy one. And this is what your alternative "f" function is doing. (Copying a function that creates an itemgetter on the fly doesn't require copying an itemgetter, of course.)

您可以将"f"转换为lambda.或编写一个琐碎的函数(名为lambda)，而无需使用itemgetter即可执行相同的操作.或编写一个可复制的itemgetter替代品(显然不会那么难).但是您不能像您想要的那样直接使用itemgetter对象.

You could turn "f" into a lambda. Or write a trivial function (named or lambda) that does the same thing without using itemgetter. Or write an itemgetter replacement that is copyable (which obviously wouldn't be all that hard). But you can't directly use itemgetter objects as-is the way you want to.

这篇关于为什么不能在multiprocessing.Pool中使用operator.itemgetter?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！