Python multiprocessing.Pool:AttributeError [英] Python multiprocessing.Pool: AttributeError

问题描述

我在一个类中有一个方法需要在循环中做很多工作，我想把工作分散到我的所有内核上.

我写了下面的代码，如果我使用普通的 map() 就可以工作，但是使用 pool.map() 会返回一个错误.

导入多处理pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)类其他类:def run(句子，图表):返回错误类 SomeClass:def __init__(self):self.sentences = [[一些字符串"]]self.graphs = [字符串"]def some_method(self):其他 = 其他类()定义单个(参数):句子，图 = 参数返回 [other.run(sentence, graph) for 句子中的句子]返回列表(pool.map(single, zip(self.sentences, self.graphs)))SomeClass().some_method()

错误 1:

<块引用>

AttributeError: Can't pickle local object 'SomeClass.some_method..single'

为什么不能pickle single()?我什至尝试将 single() 移动到全局模块范围(不在类内部 - 使其独立于上下文):

导入多处理pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)类其他类:def run(句子，图表):返回错误定义单个(参数):其他 = 其他类()句子，图 = 参数返回 [other.run(sentence, graph) for 句子中的句子]类 SomeClass:def __init__(self):self.sentences = [[一些字符串"]]self.graphs = [字符串"]def some_method(self):返回列表(pool.map(single, zip(self.sentences, self.graphs)))SomeClass().some_method()

我得到以下...

错误 2:

<块引用>

AttributeError: 无法从.../test.py"获取模块ma​​in"上的属性single"

解决方案

错误 1:

<块引用>

AttributeError: 不能pickle 本地对象'SomeClass.some_method..single'

您通过将嵌套的目标函数 single() 移到顶层，自己解决了这个错误.

背景:

池需要腌制(序列化)它发送到其工作进程的所有内容(IPC).Pickling 实际上只保存函数的名称，而 unpickling 需要按名称重新导入函数.为此，该函数需要在顶层定义，嵌套函数不能被子进程导入，并且已经尝试对它们进行腌制会引发异常(更多).

<小时>

错误 2:

<块引用>

AttributeError: 无法从模块 'main' 中获取属性 'single''.../test.py'

您在定义函数和类之前启动池，这样子进程就不能继承任何代码.将池启动移到底部并使用 if __name__ == '__main__' 保护(为什么?):

导入多处理类其他类:def run(自我，句子，图形):返回错误定义单个(参数):其他 = 其他类()句子，图 = 参数返回 [other.run(sentence, graph) for 句子中的句子]类 SomeClass:def __init__(self):self.sentences = [[一些字符串"]]self.graphs = ["字符串"]def some_method(self):返回列表(pool.map(single, zip(self.sentences, self.graphs)))if __name__ == '__main__': # <- 防止 RuntimeError for 'spawn'# 和 'forkserver' start_methods使用 multiprocessing.Pool(multiprocessing.cpu_count() - 1) 作为池:打印(SomeClass().some_method())

<小时>

附录

<块引用>

...我想将工作分散到我所有的内核上.

关于 multiprocessing.Pool 如何进行分块工作的潜在有用背景:

Python 多处理:理解 chunksize 背后的逻辑

I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores.

I wrote the following code, which works if I use normal map(), but with pool.map() returns an error.

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      other = OtherClass()

      def single(params):
          sentences, graph = params
          return [other.run(sentence, graph) for sentence in sentences]

      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

Error 1:

AttributeError: Can't pickle local object 'SomeClass.some_method..single'

Why can't it pickle single()? I even tried to move single() to the global module scope (not inside the class - makes it independent of the context):

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

and I get the following ...

Error 2:

AttributeError: Can't get attribute 'single' on module 'main' from '.../test.py'

解决方案

Error 1:

AttributeError: Can't pickle local object 'SomeClass.some_method..single'

You solved this error yourself by moving the nested target-function single() out to the top-level.

Background:

Pool needs to pickle (serialize) everything it sends to its worker-processes (IPC). Pickling actually only saves the name of a function and unpickling requires re-importing the function by name. For that to work, the function needs to be defined at the top-level, nested functions won't be importable by the child and already trying to pickle them raises an exception (more).

---

Error 2:

AttributeError: Can't get attribute 'single' on module 'main' from '.../test.py'

You are starting the pool before you define your function and classes, that way the child processes cannot inherit any code. Move your pool start up to the bottom and protect (why?) it with if __name__ == '__main__':

import multiprocessing

class OtherClass:
  def run(self, sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
   def __init__(self):
       self.sentences = [["Some string"]]
       self.graphs = ["string"]

   def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))

if __name__ == '__main__':  # <- prevent RuntimeError for 'spawn'
    # and 'forkserver' start_methods
    with multiprocessing.Pool(multiprocessing.cpu_count() - 1) as pool:
        print(SomeClass().some_method())

---

Appendix

...I would like to spread the work over all of my cores.

Potentially helpful background on how multiprocessing.Pool is chunking work:

Python multiprocessing: understanding logic behind chunksize

这篇关于Python multiprocessing.Pool:AttributeError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！