Python多处理池映射:AttributeError:无法腌制本地对象 [英] Python Multiprocessing Pool Map: AttributeError: Can't pickle local object

问题描述

我在一个类中有一个方法，该方法需要循环执行许多工作，我希望将工作分散到我的所有核心上.

I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores.

我编写了以下代码，如果我使用普通的map可以使用，但是使用pool.map返回错误.

I wrote the following code, which works if I use normal map, but with pool.map returns an error.

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      other = OtherClass()

      def single(params):
          sentences, graph = params
          return [other.run(sentence, graph) for sentence in sentences]

      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

错误:

AttributeError:无法腌制本地对象'SomeClass.some_method..single'

AttributeError: Can't pickle local object 'SomeClass.some_method..single'

为什么不能腌制single?我什至尝试将single移到全局模块作用域(不在类内部-使其独立于上下文):

Why can't it pickle single? I even tried to movesingle to the global module scope (not inside the class - makes it independent of the context):

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

，我收到以下错误消息:

and I get the following error:

AttributeError:无法从'.../test.py'

AttributeError: Can't get attribute 'single' on module 'main' from '.../test.py'

推荐答案

错误1:

AttributeError:无法腌制本地对象 'SomeClass.some_method..single'

AttributeError: Can't pickle local object 'SomeClass.some_method..single'

您可以通过将嵌套的目标函数single()移到顶层来自己解决此错误.

You solved this error yourself by moving the nested target-function single() out to the top-level.

背景:

Pool需要腌制(序列化)发送给其工作进程的所有内容( IPC ).酸洗实际上仅保存函数的名称，而酸洗则需要按名称重新导入该函数.为此，必须在顶层定义函数，子函数将无法嵌套嵌套的函数，并且已经尝试将其腌制的情况会引发异常(

Pool needs to pickle (serialize) everything it sends to its worker-processes (IPC). Pickling actually only saves the name of a function and unpickling requires re-importing the function by name. For that to work, the function needs to be defined at the top-level, nested functions won't be importable by the child and already trying to pickle them raises an exception (more).

AttributeError:无法从以下位置获取模块"main"上的属性"single" '.../test.py'

AttributeError: Can't get attribute 'single' on module 'main' from '.../test.py'

在定义函数和类之前，要在 之前启动池，这样子进程就不能继承任何代码.用if __name__ == '__main__':

You are starting the pool before you define your function and classes, that way the child processes cannot inherit any code. Move your pool start up to the bottom and protect (why?) it with if __name__ == '__main__':

import multiprocessing

class OtherClass:
  def run(self, sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
   def __init__(self):
       self.sentences = [["Some string"]]
       self.graphs = ["string"]

   def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))

if __name__ == '__main__':  # <- prevent RuntimeError for 'spawn'
    # and 'forkserver' start_methods
    with multiprocessing.Pool(multiprocessing.cpu_count() - 1) as pool:
        print(SomeClass().some_method())

---

附录

...我想将工作分散到我的所有核心.

...I would like to spread the work over all of my cores.

关于multiprocessing.Pool分块工作方式的潜在帮助背景:

Potentially helpful background on how multiprocessing.Pool is chunking work:

Python多处理:了解块大小背后的逻辑

这篇关于Python多处理池映射:AttributeError:无法腌制本地对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！