如何在不处于顶层的情况下解决python多处理的酸洗错误? [英] How to get around the pickling error of python multiprocessing without being in the top-level?

问题描述

我已经多次研究了这个问题，但是还没有找到适合我的情况或可以理解的解决方法，因此请耐心等待.

I've researched this question multiple times, but haven't found a workaround that either works in my case, or one that I understand, so please bear with me.

基本上，我具有功能的分层组织，这使我无法在顶层进行多处理.不幸的是，我不相信我可以更改程序的布局-因为我需要在初始输入后创建的所有变量.

Basically, I have a hierarchical organization of functions, and that is preventing me from multiprocessing in the top-level. Unfortunately, I don't believe I can change the layout of the program - because I need all the variables that I create after the initial inputs.

例如，说我有这个:

import multiprocessing

  def calculate(x):
    # here is where I would take this input x (and maybe a couple more inputs)
    # and build a larger library of variables that I use further down the line

    def domath(y):
      return x * y

    pool = multiprocessing.Pool(3)
    final= pool.map(domath, range(3))

calculate(2)

这会产生以下错误:

Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

我当时在想全局变量，但恐怕我不得不定义太多，这可能会使我的程序放慢速度. 是否有任何解决方法而不必重组整个程序?

I was thinking of globals, but I'm afraid that I'd have to define too many and that may slow my program down quite a bit. Is there any workaround without having to restructure the whole program?

推荐答案

您可以使用pathos.multiprocessing，它是multiprocessing的一个分叉，它使用dill序列化器而不是pickle. dill可以在python中序列化几乎所有内容.然后，无需编辑代码.

You could use pathos.multiprocessing, which is a fork of multiprocessing that uses the dill serializer instead of pickle. dill can serialize pretty much anything in python. Then, no need to edit your code.

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> 
>>> def calculate(x):
...   def domath(y):
...     return x*y
...   return Pool().map(domath, range(3))
... 
>>> calculate(2)
[0, 2, 4]

您甚至可以疯了……因为大多数东西都是腌制的.无需奇数的非pythonic解决方案，就不必使用纯multiprocessing来做饭.

You can even go nuts with it… as most things are pickled. No need for the odd non-pythonic solutions you have to cook up with pure multiprocessing.

>>> class Foo(object):
...   def __init__(self, x):
...     self.x = x
...   def doit(self, y):
...     return ProcessingPool().map(self.squared, calculate(y+self.x))
...   def squared(self, z):
...     return z*z
... 
>>> def thing(obj, y):
...   return getattr(obj, 'doit')(y)
... 
>>> ProcessingPool().map(thing, ProcessingPool().map(Foo, range(3)), range(3))
[[0, 0, 0], [0, 4, 16], [0, 16, 64]]

在此处获取pathos: https://github.com/uqfoundation

这篇关于如何在不处于顶层的情况下解决python多处理的酸洗错误?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！