无法使用setParseAction()方法腌制Pyparsing表达式.需要多处理 [英] Can't pickle Pyparsing expression with setParseAction() method. Needed for multiprocessing

问题描述

我最初的问题是我正在尝试执行以下操作:

My original issue is that I am trying to do the following:

def submit_decoder_process(decoder, input_line):
    decoder.process_line(input_line)
    return decoder

self.pool = Pool(processes=num_of_processes)
self.pool.apply_async(submit_decoder_process, [decoder, input_line]).get()

在这里要描述解码器，但是重要的是解码器是一个使用PyParsing表达式初始化的对象，该表达式调用setParseAction().这会使多重处理使用的泡菜失败，进而使上面的代码失败.

decoder is a bit involved to describe here, but the important thing is that decoder is an object that is initialized with PyParsing expression that calls setParseAction(). This fails pickle that multiprocessing uses and this in turn fails the above code.

现在，这是我已经隔离和简化的pickle/PyParsing问题. 以下代码由于泡菜故障而产生错误消息.

Now, here is the pickle/PyParsing problem that I have isolated and simplified. The following code yields an error message due to pickle failure.

import pickle
from pyparsing import *

def my_pa_func():
    pass

pickle.dumps(Word(nums).setParseAction(my_pa_func))

错误消息:

pickle.PicklingError: Can't pickle <function wrapper at 0x00000000026534A8>: it's not found as pyparsing.wrapper

现在，如果您删除了.setParseAction(my_pa_func)调用，它将可以正常使用:

Now If you remove the call .setParseAction(my_pa_func), it will work with no problems:

pickle.dumps(Word(nums))

我该如何解决?多进程使用泡菜，所以我想我不能避免它.据说使用莳萝的pathos软件包还不够成熟，至少，我在Windows-64bit上安装它时遇到了问题.我真的在这里挠头.

How can I get around it? Multiprocesing uses pickle, so I can't avoid it, I guess. The pathos package that is supposedly uses dill is not mature enough, at least, I am having problems installing it on my Windows-64bit. I am really scratching my head here.

推荐答案

好的，这是受rocksportrocker启发的解决方案:

OK, here is the solution inspired by rocksportrocker: Python multiprocessing pickling error

这个想法是使不能腌制的对象在进程之间来回传递时变钝，然后在通过之后取消填充":

The idea is to dill the object that can't be pickled while passing it back and forth between processes and then "undill" it after it has been passed:

from multiprocessing import Pool
import dill

def submit_decoder_process(decoder_dill, input_line):
    decoder = dill.loads(decoder_dill)  # undill after it was passed to a pool process
    decoder.process_line(input_line)
    return dill.dumps(decoder)  # dill before passing back to parent process

self.pool = Pool(processes=num_of_processes)

# Dill before sending to a pool process
decoder_processed = dill.loads(self.pool.apply_async(submit_decoder_process, [dill.dumps(decoder), input_line]).get())

这篇关于无法使用setParseAction()方法腌制Pyparsing表达式.需要多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！