python pathos的酸洗问题 [英] Pickling issue with python pathos

问题描述

import pathos.multiprocessing as mp
class Model_Output_File():
    """
    Class to read Model Output files
    """
    def __init__(self, ftype = ''):
        """
        Constructor
        """
        # Create a sqlite database in the analysis directory
        self.db_name = 'sqlite:///' + constants.anly_dir + os.sep + ftype + '_' + '.db'
        self.engine  = create_engine(self.db_name)
        self.ftype   = ftype

    def parse_DGN(self, fl):
        df      = pandas.read_csv(...)
        df.to_sql(self.db_name, self.engine, if_exists='append')

    def collect_epic_output(self, fls):
        pool = mp.ProcessingPool(4)
        if(self.ftype == 'DGN'):
            pool.map(self.parse_DGN, fls)
        else:
            logging.info( 'Wrong file type')

if __name__ == '__main__':
    list_fls = fnmatch.filter(...)
    obj = Model_Output_File(ftype = 'DGN')
    obj.collect_model_output(list_fls)

在上面的代码中，我正在使用pathos多处理库来避免类的python多处理问题.但是我收到一个腌制错误:

In the code above, I am using the pathos multiprocessing library to avoid python multiprocessing issues with classes. However I am getting a pickling error:

  pool.map(self.parse_DGN, fls)
  File "C:\Anaconda64\lib\site-packages\pathos-0.2a1.dev0-py2.7.egg\pathos\multiprocessing.py", line 131, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "C:\Anaconda64\lib\multiprocessing\pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Anaconda64\lib\multiprocessing\pool.py", line 567, in get
    raise self._value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

我该如何解决?

推荐答案

我是pathos的作者.您得到的是cPickle.PicklingError…，而您不应获得pathos.确保已安装multiprocess，如果已安装，则必须具有C ++编译器.您可以通过导入dill并在类内部或在外部使用类的实例进行dill.copy(self.parse_DGN)来检查是否出现酸洗错误.如果可行，那么您可能遇到了一些安装问题，其中pathos正在查找python标准库multiprocessing.如果是这样，那么您可能需要安装一个编译器……例如Microsoft Visual Studio Community.请参阅: github.com/mmckerns/tuthpc .确保在安装MS编译器后重建multiprocess.

I'm the pathos author. You are getting a cPickle.PicklingError… which you should not get with pathos. Make sure you have multiprocess installed, and if you do, that you have a C++ compiler. You can check for pickling errors by importing dill, and doing a dill.copy(self.parse_DGN) inside your class, or externally using the instance of the class. If that works, then you probably have some installation issue, where pathos is finding the python standard library multiprocessing. If so, then you probably need to install a compiler… like Microsoft Visual Studio Community. See: github.com/mmckerns/tuthpc. Make sure to rebuild multiprocess after the install of the MS compiler.

这篇关于python pathos的酸洗问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！