NotImplementedError:X不可腌制 [英] NotImplementedError: X is not picklable

问题描述

我正在使用Python 3.x遍历数据集字典( NetCDF4数据集).它们只是文件...

Using Python 3.x, I am trying to iterate over a dictionary of datasets (NetCDF4 datasets). They are just files...

我想在一个单独的过程中检查每个数据集:

I want to examine each dataset on a separate process:

def DoProcessWork(datasetId, dataset):
    parameter = dataset.variables["so2"]
    print(parameter[0,0,0,0])

if __name__ == '__main__':
    mp.set_start_method('spawn')
    processes = []
    for key, dataset in datasets.items():
        p = mp.Process(target=DoProcessWork, args=(key, dataset,))
        p.start()
        processes.append(p)

运行程序时，我会收到一些有关可拾取"的消息

When I run my program, I get some message about 'pickable'

File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "netCDF4\_netCDF4.pyx", line 1992, in netCDF4._netCDF4.Dataset.__reduce__ (netCDF4\_netCDF4.c:16805)
NotImplementedError: Dataset is not picklable

我做错了什么?我怎样才能解决这个问题? 可能是打开文件是在另一个进程上完成的，所以我收到错误消息是因为我试图将在1个进程上加载的数据传递给另一个进程?

What am I doing wrong? How can I fix this? Could it be that opening the file is done on another process, and so I am getting an error because I am trying to pass data loaded on 1 process to another process?

推荐答案

multiprocessing需要对输入进行序列化(修改)，以将其传递给将运行DoProcessWork的新过程.对于您的数据集对象是个问题，请参见

The multiprocessing needs to serialize (pickle) the inputs to pass them to the new proccess which will run the DoProcessWork. In your case the dataset object is a problem, see the list of what can be pickled.

可能的解决方法是将多重处理与另一个函数一起使用，该函数读取数据集并在其上调用DoProcessWork.

A possible workaround for you would be using multiprocessing with another function which reads the dataset and calls DoProcessWork on it.

这篇关于NotImplementedError:X不可腌制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！