如何在多处理期间修复“TypeError: can't pickle module objects"? [英] how to fix 'TypeError: can't pickle module objects' during multiprocessing?

问题描述

我正在尝试实现多处理，但是我在访问来自对象扫描的信息时遇到了困难，我正在通过 pool.map() 函数

I am trying to implement multiprocessing, but I am having difficulties accessing information from the object scans that I'm passing through the pool.map() function

在多处理之前(这非常有效):

Before multiprocessing (this works perfectly):

for sc in scans:
    my_file = scans[sc].resources['DICOM'].files[0]

多处理后(不起作用，错误如下所示):

After multiprocessing (does not work, error shown below):

def process(x):
    my_file = x.resources['DICOM'].files[0] 

def another_method():
    ...                
    pool = Pool(os.cpu_count())
    pool.map(process, [scans[sc] for sc in scans])

another_method()  

我在多处理后"代码中遇到的错误:

The error I am getting with 'After multiprocessing' code:

---> 24         pool.map(process, [scans[sc] for sc in scans])

~/opt/anaconda3/lib/python3.7/multiprocessing/pool.py in map(self, func, iterable, chunksize)
    266         in a list that is returned.
    267         '''
--> 268         return self._map_async(func, iterable, mapstar, chunksize).get()
    269 
    270     def starmap(self, func, iterable, chunksize=None):

~/opt/anaconda3/lib/python3.7/multiprocessing/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658 
    659     def _set(self, i, obj):

~/opt/anaconda3/lib/python3.7/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    429                         break
    430                     try:
--> 431                         put(task)
    432                     except Exception as e:
    433                         job, idx = task[:2]

~/opt/anaconda3/lib/python3.7/multiprocessing/connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
    207 
    208     def recv_bytes(self, maxlength=None):

~/opt/anaconda3/lib/python3.7/multiprocessing/reduction.py in dumps(cls, obj, protocol)
     49     def dumps(cls, obj, protocol=None):
     50         buf = io.BytesIO()
---> 51         cls(buf, protocol).dump(obj)
     52         return buf.getbuffer()
     53 

TypeError: can't pickle module objects

推荐答案

您没有提供完整的数据结构，但这可能会有所帮助.多处理对对象有点敏感......某些对象不能像文件对象一样被腌制.Python:无法pickle模块对象错误

You didn't provide the full data structure, but this might help. Multiprocessing is kinda sensible to objects... some object can't be pickled like file objects. Python: can't pickle module objects error

如果您只需要文件名，请在映射函数中使用该文件名而不是进程

If you need only the file name use that in the map function instead of process

这篇关于如何在多处理期间修复“TypeError: can't pickle module objects"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！