Python 多处理池突然停止 [英] Python Multiprocessing Pool Stops Abruptly

问题描述

我正在尝试根据我的要求执行并行处理，对于 4k-5k 元素，代码似乎按预期工作.但是一旦要处理的元素开始增加，代码就会处理一些列表，然后没有抛出任何错误，程序就会突然停止运行.

I am trying to perform parallel processing for my requirements, and the code seems to be working as expected for 4k-5k elements in parallel. But as soon as the elements to be processed start increasing, the code processes a few listings and then without throwing any error, the program stops running abruptly.

我检查过，程序没有挂起，RAM 可用(我有 16 Gb RAM)，CPU 利用率甚至不到 30%.似乎无法弄清楚发生了什么.我有 100 万个元素要处理.

I checked and the program is not hung, the RAM is available (I have a 16 Gb RAM) and CPU Utilization is not even 30%. Can't seem to figure out what is happening. I have 1 million elements to be processed.

def get_items_to_download():
    #iterator to fetch all items that are to be downloaded
    yield download_item

def start_download_process():
    multiproc_pool = multiprocessing.Pool(processes=10)
    for download_item in get_items_to_download():
        multiproc_pool.apply_async(start_processing, args = (download_item, ), callback = results_callback)
    
    multiproc_pool.close()
    multiproc_pool.join()

def start_processing(download_item):
    try:
        # Code to download item from web API
        # Code to perform some processing on the data
        # Code to update data into database
        return True
    except Exception as e:
        return False

def results_callback(result):
    print(result)

if __name__ == "__main__":
    start_download_process()

更新 -

发现错误 - BrokenPipeError: [Errno 32] Broken pipe

Found the error- BrokenPipeError: [Errno 32] Broken pipe

跟踪 -

Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 125, in worker
put((job, i, result))
File "/usr/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/usr/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

推荐答案

我在 Linux 上使用 Python 3.8 也有同样的体验.我使用 Python 3.7 设置了一个新环境，multiprocessing.Pool() 现在可以正常工作了.

I had the same experience with Python 3.8 on Linux. I set up a new environment with Python 3.7 and multiprocessing.Pool() works now without any issue.

这篇关于Python 多处理池突然停止的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！