多处理冻结计算机 [英] multiprocessing freeze computer

问题描述

我通过使用多处理改进了我的执行时间，但我不确定 PC 的行为是否正确，它会冻结系统，直到所有进程都完成.我使用的是 Windows 7 和 Python 2.7.

I improved my execution time by using multiprocessing but I am not sure whether the behavior of the PC is correct, it freezes the system until all processes are done. I am using Windows 7 and Python 2.7.

也许我做错了，这是我所做的:

Perhaps I am doing a mistake, here is what I did:

def do_big_calculation(sub_list, b, c):
    # do some calculations here with the sub_list

if __name__ == '__main__':
    list = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
    jobs = []
    for sub_l in list :
        j = multiprocessing.Process(target=do_big_calculation, args=(sub_l, b, c))
        jobs.append(j)
    for j in jobs:
        j.start()

推荐答案

在这里，您为每个任务创建 1 个 Process.这将并行运行您的所有任务，但它会给您的计算机带来沉重的开销，因为您的调度程序需要管理许多进程.这可能会导致系统冻结，因为您的程序使用了太多资源.

Here, you are creating 1 Process per task. This is will run all your tasks in parallel but it imposes an heavy overhead on your computer as your scheduler will need to manage many processes. This can cause a system freeze as too many ressources are used for your program.

这里的解决方案可能是使用 multiprocessing.Pool 来运行给定数量的进程同时执行某些任务:

A solution here could be to use the multiprocessing.Pool to run a given number of processes simultaneously performing some tasks:

import multiprocessing as mp

def do_big_calculation(args):
    sub_list, b, c = args
    return 1

if __name__ == '__main__':
    b, c = 1, 1
    ll = [([1, 2, 3, 4], b, c),
          ([5, 6, 7, 8], b, c),
          ([9, 10, 11, 12], b, c)]
    pool = mp.Pool(4)
    result = pool.map(do_big_calculation, ll)
    pool.terminate()
    print(result)

如果你准备使用第三方库，你也可以看看concurrent.futures(你需要在python2.7中安装它，但它存在于python3.4+)或joblib(可通过 pip 获得):

If you are ready to use third party library, you could also take a look at concurrent.futures (you need to install it in python2.7 but it exists for python3.4+) or joblib (available with pip):

from joblib import Parallel, delayed

def do_big_calculation(sub_list, b, c):
    return 1

if __name__ == '__main__':
    b, c = 1, 1
    ll = [([1, 2, 3, 4], b, c),
          ([5, 6, 7, 8], b, c),
          ([9, 10, 11, 12], b, c)]
    result = Parallel(n_jobs=-1)(
        delayed(do_big_calculation)(l, b, c) for l in ll)
    print(result)

这种库的主要优点是它正在开发，而python2.7中的multiprocessing被冻结.因此，错误修复和改进相对频繁.
它还实现了一些巧妙的工具来减少计算的开销.例如，它使用大 numpy 数组的内存映射(减少启动所有作业的内存占用).

The main advantage of such library is that it is developing whereas multiprocessing in python2.7 is freezed. Thus, there are bug fixes and improvements relatively often.
It also implements some clever tools to reduce the overhead for the computation. For instance, it uses memory mapping for big numpy array (reducing the memory footprint of starting all jobs).

这篇关于多处理冻结计算机的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！