Python中的multiprocessing.dummy没有使用100％cpu [英] multiprocessing.dummy in Python is not utilising 100% cpu

问题描述

我正在用Python做一个机器学习项目，所以我必须做并行预测功能，这是我在程序中使用的功能.

I am doing a machine learning project in Python, so I have to do parallel predict function, which I'm using in my program.

from multiprocessing.dummy import Pool
from multiprocessing import cpu_count


def multi_predict(X, predict, *args, **kwargs):
    pool = Pool(cpu_count())
    results = pool.map(predict, X)
    pool.close()
    pool.join()
    return results

问题是我所有的CPU仅加载20％至40％的负载(总计为100％).我使用multiprocessing.dummy，因为在酸洗功能中的多处理模块存在一些问题.

The problem is that all my CPUs loaded only on 20-40% (in sum it's 100%). I use multiprocessing.dummy because I have some problems with multiprocessing module in pickling function.

推荐答案

使用 multiprocessing.dummy ，您使用的是线程，而不是进程:

When you use multiprocessing.dummy, you're using threads, not processes:

multiprocessing.dummy复制multiprocessing的API，但不是 不仅仅是threading模块的包装器.

multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.

这意味着您受到全局解释器锁定(GIL)的限制，并且只有一个线程可以实际执行CPU一次绑定操作.这将使您无法充分利用CPU.如果要在所有可用内核之间获得完全的并行性，则需要解决使用multiprocessing.Pool遇到的酸洗问题.

That means you're restricted by the Global Interpreter Lock (GIL), and only one thread can actually execute CPU-bound operations at a time. That's going to keep you from fully utilizing your CPUs. If you want get full parallelism across all available cores, you're going to need to address the pickling issue you're hitting with multiprocessing.Pool.

请注意，如果需要并行化的工作是IO绑定的，或者使用释放GIL的C扩展名，则multiprocessing.dummy可能仍然有用.但是，对于纯Python代码，您需要multiprocessing.

Note that multiprocessing.dummy might still be useful if the work you need to parallelize is IO bound, or utilizes a C-extension that releases the GIL. For pure Python code, however, you'll need multiprocessing.

这篇关于Python中的multiprocessing.dummy没有使用100％cpu的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！