Joblib Parallel + Cython永久悬挂 [英] Joblib Parallel + Cython hanging forever

问题描述

在使用Cython创建一个使用 joblib.Parallel 的Cython扩展时，我遇到了一个非常奇怪的问题.

I have a very weird problem while creating a Python extension with Cython that uses joblib.Parallel.

以下代码按预期工作:

from joblib import Parallel, delayed
from math import sqrt

print(Parallel(n_jobs=4)(delayed(sqrt)(x) for x in range(4)))

以下代码永远挂起:

from joblib import Parallel, delayed

def mult(x):
    return x*3

print(Parallel(n_jobs=4)(delayed(mult)(x) for x in range(4)))

我不知道为什么.我使用以下 setup.py :

I have no clues why. I use the following setup.py:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("file.pyx")
)

我使用 python setup.py build_ext --inplace 创建扩展名，并将其作为 import file 导入.

I create the extension with python setup.py build_ext --inplace and I import it as import file.

谢谢！

推荐答案

一段时间后，我终于找到了解决方案:腌制程序状态以将其发送到不同的CPU时会出现死锁.我不能完全确定原因，但是检查源代码后，似乎生成了新的线程来腌制对象，而这些线程正是导致死锁的原因.

After some time I finally found the solution: there is a deadlock while pickling the program status to send it to different CPUs. I am not totally sure of the cause, but inspecting the source code, it looked as if new threads are generated to pickle the objects and these threads are the ones to cause the deadlock.

一旦生成了进程，它们便可以正常运行:通过库 multiprocessing 手动创建进程即可解决此问题.

Once the processes are generated they run normally: manually creating the processes through the library multiprocessing fixes the problem.

或者，您可以使用 multiprocessing.Pool 手动指定 start_method :

Alternatively, you can use multiprocessing.Pool manually specifying the start_method:

from multiprocessing import get_context()

if __name__ == '__main__':
    with get_context("spawn").Pool() as pool:
        ...

您可以自由选择 spawn 或 forkserver 作为 start_method .

You can freely choose spawn or forkserver as start_method.

如果需要更多信息，请访问此页面.

Visit this page if you want more information.

这篇关于Joblib Parallel + Cython永久悬挂的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！