Pybind11多进程挂起 [英] Pybind11 multiprocessing hangs

问题描述

我正在编写一个使用Pybind11嵌入Python解释器(Windows，64位，Visual C++2017)的应用程序。从Python中，我需要产生多个进程，但它似乎不起作用。我尝试使用以下代码作为测试：

import multiprocessing
import os
import sys
import time
print("This is the name of the script: ", sys.argv[0])
print("Number of arguments: ", len(sys.argv))
print("The arguments are: " , str(sys.argv))
prefix=str(os.getpid())+"-"
if len(sys.argv) > 1:
    __name__ = "__mp_main__"

def print_cube(num):
    """
    function to print cube of given num
    """
    print("Cube: {}".format(num * num * num))

def print_square(num):
    """
    function to print square of given num
    """
    print("Square: {}".format(num * num))

print(__name__)

if __name__ == "__main__":
    print(prefix, "checkpoint 1")
    # creating processes
    p1 = multiprocessing.Process(target=print_square, args=(10, ))
    p1.daemon = True
    p2 = multiprocessing.Process(target=print_cube, args=(10, ))

    # starting process 1
    p1.start()
    print(prefix, "checkpoint 2")

    # starting process 2
    p2.start()
    print(prefix, "checkpoint 3")

    # wait until process 1 is finished
    print(prefix, "checkpoint 4")
    p1.join()
    print(prefix, "checkpoint 5")
    # wait until process 2 is finished
    p2.join()
    print(prefix, "checkpoint 6")

    # both processes finished
    print("Done!")
print(prefix, "checkpoint 7")
time.sleep(10)

从命令提示符用Python运行它，我获得：

This is the name of the script:  mp.py
Number of arguments:  1
The arguments are:  ['mp.py']
__main__
12872- checkpoint 1
12872- checkpoint 2
This is the name of the script:  C:	mpmp.py
Number of arguments:  1
The arguments are:  ['C:\tmp\mp.py']
__mp_main__
7744- checkpoint 7
Square: 100
12872- checkpoint 3
12872- checkpoint 4
12872- checkpoint 5
This is the name of the script:  C:	mpmp.py
Number of arguments:  1
The arguments are:  ['C:\tmp\mp.py']
__mp_main__
15020- checkpoint 7
Cube: 1000
12872- checkpoint 6
Done!
12872- checkpoint 7

哪一个是正确的。如果我使用Pybind11从C++项目中尝试相同的方法，则输出为：

This is the name of the script:  C:AGPXDocumentiTestPyBindx64DebugTestPyBind.exe
Number of arguments:  1
The arguments are:  ['C:\AGPX\Documenti\TestPyBind\x64\Debug\TestPyBind.exe']
__main__
4440- checkpoint 1
This is the name of the script:  C:AGPXDocumentiTestPyBindx64DebugTestPyBind.exe
Number of arguments:  4
The arguments are:  ['C:\AGPX\Documenti\TestPyBind\x64\Debug\TestPyBind.exe', '-c', 'from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=4440, pipe_handle=128)', '--multiprocessing-fork']
__mp_main__
10176- checkpoint 7

注意，在本例中，变量__name__始终设置为‘__main__’，因此我必须手动将其更改为‘__mp_main__’(多亏了sys.argv，我才能检测子进程)。这是第一个奇怪的行为。 父进程的PID4440，我可以在进程资源管理器中看到该进程。 第一个子进程的ID为10176，它到达最后的‘检查点7’，进程从进程资源管理器中消失。然而，主进程没有打印‘Checkpoint 2’，也就是说它看起来挂在‘p1.start()’上，我不明白为什么。 完整的C++代码为：

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <pybind11/embed.h>
#include <iostream>

namespace py = pybind11;
using namespace py::literals;

int wmain(int argc, wchar_t **argv)
{
    py::initialize_interpreter();
    PySys_SetArgv(argc, argv);

    std::string pyCode = std::string(R"(
import multiprocessing
import os
import sys
import time
print("This is the name of the script: ", sys.argv[0])
print("Number of arguments: ", len(sys.argv))
print("The arguments are: " , str(sys.argv))
prefix=str(os.getpid())+"-"
if len(sys.argv) > 1:
    __name__ = "__mp_main__"

def print_cube(num):
    """
    function to print cube of given num
    """
    print("Cube: {}".format(num * num * num))

def print_square(num):
    """
    function to print square of given num
    """
    print("Square: {}".format(num * num))

print(__name__)

if __name__ == "__main__":
    print(prefix, "checkpoint 1")
    # creating processes
    p1 = multiprocessing.Process(target=print_square, args=(10, ))
    p1.daemon = True
    p2 = multiprocessing.Process(target=print_cube, args=(10, ))

    # starting process 1
    p1.start()
    print(prefix, "checkpoint 2")

    # starting process 2
    p2.start()
    print(prefix, "checkpoint 3")

    # wait until process 1 is finished
    print(prefix, "checkpoint 4")
    p1.join()
    print(prefix, "checkpoint 5")
    # wait until process 2 is finished
    p2.join()
    print(prefix, "checkpoint 6")

    # both processes finished
    print("Done!")
print(prefix, "checkpoint 7")
time.sleep(10)
)");
    try
    {
        py::exec(pyCode);
    } catch (const std::exception &e) {
        std::cout << e.what();
    }
    py::finalize_interpreter();
}

有人能给我解释一下如何克服这个问题吗？

提前谢谢(我为我的英语道歉)。

Windows

好了，多亏了这个链接：https://blender.stackexchange.com/questions/8530/how-to-get-python-multiprocessing-module-working-on-windows，我解决了这个奇怪的问题(这似乎与推荐答案有关)。 这不是一个Pybind11问题，而是一个PythonC API本身。 您可以通过将sys.executable设置为等于python解释器可执行文件(python.exe)的路径，并将python代码写入文件并设置__file__变量的路径来解决此问题。也就是说，我必须添加：

import sys
sys.executable = "C:\Users\MyUserName\Miniconda3\python.exe"
__file__ = "C:\tmp\run.py"

我需要将python代码写到__file__指定的文件中，即：

FILE *f = nullptr;
fopen_s(&f, "c:\tmp\run.py", "wt");
fprintf(f, "%s", pyCode.c_str());
fclose(f);

就在执行前py::exec(pyCode)。

另外代码：

if len(sys.argv) > 1:
    __name__ = "__mp_main__"

不再是必需的。但是，请注意，在这种方式下，运行的进程不再嵌入，并且不幸的是，如果您想要直接将C++模块传递给它们，则不能这样做。

希望这能帮助其他人。

这篇关于Pybind11多进程挂起的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！