在多处理中启动嵌套流程 [英] Launching nested processes in multiprocessing

问题描述

我有一个主文件，它启动多个进程，而其中一个进程又启动多个进程.我在启动嵌套的过程集时遇到问题.

I have a main file that launches multiple processes and one of the processes again launches multiple processes. I am having problems launching the nested set of processes.

我在一个文件中包含以下代码:

I have the following code in one file:

# parallel_test.py
import Queue
import multiprocessing
import time
import threading


def worker(q):
    while not q.empty():
        try:
            row = q.get(False)
            print row

            time.sleep(1)

        except Queue.Empty:
            break


def main():
    print 'creating queue'
    q = multiprocessing.Queue()

    print 'enqueuing'
    for i in range(100):
        q.put(i)

    num_processes = 15
    pool = []

    for i in range(num_processes):
        print 'launching process {0}'.format(i)
        p = multiprocessing.Process(target=worker, args=(q,))
        p.start()
        pool.append(p)

    for p in pool:
        p.join()

if __name__ == '__main__':
    main()

单独运行此文件python parallel_test.py可以正常工作，并按预期打印数字.但是从另一个文件作为另一个进程启动它会导致问题.我的主文件:

Running this file alone python parallel_test.py works fine and prints the numbers as expected. But launching it from another file as another Process causes problem. My main file:

# main_loop_test.py
import parallel_test
from multiprocessing import Pool
import time


def main():
    targets = [parallel_test.main]

    running = True

    while running:
        try:
            p = Pool(12)

            for target in targets:
                p.apply_async(target)

            p.close()  # For some reason you need to run close() before join()
            p.join()  # What for all the steps to be done

            print 'All steps done'

            time.sleep(2)

        except KeyboardInterrupt as e:
            print "<<<<<<<<<<<<<<<<<<CAUGHT KEYBOARD INTERRUPT FROM USER>>>>>>>>>>>>>>>>>>>"
            running = False


if __name__ == '__main__':
    main()

它似乎尝试启动一个进程(不执行任何操作)，然后退出该功能，然后main_loop_test.py打印所有步骤完成".没有数字被打印过.输出:

It parallel_test.py seems to try and launch one process (which does nothing) and then exits the function and main_loop_test.py prints 'All steps done'. No numbers are ever printed. Output:

creating queue
enqueuing
launching process 0
All steps done
creating queue
enqueuing
launching process 0
All steps done

出了什么问题?使用Pool而不是自己在parallel_test.py中管理进程时，我遇到了同样的问题.不过，用线程代替多处理是可行的.

What's going wrong? I get the same problem using Pool instead of managing the processes myself in parallel_test.py. Replacing multiprocessing with threading works though.

推荐答案

当您从另一个程序中将其作为子进程调用时，您无法从parallel_test创建子进程，原因是该进程已被创建为守护进程，并且如链接 https://docs.python.org/2/library/multiprocessing.html ，不允许守护进程创建子进程.您必须通过将进程的daemon属性设置为false来将其创建为非守护进程，如下所示.

You are not able to create child process from parallel_test when you invoke it as child process from another program for the reason that the process is getting created as daemonic process and as it is mentioned in the link https://docs.python.org/2/library/multiprocessing.html , daemonic process is not allowed to create child process. You have to create the process as non daemonic process by setting the daemon property of the process to be false as like below.

p = multiprocessing.Process(target=test.main)
p.daemon = False
p.start()
p.join()

我不确定通过Pool模块创建子进程时如何设置守护程序属性.您可以尝试通过初始值设定项列表传递此属性.

I am not sure how to set the daemon property when you create child process through Pool module. You can try to pass this property through initializer list.

这篇关于在多处理中启动嵌套流程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！