python multiprocessing，管理器启动进程生成循环 [英] python multiprocessing, manager initiates process spawn loop

问题描述

我有一个简单的python多处理脚本，该脚本设置了一个工作池，这些工作池试图将工作输出追加到Manager列表.该脚本具有3个调用堆栈:-主调用f1产生几个调用另一个函数g1的工作进程.当尝试调试脚本时(偶然在Windows 7/64 bit/VS 2010/PyTools上)，脚本会运行到嵌套的进程创建循环中，从而产生了无数个进程.谁能确定原因?我确定我缺少一些非常简单的东西.这是有问题的代码:-

I have a simple python multiprocessing script that sets up a pool of workers that attempt to append work-output to a Manager list. The script has 3 call stacks: - main calls f1 that spawns several worker processes that call another function g1. When one attempts to debug the script (incidentally on Windows 7/64 bit/VS 2010/PyTools) the script runs into a nested process creation loop, spawning an endless number of processes. Can anyone determine why? I'm sure I am missing something very simple. Here's the problematic code: -

import multiprocessing
import logging

manager = multiprocessing.Manager()
results = manager.list()

def g1(x):
    y = x*x
    print "processing: y = %s" % y
    results.append(y)

def f1():
    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    pool = multiprocessing.Pool(processes=4)
    for (i) in range(0,15):
        pool.apply_async(g1, [i])
    pool.close()
    pool.join()

def main():
    f1()

if __name__ == "__main__":
    main()

PS:尝试将multiprocessing.freeze_support()添加到main无效.

PS: tried adding multiprocessing.freeze_support() to main to no avail.

推荐答案

基本上，sr2222在他的评论中提到的是正确的.从多处理管理器文档中，它表示____main____模块必须是可导入的由孩子们.每个管理器对象对应于一个生成的子进程" ，因此每个子进程基本上都在重新导入您的模块(您可以通过在模块作用域中向我的固定版本添加打印语句来看到它！)...导致无限递归.

Basically, what sr2222 mentions in his comment is correct. From the multiprocessing manager docs, it says that the ____main____ module must be importable by the children. Each manager " object corresponds to a spawned child process", so each child is basically re-importing your module (you can see by adding a print statement at module scope to my fixed version!)...which leads to infinite recursion.

一种解决方案是将您的经理代码移入f1():

One solution would be to move your manager code into f1():

import multiprocessing
import logging

def g1(results, x):
    y = x*x
    print "processing: y = %s" % y
    results.append(y)

def f1():
    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)
    manager = multiprocessing.Manager()
    results = manager.list()
    pool = multiprocessing.Pool(processes=4)
    for (i) in range(0,15):
        pool.apply_async(g1, [results, i])
    pool.close()
    pool.join()


def main():
    f1()

if __name__ == "__main__":
    main()

这篇关于python multiprocessing，管理器启动进程生成循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！