对函数内部循环内的循环进行多处理 [英] Multiprocessing a loop inside a loop inside a function

问题描述

我写了一些代码将for循环分解为多个进程，以加快计算速度.

I wrote some code to break up a for loop into multiple processes to speed up calculations.

import numpy as np
import formfactors
from subdivide_loop import subdivide_loop
import multiprocessing


def worker(start, end, triangleI, areaI, scene, kdtree, samples, output):
    form_factors = np.zeros(end-start)
    for j in range(start, end):
        triangleJ = np.array(scene[j][0:4])
        form_factors[start] = formfactors.uniform(triangleJ, triangleI, areaI, kdtree, samples)
    result = output.get(block=True)
    for j in range(start, end):
        result[j] = form_factors[j]
    output.put(result)


def calculate_formfactors(start, end, triangleI, areaI, scene, kdtree, samples, output, nb_processes,
                          max_interval_length):
    intervals = subdivide_loop(start, end, max_interval_length, nb_processes)
    print("start")
    jobs = []
    for k in range(nb_processes):
        p = multiprocessing.Process(target=worker,
                                    args=(intervals[k][0], intervals[k][1], triangleI, areaI, scene, kdtree,
                                          samples, output))
        jobs.append(p)
    for p in jobs:
        p.start()
    for p in jobs:
        p.join()
    results = output.get()
    return results

我希望能够在循环内的函数内调用calculate_formfactors()，如下所示:

I would like to be able to call calculate_formfactors() inside a function inside a loop, like this:

def outer_function():
    for i in range(1000):
        for j in range(i + 1, 1000, max_interval_length):
            form_factors = calculate_formfactors(args)

但是运行它会出现错误:

But running this gives an error:

An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:

    if __name__ == '__main__':
        freeze_support()
        ...

由于外部函数是如何工作的，因此无法分解outer_function()而不是calculate_formfactors().

Because of how the outer function works, breaking up outer_function() instead of calculate_formfactors() is not possible.

那么，有关如何执行此操作的任何建议?

So, any advice on how to do this?

推荐答案

如错误提示所示，请确保从 __ main__内调用 outer_function()(或发起它的任何方法)后卫，例如

As the error suggests, make sure your outer_function() (or whatever initiates it) is called from within an __main__ guard, e.g.

if __name__ == "__main__":
    outer_function()

它不一定是 outer_function()，但是您需要将其全部追溯到第一步，该步骤初始化了最终导致调用 multiprocessing.Process的链.()并放在上面的代码块中.

It doesn't have to be the outer_function() but you need to trace it all back to the first step that initializes the chain that ultimately leads to the call to multiprocessing.Process() and put it within the above block.

这是因为在非分支系统上，多个进程本质上是作为子进程运行的，因此从主脚本创建新进程最终将导致无限递归/进程产生.您可以在此答案中了解有关此内容的更多信息.因此，您必须确保您的多处理初始化代码仅执行一次，这是 __ main __ 保护进入的位置.

This is because on non-forking systems multiple processes are essentially run as subprocesses so creating new processes from the main script would end up with an infinite recursion/processes spawning. You can read more about it in this answer. Because of that, you have to make sure your multiprocessing initialization code executes only once which is where the __main__ guard comes in.

这篇关于对函数内部循环内的循环进行多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！