如何在导入的模块中使用multiprocessing.Pool? [英] How to use multiprocessing.Pool in an imported module?

问题描述

我无法在此处实施建议:

I have not been able to implement the suggestion here: Applying two functions to two lists simultaneously.

我猜是因为该模块是由另一个模块导入的，因此我的Windows生成了多个python进程?

I guess it is because the module is imported by another module and thus my Windows spawns multiple python processes?

我的问题是:如果没有if __name__ == "__main__":

My question is: how can I use the code below without the if if __name__ == "__main__":

args_m = [(mortality_men, my_agents, graveyard, families, firms, year, agent) for agent in males]
args_f = [(mortality_women, fertility, year, families, my_agents, graveyard, firms, agent) for agent in females]

with mp.Pool(processes=(mp.cpu_count() - 1)) as p:
    p.map_async(process_males, args_m)
    p.map_async(process_females, args_f)

process_males和process_females都是功能. args_m, args_f是迭代器

Both process_males and process_females are fuctions. args_m, args_f are iterators

此外，我不需要返回任何东西.代理是需要更新的类实例.

Also, I don't need to return anything. Agents are class instances that need updating.

推荐答案

if __name__ == '__main__':的想法是避免产生无限的进程.

The idea of if __name__ == '__main__': is to avoid infinite process spawning.

当腌制主脚本中定义的函数时，python必须弄清楚主脚本的哪一部分是函数代码.基本上，它将重新运行您的脚本.如果创建Pool的代码在相同的脚本中，并且不受"if main"的保护，那么通过尝试导入该函数，您将尝试启动另一个Pool，而该Pool将尝试启动另一个Pool. ...

When pickling a function defined in your main script, python has to figure out what part of your main script is the function code. It will basically re run your script. If your code creating the Pool is in the same script and not protected by the "if main", then by trying to import the function, you will try to launch another Pool that will try to launch another Pool....

因此，您应该将函数定义与实际的主脚本分开:

Thus you should separate the function definitions from the actual main script:

from multiprocessing import Pool

# define test functions outside main
# so it can be imported withou launching
# new Pool
def test_func():
    pass

if __name__ == '__main__':
    with Pool(4) as p:
        r = p.apply_async(test_func)
        ... do stuff
        result = r.get()

这篇关于如何在导入的模块中使用multiprocessing.Pool?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！