使用importlib选择一个模块并在多处理功能中使用 [英] Select a module using importlib and use in multiprocessing functions

问题描述

我想在我的主函数中根据传递给Python脚本的参数选择要导入的模块.所以，我正在使用其中之一

I'd like to select in my main function which module to import based on an argument passed to the Python script. So, I'm using one of

blah = importlib.import_module("blah1")
blah = importlib.import_module("blah2")

'blahX'是同一接口的不同实现.

where the 'blahX' are different implementations of the same interface.

我还想使用multiprocessing模块将工作传递给不同的进程.

I also want to pass the work off to different processes using the multiprocessing module.

blah = None

def f(a, b):
    print blah.f(a,b)

if __name__ == '__main__':

    # call importlib.import_module here...

    a = 1
    b = 2
    p = multiprocessing.Process(target=f, args=(a, b))
    p.start()
    p.join()

问题在于传递给multiprocessing.Process的函数不知道我在main中导入的模块.这与我使用import

The problem is that the functions passed to multiprocessing.Process aren't aware of the module I imported in main. This is different than if I use import

import blah1 as blah
#import blah2 as blah

但是后来我失去了在运行时选择模块的能力.

but then I loose the ability to choose a module at run-time.

如何修复此设计?

推荐答案

调用mp.Process(...)时，多处理模块派生一个子进程(在Unix上)或启动一个新的Python进程并导入调用模块(在Windows上).在Unix上，您当前的代码可以正常工作，因为派生时的所有全局变量都由新的子进程复制.

When you call mp.Process(...), the multiprocessing module forks a subprocess (on Unix) or starts a new Python process and imports the calling module (on Windows). On Unix your current code would work because all globals at the time of the fork are copied by the new subprocess.

但是，在Windows上，调用模块已导入.由于blah的定义，例如

On Windows, however, the calling module is imported. Since the definition of blah, e.g.,

blah = importlib.import_module("math")

在内部受到保护

if __name__ == '__main__':

blah的新定义不会转移到子流程中.

the new definition of blah does not transfer to the subprocess.

因此，要使其与Windows一起使用，您可以将模块的名称传递给目标函数，然后在其中调用importlib.import_module:

So to make it work with Windows, you could pass the name of the module to the target function, and call importlib.import_module there:

import importlib
import multiprocessing as mp
blah = None

def f(a, b, name):
    blah = importlib.import_module(name)
    print blah.hypot(a,b)

if __name__ == '__main__':
    a = 1
    b = 2
    p = mp.Process(target=f, args=(a, b, "math"))
    p.start()
    p.join()

这篇关于使用importlib选择一个模块并在多处理功能中使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！