在具有自己路径的不同python可执行文件下生成multiprocessing.Process [英] Spawn multiprocessing.Process under different python executable with own path

问题描述

我有两个版本的Python（实际上是两个conda环境）

I have two versions of Python (these are actually two conda environments)

/path/to/bin-1/python
/path/to/bin-2/python

从一个python版本开始希望使用 multiprocessing.Process 对象启动在其他版本中运行的功能。事实证明，使用 set_executable 方法是可行的：

From one version of python I want to launch a function that runs in the other version using something like the multiprocessing.Process object. It turns out that this is doable using the set_executable method:

ctx = multiprocess.get_context('spawn')
ctx.set_executable('/path/to/bin-2/python')

实际上我们可以看到它实际上是使用该可执行文件启动的：

And indeed we can see that this does in fact launch using that executable:

def f(q):
    import sys
    q.put(sys.executable)

if __name__ == '__main__':
    import multiprocessing
    ctx = multiprocessing.get_context('spawn')
    ctx.set_executable('/path/to/bin-2/python')
    q = ctx.Queue()
    proc = ctx.Process(target=f, args=(q,))
    proc.start()
    print(q.get())

$ python foo.py
/path/to/bin-2/python

但是路径错误

但是，当我对 sys.path 而不是 sys.executable 做同样的事情时，而是打印出托管python进程的sys.path她比我从运行 / path / to / bin-2 / python -c import sys;会发现的sys.path; print（sys.path）。

However Path is Wrong

However when I do the same thing with sys.path rather than sys.executable I find that the sys.path for the hosting python process is printed out instead, rather than the sys.path I would find from running /path/to/bin-2/python -c "import sys; print(sys.path)" directly.

如果我使用fork，我已经习惯了这种事情。 code>'spawn'的行为就像我从外壳进入python解释器一样。

I'm used to this sort of thing if I use fork. I would have expected 'spawn' to act the same as though I had entered the python interpreter from the shell.

是否有可能使用多重处理库来运行函数并使用另一个Python可执行文件中的队列，而该环境与我从外壳启动它所需要的环境一样？

Is it possible to use the multiprocessing library to run functions and use Queues from another Python executable with the environment that it would have had had I started it from the shell?

更广泛地说，如何填充sys.path？以这种方式使用多处理与直接启动解释器之间有什么区别？

More broadly, how does sys.path get populated and what is different between using multiprocessing in this way and launching the interpreter directly?

推荐答案

我遇到了同样的问题，我的系统级Python可执行文件位于 / path / to / bin-1 / python ，然后我使用 virtualenv 创建了一个虚拟环境，其中包含位于 / path / to / bin-2 / python 的另一个Python可执行文件。为 / p所需的生成过程设置正确的路径/环境ath / to / bin-2 / python ，我最终将代码从 activate_this.py 复制到 virtualenv 文件夹到 f（q）。

I ran into the same problem. My system wide Python executable is at /path/to/bin-1/python, and I created a virtual environment using virtualenv containing another Python executable at /path/to/bin-2/python. To set up the right path / environment for the spawned process needed for /path/to/bin-2/python, I ended up copying the code from activate_this.py in the virtualenv folder to f(q).

def f(q):
    import sys, os

    def active_virtualenv(exec_path):
        """
        copy virtualenv's activate_this.py
        exec_path: the python.exe path from sys.executable
        """
        # set env. var. PATH
        old_os_path = os.environ.get('PATH', '')
        os.environ['PATH'] = os.path.dirname(os.path.abspath(exec_path)) + os.pathsep + old_os_path
        base = os.path.dirname(os.path.dirname(os.path.abspath(exec_path)))
        # site-pachages path
        if sys.platform == 'win32':
            site_packages = os.path.join(base, 'Lib', 'site-packages')
        else:
            site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
        # modify sys.path
        prev_sys_path = list(sys.path)
        import site
        site.addsitedir(site_packages)
        sys.real_prefix = sys.prefix
        sys.prefix = base
        # Move the added items to the front of the path:
        new_sys_path = []
        for item in list(sys.path):
            if item not in prev_sys_path:
                new_sys_path.append(item)
                sys.path.remove(item)
        sys.path[:0] = new_sys_path
        return None

    active_virtualenv(sys.executable)
    q.put(sys.executable)
    # check some unique package in this env.
    import special_package
    print "package version: {}".format(special_package.__version__)


if __name__ == '__main__':
    import multiprocessing
    multiprocessing.set_executable('/path/to/bin-2/python')
    q = multiprocessing.Queue()
    proc = multiprocessing.Process(target=f, args=(q,))
    proc.start()
    proc.join()
    print(q.get())

标准输出：

$ python foo.py
/path/to/bin-2/python
package version: unique_version_only_in_virtualenv

我不是一件事因此可以确定 sys 和 os 是 import 之前 active_virtualenv（），这意味着它们来自系统范围的Python env。但是我在 f（q）中需要的其他软件包来自虚拟环境。切换环境后，也许值得重新导入。

One thing I'm not so certain is sys and os are imported before active_virtualenv(), which means they are from system wide Python env. But other packages I need in f(q) are from virtual env. Maybe it's worth re-import them after switching env.

这篇关于在具有自己路径的不同python可执行文件下生成multiprocessing.Process的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！