python子进程Popen环境路径? [英] python subprocess Popen environment PATH?

问题描述

我对使用 Popen() 时 subprocess 如何搜索可执行文件感到困惑.如果给定子进程的绝对路径，它会起作用，但我正在尝试使用相对路径.我发现如果我设置了环境变量 PYTHONPATH 那么我可以从那个路径中获取导入的模块，并且 PYTHONPATH 在 sys.path 中，但它似乎对行为没有帮助subprocess.Popen.我还尝试编辑 sitecustomize.py 文件，将 PYTHONPATH 添加到 os.environ，就像这样

# 将 PYTHONPATH 环境变量复制到 PATH 中以允许我们的东西使用# 子进程生成的相对路径导入操作系统如果 os.getenv('PYTHONPATH') 不是 None 并且 os.getenv('PATH') 不是 none:os.environ['PATH'] = ':'.join([os.getenv('PATH'), os.getenv('PYTHONPATH')])

并验证在启动 python 时，无论是交互方式，使用 ipython，还是通过从命令行运行脚本，PYTHONPATH 成功出现在 os.environ 中.但是，subrocess.Popen 仍然 不会在那里搜索可执行文件.如果没有指定 env kwarg，我认为它应该继承父环境?接下来，我尝试明确给出 env，首先通过制作 os.getenv 的副本，然后通过给出 env={'PATH': '/explicit/path/to/search/from'}，它仍然没有找到可执行文件.现在我被难住了.

希望一个例子能帮助更清楚地解释我的问题:

/dir/subdir1/some_executable
/dir/subdir2/some_script.py

# some_script.py从子流程导入 Popen, PIPE垃圾邮件，鸡蛋 = Popen(['../subdir1/some_executable'], stdout=PIPE, stderr=PIPE).communicate()

如果我在 /dir/subdir2 并且我运行 python some_script.py 它可以工作，但是如果我在 /dir> 我运行 python subdir2/some_script.py 即使 /dir/subdir2 在 os.environ['PATH'] 中，然后子进程将抛出 OSError: [Errno 2] No such file or directory.

解决方案

(从评论中填写详细信息，单独作答)

首先，相对路径(包含斜线的路径)永远不会在任何 PATH 中被检查，无论你做什么.它们仅相对于当前工作目录.如果您需要解析相对路径，则必须手动搜索 PATH，或者修改 PATH 以包含子目录，然后按照我的建议使用命令名称下面.

如果你想运行一个程序相对于 Python 脚本的位置，使用 __file__ 并从那里找到程序的绝对路径，然后使用 Popen 中的绝对路径.

在当前进程的环境变量中搜索PATH

其次，关于 Python 如何处理裸命令(没有斜线).基本上，在 Unix/Mac Popen 当参数为 env=None 时，其行为类似于 os.execvp(一些意外的行为已被观察到并在结尾处注明):

<块引用>

在 POSIX 上，该类使用类似 os.execvp() 的行为来执行子程序.

这实际上对于 shell=False 和 shell=True 都是正确的，前提是 env=None.函数 os 的文档中解释了此行为的含义.execvp:

<块引用>

在末尾附近包含p"的变体(execlp()、execlpe()、execvp() 和 execvpe()) 将使用 PATH 环境变量来定位程序 file.当环境被替换时(使用 exec*e 变体之一，将在下一段讨论)，新环境被用作 PATH 变量的源.

<块引用>

对于 execle()、execlpe()、execve() 和 execvpe()(注意这些都以e"结尾)，env 参数必须是一个映射，用于定义新进程的环境变量(这些用于代替当前进程的环境)；函数 execl()、execlp()、execv() 和 execvp() 都会导致新的进程继承当前进程的环境.

引用的第二段暗示 execvp 将使用当前进程的环境变量.结合第一个引用的段落，我们从当前进程的环境推导出execvp会使用环境变量PATH的值.这意味着 Popen 查看 PATH Python 启动时的值(运行 Popen 实例化)并且没有任何改变 os.environ 可以帮助您解决这个问题.

此外，在带有 shell=False 的 Windows 上，Popen 根本不关注 PATH，只会查看相对于当前工作目录.

shell=True 的作用

如果我们将 shell=True 传递给 Popen 会发生什么?在这种情况下，Popen 只需调用外壳:

<块引用>

shell 参数(默认为 False)指定是否使用 shell 作为要执行的程序.

<块引用>

也就是说，Popen相当于:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

换句话说，如果使用 shell=True，Python 将直接执行 /bin/sh，无需任何搜索(将参数 executable 传递给Popen 可以改这个，貌似如果是没有斜线的字符串，那么Python会解释为shell程序的名字，在PATH 来自当前进程的环境，即，当它在上述 shell=False 情况下搜索程序时).

反过来，/bin/sh(或我们的shellexecutable)会在它自己的环境的PATH中寻找我们想要运行的程序>，与Python(当前进程)的PATH相同，从短语也就是说..."后面的代码推导出来上面(因为该调用具有 shell=False，所以前面已经讨论过这种情况).因此，只要 env=无.

将 env 传递给 Popen

那么如果我们将 env=dict(PATH=...) 传递给 Popen(从而定义环境变量 PATH 在Popen) 运行的程序的环境?

在这种情况下，新环境用于搜索要执行的程序.引用 Popen 的文档:

<块引用>

如果env不是None，它必须是一个映射，定义了新进程的环境变量；这些用于代替继承当前进程环境的默认行为.

结合上述观察，以及使用 Popen 的实验，这意味着 Popen 在这种情况下的行为类似于函数 os.execvpe.如果 shell=False，Python 在新定义的 PATH 中搜索给定的程序.正如上面对于 shell=True 已经讨论过的，在这种情况下，程序要么是 /bin/sh，或者，如果一个程序名是用参数 executable 给出的，然后在新定义的PATH中搜索这个替代(shell)程序.

此外，如果shell=True，则在shell内部shell将用于查找args 是通过 env 传递给 Popen 的 PATH 的值.

所以使用 env != None，Popen 在 env 的键 PATH 的值中搜索(如果 env 中存在键 PATH).

将 PATH 以外的环境变量作为参数传播

有一个关于 PATH 之外的环境变量的警告:如果命令中需要这些变量的值(例如，作为正在运行的程序的命令行参数)，那么即使这些存在于提供给 Popen 的 env 中，如果没有 shell=True，它们将不会被解释.这很容易避免而无需更改 shell=True:将这些值直接插入到 list 参数 args 中，该参数提供给 Popen.(此外，如果这些值来自 Python 自身的环境，则可以使用 os.environ.get 方法来获取它们的值).

使用/usr/bin/env

如果您只需要路径评估并且不想通过 shell 运行命令行，并且在 UNIX 上，我建议使用 env 而不是 shell=True，如

path = '/dir1:/dir2'subprocess.Popen(['/usr/bin/env', '-P', path, 'progtorun', other, args], ...)

这让您可以将不同的 PATH 传递给 env 进程(使用选项 -P)，它将使用它来查找程序.它还避免了 shell 元字符问题和通过 shell 传递参数的潜在安全问题.显然，在 Windows(几乎是唯一没有 /usr/bin/env 的平台)上，您需要做一些不同的事情.

关于shell=True

引用 Popen 文档:

<块引用>

如果 shell 是 True，建议将 args 作为字符串而不是序列传递.

<块引用>

注意:阅读安全注意事项<使用 shell=True 之前的/a> 部分.

意外观察

观察到以下行为:

• 正如预期的那样，这个调用会引发 FileNotFoundError:

  subprocess.call(['sh'], shell=False, env=dict(PATH=''))

• 这个调用找到了sh，这是意外的:

  subprocess.call(['sh'], shell=False, env=dict(FOO=''))

  在打开的shell中输入echo $PATH，发现PATH的值不为空，也与PATH 在 Python 环境中.所以看起来 PATH 确实不是从 Python 继承的(正如预期的在 env != None 存在的情况下)，但它仍然是 PATH> 非空.不知道为什么会这样.

• 正如预期的那样，这个调用会引发 FileNotFoundError:

  subprocess.call(['tree'], shell=False, env=dict(FOO=''))

• 这会按预期找到 tree:

  subprocess.call(['tree'], shell=False, env=None)

I'm confused about how subprocess searches for the executable when using Popen(). It works if given absolute paths to the child process, but I'm trying to use relative paths. I've found that if I set the environment variable PYTHONPATH then I can get imported modules from that path ok, and PYTHONPATH is there in sys.path, but it doesn't seem to help with the behaviour of subprocess.Popen. I've also tried editing the sitecustomize.py file adding PYTHONPATH to os.environ, like so

# copy PYTHONPATH environment variable into PATH to allow our stuff to use
# relative paths for subprocess spawning
import os
if os.getenv('PYTHONPATH') is not None and os.getenv('PATH') is not none:
    os.environ['PATH'] = ':'.join([os.getenv('PATH'), os.getenv('PYTHONPATH')])

and verified that when starting up python , either interactively, with ipython, or by running a script from the command line, that PYTHONPATH is successfully appearing in os.environ. However, subrocess.Popen still doesn't search there for the executable. I thought it was supposed to inherit the parents environment, if no env kwarg is specified? Next I tried giving env explicitly, first by making a copy of os.getenv and secondly just by giving env={'PATH': '/explicit/path/to/search/from'}, and it still does not find the executable. Now I'm stumped.

Hopefully an example will help explain my problem more clearly:

/dir/subdir1/some_executable
/dir/subdir2/some_script.py

# some_script.py
from subprocess import Popen, PIPE
spam, eggs = Popen(['../subdir1/some_executable'], stdout=PIPE, stderr=PIPE).communicate()

If I'm in /dir/subdir2 and I run python some_script.py it works, but if I'm in /dir and I run python subdir2/some_script.py even though /dir/subdir2 is in the os.environ['PATH'], then subprocess will throw OSError: [Errno 2] No such file or directory.

解决方案

(filling in details from a comment to make a separate answer)

First off, relative paths (paths containing slashes) never get checked in any PATH, no matter what you do. They are relative to the current working directory only. If you need to resolve relative paths, you will have to search the PATH manually, or munge the PATH to include the subdirectories and then just use the command name as in my suggestion below.

If you want to run a program relative to the location of the Python script, use __file__ and go from there to find the absolute path of the program, and then use the absolute path in Popen.

Searching in the current process' environment variable PATH

Secondly, there is an issue in the Python bug tracker about how Python deals with bare commands (no slashes). Basically, on Unix/Mac Popen behaves like os.execvp when the argument env=None (some unexpected behavior has been observed and noted at the end):

On POSIX, the class uses os.execvp()-like behavior to execute the child program.

This is actually true for both shell=False and shell=True, provided env=None. What this behavior means is explained in the documentation of the function os.execvp:

The variants which include a "p" near the end (execlp(), execlpe(), execvp(), and execvpe()) will use the PATH environment variable to locate the program file. When the environment is being replaced (using one of the exec*e variants, discussed in the next paragraph), the new environment is used as the source of the PATH variable.

For execle(), execlpe(), execve(), and execvpe() (note that these all end in "e"), the env parameter must be a mapping which is used to define the environment variables for the new process (these are used instead of the current process’ environment); the functions execl(), execlp(), execv(), and execvp() all cause the new process to inherit the environment of the current process.

The second quoted paragraph implies that execvp will use the current process' environment variables. Combined with the first quoted paragraph, we deduce that execvp will use the value of the environment variable PATH from the environment of the current process. This means that Popen looks at the value of PATH as it was when Python launched (the Python that runs the Popen instantiation) and no amount of changing os.environ will help you fix that.

Also, on Windows with shell=False, Popen pays no attention to PATH at all, and will only look in relative to the current working directory.

What shell=True does

What happens if we pass shell=True to Popen? In that case, Popen simply calls the shell:

The shell argument (which defaults to False) specifies whether to use the shell as the program to execute.

That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

In other words, with shell=True Python will directly execute /bin/sh, without any searching (passing the argument executable to Popen can change this, and it seems that if it is a string without slashes, then it will be interpreted by Python as the shell program's name to search for in the value of PATH from the environment of the current process, i.e., as it searches for programs in the case shell=False described above).

In turn, /bin/sh (or our shell executable) will look for the program we want to run in its own environment's PATH, which is the same as the PATH of the Python (current process), as deduced from the code after the phrase "That is to say..." above (because that call has shell=False, so it is the case already discussed earlier). Therefore, the execvp-like behavior is what we get with both shell=True and shell=False, as long as env=None.

Passing env to Popen

So what happens if we pass env=dict(PATH=...) to Popen (thus defining an environment variable PATH in the environment of the program that will be run by Popen)?

In this case, the new environment is used to search for the program to execute. Quoting the documentation of Popen:

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of the default behavior of inheriting the current process’ environment.

Combined with the above observations, and from experiments using Popen, this means that Popen in this case behaves like the function os.execvpe. If shell=False, Python searches for the given program in the newly defined PATH. As already discussed above for shell=True, in that case the program is either /bin/sh, or, if a program name is given with the argument executable, then this alternative (shell) program is searched for in the newly defined PATH.

In addition, if shell=True, then inside the shell the search path that the shell will use to find the program given in args is the value of PATH passed to Popen via env.

So with env != None, Popen searches in the value of the key PATH of env (if a key PATH is present in env).

Propagating environment variables other than PATH as arguments

There is a caveat about environment variables other than PATH: if the values of those variables are needed in the command (e.g., as command-line arguments to the program being run), then even if these are present in the env given to Popen, they will not get interpreted without shell=True. This is easily avoided without changing shell=True: insert those value directly in the list argument args that is given to Popen. (Also, if these values come from Python's own environment, the method os.environ.get can be used to get their values).

Using /usr/bin/env

If you JUST need path evaluation and don't really want to run your command line through a shell, and are on UNIX, I advise using env instead of shell=True, as in

path = '/dir1:/dir2'
subprocess.Popen(['/usr/bin/env', '-P', path, 'progtorun', other, args], ...)

This lets you pass a different PATH to the env process (using the option -P), which will use it to find the program. It also avoids issues with shell metacharacters and potential security issues with passing arguments through the shell. Obviously, on Windows (pretty much the only platform without a /usr/bin/env) you will need to do something different.

About shell=True

Quoting the Popen documentation:

If shell is True, it is recommended to pass args as a string rather than as a sequence.

Note: Read the Security Considerations section before using shell=True.

Unexpected observations

The following behavior was observed:

• This call raises FileNotFoundError, as expected:

  subprocess.call(['sh'], shell=False, env=dict(PATH=''))
  

• This call finds sh, which is unexpected:

  subprocess.call(['sh'], shell=False, env=dict(FOO=''))
  

  Typing echo $PATH inside the shell that this opens reveals that the PATH value is not empty, and also different from the value of PATH in the environment of Python. So it seems that PATH was indeed not inherited from Python (as expected in the presence of env != None), but still, it the PATH is nonempty. Unknown why this is the case.

• This call raises FileNotFoundError, as expected:

  subprocess.call(['tree'], shell=False, env=dict(FOO=''))
  

• This finds tree, as expected:

  subprocess.call(['tree'], shell=False, env=None)
  

这篇关于python子进程Popen环境路径?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！