为什么 subprocess.Popen 参数长度限制小于操作系统报告的长度? [英] Why is the subprocess.Popen argument length limit smaller than what the OS reports?

问题描述

我在 Linux 3.16.0 上运行 Python 3.4.3.我想使用 subprocess.Popen 运行一个带有长单个参数(复杂的 Bash 调用)的命令，大约 200KiB.

I am running Python 3.4.3 on Linux 3.16.0. I want to use subprocess.Popen to run a command with a long single argument (a complex Bash invocation), roughly 200KiB.

根据 getconf 和 xargs，这应该在我的范围内:

According to getconf and xargs, this should be well within my limits:

$ getconf ARG_MAX
2097152
$ xargs --show-limits < /dev/null
Your environment variables take up 3364 bytes
POSIX upper limit on argument length (this system): 2091740
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088376
Size of command buffer we are actually using: 131072

但是，Python 失败的限制要小得多:

However, Python fails with quite smaller limits:

>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (131072-4096)), shell=True, executable='/bin/bash')
<subprocess.Popen object at 0x7f4613b58410>
>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (262144-4096)), shell=True, executable='/bin/bash')
Traceback (most recent call last):
  [...]
OSError: [Errno 7] Argument list too long

请注意，Python 限制与 xargs 报告的实际使用"命令缓冲区大致相同.这表明 xargs 在某种程度上足够聪明，可以从较小的限制开始并根据需要增加它，但 Python 不是.

Note that the Python limit is roughly the same as the "actually using" command buffer xargs reports. This suggests that xargs is somehow smart enough to start with a smaller limit and increase it as needed, but Python is not.

问题:

1. 为什么 Python 限制小于 2MiB 的操作系统限制?
2. 我可以增加 Python 限制吗?
3. 如果是，怎么做?

推荐答案

单个字符串参数的最大大小限制为131072，与python无关:

The maximum size for a single string argument is limited to 131072. It has nothing to do with python:

~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null
~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/null
bash: /bin/echo: Argument list too long

实际上是 MAX_ARG_STRLEN 决定了单个字符串的最大大小:

It is actually the MAX_ARG_STRLEN that decides the max size for a single string:

并且作为自 2.6.23 以来的附加限制，一个参数不得长于 MAX_ARG_STRLEN (131072).如果您生成像sh -c 'generated with long arguments'"这样的长调用，这可能会变得相关.(由 Xan Lopez 和 Ralf Wildenhues 指出)

And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072). This might become relevant if you generate a long call like "sh -c 'generated with long arguments'". (pointed out by Xan Lopez and Ralf Wildenhues)

参见关于ARG_MAX的讨论，在参数数量和一个参数的最大长度"下，以及 unix.stackexchange<上的 this question/代码>.

你可以在binfmts.h中看到它:

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

~$ echo $(( $(getconf PAGE_SIZE)*32 )) 
131072

可以传递多个长度为131071的字符串:

You can pass multiple strings of length 131071:

subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w"))

但单个字符串 arg 不能超过 131071 字节.

But a single string arg cannot be longer than 131071 bytes.

这篇关于为什么 subprocess.Popen 参数长度限制小于操作系统报告的长度?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！