为什么subprocess.Popen参数长度限制小于操作系统报告的内容? [英] Why is the subprocess.Popen argument length limit smaller than what the OS reports?

问题描述

我正在Linux 3.16.0上运行Python 3.4.3.我想使用subprocess.Popen来运行一个带有单个长参数(复杂的Bash调用)的命令，大约200KiB.

I am running Python 3.4.3 on Linux 3.16.0. I want to use subprocess.Popen to run a command with a long single argument (a complex Bash invocation), roughly 200KiB.

根据getconf和xargs，这应该在我的限制范围内:

According to getconf and xargs, this should be well within my limits:

$ getconf ARG_MAX
2097152
$ xargs --show-limits < /dev/null
Your environment variables take up 3364 bytes
POSIX upper limit on argument length (this system): 2091740
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088376
Size of command buffer we are actually using: 131072

但是，Python失败的局限性很小:

However, Python fails with quite smaller limits:

>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (131072-4096)), shell=True, executable='/bin/bash')
<subprocess.Popen object at 0x7f4613b58410>
>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (262144-4096)), shell=True, executable='/bin/bash')
Traceback (most recent call last):
  [...]
OSError: [Errno 7] Argument list too long

请注意，Python限制与实际使用"命令缓冲区xargs报告大致相同.这表明xargs在某种程度上足够聪明，可以从一个较小的限制开始并根据需要增加它，但是Python却没有.

Note that the Python limit is roughly the same as the "actually using" command buffer xargs reports. This suggests that xargs is somehow smart enough to start with a smaller limit and increase it as needed, but Python is not.

问题:

1. 为什么Python限制小于2MiB的操作系统限制?
2. 我可以增加Python限制吗?
3. 如果是，怎么办?

推荐答案

单个字符串参数的最大大小限制为131072.与python无关:

The maximum size for a single string argument is limited to 131072. It has nothing to do with python:

~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null
~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/null
bash: /bin/echo: Argument list too long

实际上，MAX_ARG_STRLEN决定单个字符串的最大大小:

It is actually the MAX_ARG_STRLEN that decides the max size for a single string:

作为自2.6.23以来的附加限制，一个参数不得超过MAX_ARG_STRLEN(131072). 如果您生成一个长调用，例如"sh -c'使用长参数生成'"，这可能变得很重要. (由Xan Lopez和Ralf Wildenhues指出)

And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072). This might become relevant if you generate a long call like "sh -c 'generated with long arguments'". (pointed out by Xan Lopez and Ralf Wildenhues)

请参见对ARG_MAX 的讨论，位于"Number of of参数和一个参数的最大长度"，以及unix.stackexchange上的此问题.

您可以在binfmts.h中看到它:

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

~$ echo $(( $(getconf PAGE_SIZE)*32 )) 
131072

您可以传递多个长度为131071的字符串:

You can pass multiple strings of length 131071:

subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w"))

但是单个字符串arg不能超过131071个字节.

But a single string arg cannot be longer than 131071 bytes.

这篇关于为什么subprocess.Popen参数长度限制小于操作系统报告的内容?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！