如何绕过 Linux “Too Many Arguments"限制 [英] How to get around the Linux "Too Many Arguments" limit

问题描述

我必须将 256Kb 的文本作为参数传递给aws sqs"命令，但在命令行中遇到了大约 140Kb 的限制.这已经在很多地方讨论过，它在 2.6.23 内核的 Linux 内核中得到了解决.

但无法让它工作.我正在使用 3.14.48-33.39.amzn1.x86_64

这是一个简单的测试示例:

#!/bin/bash大小=1000而 [ $SIZE -lt 300000 ]做回声$SIZE"VAR="`head -c $SIZE </dev/zero | tr '' 'a'`"./foo "$VAR"让 SIZE="( $SIZE * 20 )/19"完毕

foo 脚本就是:

#!/bin/bashecho -n "$1" |wc -c

我的输出是:

117037123196123196129680129680136505./testCL: line 11: ./foo: 参数列表太长143689./testCL: line 11: ./foo: 参数列表太长151251./testCL: line 11: ./foo: 参数列表太长159211

那么，问题是如何修改testCL 脚本是否可以传递256Kb 的数据?顺便说一句，我尝试将 ulimit -s 65536 添加到脚本中，但没有帮助.

如果这显然是不可能的，我可以解决这个问题，但你能否从我上面的链接中阐明这句话

<块引用>

"虽然 Linux 不是 Plan 9，但在 2.6.23 Linux 中添加了变量参数长度.理论上你不应该经常打参数"再次列出太长"错误，但此补丁也限制了最大值参数长度为最大堆栈限制 (ulimit -s) 的 25%."

解决方案

---

我终于能够将 <= 256 KB 作为单个命令行参数传递(请参阅底部的编辑 (4)).但是，请仔细阅读我是如何做到的，然后自己决定是否要采用这种方式.至少你应该能够理解为什么你会被我发现的卡住".

<小时>

随着ARG_MAX 与ulim -s/4 的耦合，引入了MAX_ARG_STRLEN 作为最大值.参数的长度:

/** linux/fs/exec.c** 版权所有 (C) 1991, 1992 Linus Torvalds*/

<块引用>

...

#ifdef CONFIG_MMU/** 新生的 bprm->mm 在 exec_mmap() 之前是不可见的，但它可以* 占用大量内存，将这些页面记入当前->mm 临时* 对于 oom_badness()->get_mm_rss().一旦 exec 成功或失败，我们* 通过 acct_arg_size(0) 将计数器改回来.*/

<块引用>

...

static bool valid_arg_len(struct linux_binprm *bprm, long len){返回 len <= MAX_ARG_STRLEN;}

<块引用>

...

#else

<块引用>

...

static bool valid_arg_len(struct linux_binprm *bprm, long len){返回 len <= bprm->p;}#endif/* CONFIG_MMU */

<块引用>

...

static int copy_strings(int argc, struct user_arg_ptr argv,结构 linux_binprm *bprm){

<块引用>

...

 str = get_user_arg_ptr(argv, argc);

<块引用>

...

 len = strnlen_user(str, MAX_ARG_STRLEN);如果(！len)出去；ret = -E2BIG;如果(！valid_arg_len(bprm，len))出去；

<块引用>

...

}

<块引用>

...

MAX_ARG_STRLEN 定义为 linux/include/uapi/linux/binfmts.h 中页面大小的 32 倍:

<块引用>

...

/** 这些是传递给字符串的最大长度和最大数量* execve() 系统调用.MAX_ARG_STRLEN 本质上是随机的，但用于* 防止内核受到地址错误的指针的过度影响.* 选择 MAX_ARG_STRINGS 以适合有符号的 32 位整数.*/#define MAX_ARG_STRLEN (PAGE_SIZE * 32)#define MAX_ARG_STRINGS 0x7FFFFFFF

<块引用>

...

默认页面大小为 4 KB，因此您不能传递超过 128 KB 的参数.

我现在无法尝试，但如果可能，在您的系统上切换到大页面模式(页面大小为 4 MB)可以解决此问题.

有关更多详细信息和参考，请参阅此答案到一个关于 Unix & 的类似问题Linux SE.

<小时>

(1)根据 this answer 可以通过以下方式将 x86_64 Linux 的页面大小更改为 1 MB在内核配置中启用 CONFIG_TRANSPARENT_HUGEPAGE 并将 CONFIG_TRANSPARENT_HUGEPAGE_MADVISE 设置为 n.

(2)使用上述配置更改重新编译内核后，getconf PAGESIZE 仍然返回 4096.根据 这个答案 还需要 CONFIG_HUGETLB_PAGE 我可以通过 CONFIG_HUGETLBFS.我正在重新编译，将再次测试.

(3)我在启用 CONFIG_HUGETLBFS 的情况下重新编译了我的内核，现在 /proc/meminfo 包含相应的 HugePages_* 条目在 内核文档的相应部分.但是，根据getconf PAGESIZE 的页面大小仍然没有变化.因此，虽然我现在应该能够通过 mmap 调用请求大页面，但内核的默认页面大小决定 MAX_ARG_STRLEN 仍然固定为 4 KB.

(4)我将 linux/include/uapi/linux/binfmts.h 修改为 #define MAX_ARG_STRLEN (PAGE_SIZE * 64)，重新编译我的内核，现在你的代码生成:

<块引用>

...

117037123196123196129680129680136505143689151251159211

<块引用>

...

227982227982239981239981252611252611265906./testCL: line 11: ./foo: 参数列表太长279901./testCL: line 11: ./foo: 参数列表太长294632./testCL: line 11: ./foo: 参数列表太长

所以现在限制按预期从 128 KB 移动到 256 KB.我不知道潜在的副作用.据我所知，我的系统似乎运行得很好.

I have to pass 256Kb of text as an argument to the "aws sqs" command but am running into a limit in the command-line at around 140Kb. This has been discussed in many places that it been solved in the Linux kernel as of 2.6.23 kernel.

But cannot get it to work. I am using 3.14.48-33.39.amzn1.x86_64

Here's a simple example to test:

#!/bin/bash

SIZE=1000
while [ $SIZE -lt 300000 ]
do
   echo "$SIZE"
   VAR="`head -c $SIZE < /dev/zero | tr '' 'a'`"
   ./foo "$VAR"
   let SIZE="( $SIZE * 20 ) / 19"
done

And the foo script is just:

#!/bin/bash
echo -n "$1" | wc -c

And the output for me is:

117037
123196
123196
129680
129680
136505
./testCL: line 11: ./foo: Argument list too long
143689
./testCL: line 11: ./foo: Argument list too long
151251
./testCL: line 11: ./foo: Argument list too long
159211

So, the question how do I modify the testCL script is it can pass 256Kb of data? Btw, I have tried adding ulimit -s 65536 to the script and it didn't help.

And if this is plain impossible I can deal with that but can you shed light on this quote from my link above

"While Linux is not Plan 9, in 2.6.23 Linux is adding variable argument length. Theoretically you shouldn't hit frequently "argument list too long" errors again, but this patch also limits the maximum argument length to 25% of the maximum stack limit (ulimit -s)."

解决方案

---

edit:

I was finally able to pass <= 256 KB as a single command line argument (see edit (4) in the bottom). However, please read carefully how I did it and decide for yourself if this is a way you want to go. At least you should be able to understand why you are 'stuck' otherwise from what I found out.

---

With the coupling of ARG_MAX to ulim -s / 4 came the introduction of MAX_ARG_STRLEN as max. length of an argument:

/*
 *  linux/fs/exec.c
 *
 *  Copyright (C) 1991, 1992  Linus Torvalds
 */

...

#ifdef CONFIG_MMU
/*
 * The nascent bprm->mm is not visible until exec_mmap() but it can
 * use a lot of memory, account these pages in current->mm temporary
 * for oom_badness()->get_mm_rss(). Once exec succeeds or fails, we
 * change the counter back via acct_arg_size(0).
 */

...

static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
 return len <= MAX_ARG_STRLEN;
}

...

#else

...

static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
  return len <= bprm->p;
}

#endif /* CONFIG_MMU */

...

static int copy_strings(int argc, struct user_arg_ptr argv,
      struct linux_binprm *bprm)
{

...

    str = get_user_arg_ptr(argv, argc);

...

    len = strnlen_user(str, MAX_ARG_STRLEN);
    if (!len)
      goto out;

    ret = -E2BIG;
    if (!valid_arg_len(bprm, len))
      goto out;

...

}

...

MAX_ARG_STRLEN is defined as 32 times the page size in linux/include/uapi/linux/binfmts.h:

...

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

...

The default page size is 4 KB so you cannot pass arguments longer than 128 KB.

I can't try it now but maybe switching to huge page mode (page size 4 MB) if possible on your system solves this problem.

For more detailed information and references see this answer to a similar question on Unix & Linux SE.

---

edits:

(1) According to this answer one can change the page size of x86_64 Linux to 1 MB by enabling CONFIG_TRANSPARENT_HUGEPAGE and setting CONFIG_TRANSPARENT_HUGEPAGE_MADVISE to n in the kernel config.

(2) After recompiling my kernel with the above configuration changes getconf PAGESIZE still returns 4096. According to this answer CONFIG_HUGETLB_PAGE is also needed which I could pull in via CONFIG_HUGETLBFS. I am recompiling now and will test again.

(3) I recompiled my kernel with CONFIG_HUGETLBFS enabled and now /proc/meminfo contains the corresponding HugePages_* entries mentioned in the corresponding section of the kernel documentation. However, the page size according to getconf PAGESIZE is still unchanged. So while I should be able now to request huge pages via mmap calls, the kernel's default page size determining MAX_ARG_STRLEN is still fixed at 4 KB.

(4) I modified linux/include/uapi/linux/binfmts.h to #define MAX_ARG_STRLEN (PAGE_SIZE * 64), recompiled my kernel and now your code produces:

...

117037
123196
123196
129680
129680
136505
143689
151251
159211

...

227982
227982
239981
239981
252611
252611
265906
./testCL: line 11: ./foo: Argument list too long
279901
./testCL: line 11: ./foo: Argument list too long
294632
./testCL: line 11: ./foo: Argument list too long

So now the limit moved from 128 KB to 256 KB as expected. I don't know about potential side effects though. As far as I can tell, my system seems to run just fine.

这篇关于如何绕过 Linux “Too Many Arguments"限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！