如何处理制作配方中的极长文件列表? [英] How do I process extremely long lists of files in a make recipe?

问题描述

因为 GNU make 允许变量尽可能大内存允许，构建大量的依赖列表没有问题.但是，如果您想在配方中实际使用这些文件列表(用于构建目标的Shell命令序列)，则会遇到问题:该命令可能超出了 shell的命令行长度限制，产生诸如参数列表过长"之类的错误.

Because GNU make allows variables to be as large as memory allows, it has no problem building massive dependency lists. However, if you want to actually use these lists of files in a recipe (sequence of shell commands for building a target), you run into a problem: the command might exceed the shell's command line length limit, producing an error such as "Argument list too long".

例如，假设我要连接列表$(INPUTS)中包含的几个文件以生成文件combined.txt.通常，我可以使用:

For example, suppose I want to concatenate several files contained in the list $(INPUTS) to produce a file combined.txt. Ordinarily, I could use:

combined.txt: $(INPUTS)
        cat $^ > $@

但是，如果$(INPUTS)包含成千上万个文件(如我的情况)，则对cat的调用太长且失败.通常有办法解决这个问题吗?可以肯定地假设存在一些命令序列，它们的行为与一个巨大的命令相同-在这种情况下，一系列cat命令(每个输入文件一个)使用>>追加到combined.txt会工作.但是如何说服make生成那些命令?

But if $(INPUTS) contains many thousands of files, as it does in my case, the call to cat is too long and fails. Is there a way to get around this problem in general? It's safe to assume that there exists some sequence of commands that have identical behaviour to the one enormous command -- in this case, a series of cat commands, one per input file, that use >> to append to combined.txt would work. But how can make be persuaded to generate those commands?

推荐答案

在寻找答案时，关于我能找到的最佳建议是

In looking for the answer, about the best suggestion I could find was to break up the list into a series of smaller lists and process them using shell for loops. But you can't always do that, and even when you can it's a messy hack: for example, it's not obvious how to get the usual make behaviour of stopping as soon as a command fails. Luckily, after much searching and experimentation, it turns out that a general solution does exist.

make配方为配方中的每一行调用一个单独的子外壳.此行为可能令人讨厌且违反直觉:例如，一行上的cd命令不会影响后续命令，因为它们在单独的子shell中运行.尽管如此，实际上这是我们需要获取make来对很长的文件列表执行操作的原因.

make recipes invoke a separate subshell for each line in the recipe. This behaviour can be annoying and counterintuitive: for example, a cd command on one line will not affect subsequent commands because they are run in separate subshells. Nevertheless it's actually what we need to get make to perform actions on very long lists of files.

通常，如果使用带反斜杠的多行换行语句来创建带有常规变量分配的文件多行"列表，则make会删除所有换行符:

Ordinarily, if you build a "multiline" list of files with a regular variable assignment that uses backslashes to break the statement over multiple lines, make removes all newlines:

# The following two statements are equivalent
FILES := a b c

FILES := \
a \
b \
c

但是，使用define指令，可以构建包含换行符的变量值.此外，如果将这样的变量替换为配方，则实际上每行都将使用单独的子外壳程序运行，例如，使用下面的makefile从/home/jbloggs中运行make test(并假设不存在名为test的文件) )会产生输出/home/jbloggs，因为cd ..命令的效果在子shell结束时会丢失:

However, using the define directive, it's possible to build variable values that contain newlines. What's more, if you substitute such a variable into a recipe, each line will indeed be run using a separate subshell, so that for example running make test from /home/jbloggs with the makefile below (and assuming no file called test exists) will produce the output /home/jbloggs, because the effect of the cd .. command is lost when its subshell ends:

define CMDS
cd ..
pwd
endef

test:
        $(CMDS)

如果我们使用define创建包含换行符的变量，则可以照常将其与其他文本连接起来，并使用所有常用的make函数进行处理.结合$(foreach)函数，可以让我们得到想要的东西:

If we create a variable that contains newlines using define, it can be concatenated with other text as usual, and processed using all the usual make functions. This, combined with the $(foreach) function, allows us to get what we want:

# Just a single newline!  Note 2 blank lines are needed.
define NL


endef

combined.txt: $(INPUTS)
        rm $@
        $(foreach f,$(INPUTS),cat $(f) >> $@$(NL))

我们要求$(foreach)将每个文件名转换为以换行符结尾的命令，该命令将在其自己的子shell中执行.对于更复杂的需求，您可以使用一系列echo命令将文件名列表写出到文件中，然后使用

We ask $(foreach) to convert each filename into a newline-terminated command, which will be executed in its own subshell. For more complicated needs, you could instead write out the list of filenames to a file with a series of echo commands and then use xargs.

• define指令被描述为可选地在第一行末尾使用=，:=或+=令牌来确定哪个此处.
• 所有配方行必须以文字制表符开头，而不是我在此处使用的8个空格.

• The define directive is described as optionally taking a =, := or += token on the end of the first line to determine which variable flavour is to be created -- but note that that only works on versions of GNU make 3.82 and up! You may well be running the popular version 3.81, as I was, which silently assigns nothing to the variable if you add one of these tokens, leading to much frustration. See here for more.
• All recipe lines must begin with a literal tab character, not the 8 spaces I have used here.

这篇关于如何处理制作配方中的极长文件列表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！