许多Shell命令架构 [英] many Shell Commands architecture

问题描述

在工作中，我们使用docker和docker-compose，我们的开发人员需要在本地启动许多容器并导入大型数据库，有许多服务需要一起运行才能使开发成功且容易。

at work, we are using docker and docker-compose, our developers need to start many containers locally and import a large database, there are many services that need to run together for development to be successful and easy.

所以我们将可重用函数定义为 make 命令，以使代码更易于维护，还有另一种方法可以更好地定义和重用许多Shell命令

so we sort of define reusable functions as make commands to make the code easier to maintain, is there another way to define and reuse many shell commands better than make.

对我们来说，由于网络限制，在本地运行docker是唯一的选择。

for us due to network limitations running docker locally is the only option.

我们设法解决了这一难题，并使开发人员能够通过将复杂的shell命令抽象到多个 make 目标后面，并为了拆分控制这些众多的 make 目标，可以使工作更轻松我们的Docker基础架构和容器我们决定将目标分散在扩展名为 .mk 的许多文件中。

we managed to solve this challenge and make our developers' life easier by abstracting away complex shell commands behind multiple make targets, and in order to split these numerous make targets that control our docker infrastructure and containers we decided to split the targets among many files with .mk extension.

有多个 make 命令，就像其中的40条命令一样，其中一些是低级命令，有些是卑鄙的命令

there are multiple make commands, like 40 of them, some of them are low level, some are meant to be called by developers to do certain tasks.

make launch_app
make import_db_script
make build_docker_images

，但是最近事情开始变得有点慢了，通过 make 命令调用其他<在内部使用code> make 命令，每个 make 调用都花费大量时间，因为每个较低级别的 make 调用必须遍历所有定义的 .mk 文件，并进行一些计算，如我们在运行 make时所示- d ，所以它开始增加了可观的开销。

but lately things are starting to become a little slow, with make commands calling other make commands internally, each make call is taking significant amount of time, since each lower level make call has to go through all defined .mk files, and do some calculations, as it shows when we run make -d, so it starts to add up to a considerable overhead.

是否可以使用make以外的任何方式来管理一组复杂的shell命令，而

is there any way to manage a set of complex shell commands using anything other than make, while still being easy for our developers to call.

预先感谢。

推荐答案

，您总是可以只使用Shell脚本而不是Makefile编写Shell命令。使用外壳函数，外壳变量等可以对其进行管理。您不会在示例中说明使用make构造的复杂程度。

Well, you could always just write your shell commands in a shell script instead of a makefile. Using shell functions, shell variables, etc. it can be managed. You don't give examples of how complex your use of make constructs is.

StackOverflow并不是真正的开放式问题，例如什么是最好的XYZ。因此，我将这个问题视为我如何加快我的makefile的速度。

StackOverflow is not really a place to ask open-ended questions like "what's the best XYZ". So instead I'll treat this question as, "how can I speed up my makefiles".

对我来说，听起来你的makefile编写得很差。再说一次，您没有显示任何示例，但是听起来您的规则正在调用许多子命令（例如，规则配方运行 $（MAKE）等）。意味着调用了许多进程，解析了许多makefile，等等。为什么您不只拥有make的一个实例并使用先决条件而不是子make来运行其他目标？您仍然可以将makefile拆分为单独的文件，然后使用 include ... 将它们全部收集到单个make实例中。

To me it sounds like you just have poorly written makefiles. Again, you don't show any examples but it sounds like your rules are invoking lots of sub-makes (e.g., your rule recipes run $(MAKE) etc.) That means lots of processes invoked, lots of makefiles parsed, etc. Why don't you just have a single instance of make and use prerequisites, instead of sub-makes, to run other targets? You can still split the makefiles up into separate files then use include ... to gather them all into a single instance of make.

此外，如果您不需要自己重建makefile，则应确保禁用可能会尝试这样做的内置规则。实际上，如果您仅使用make运行docker程序，则可以禁用 all 内置规则并加快处理速度。只需将其添加到您的makefile中即可：

Also, if you don't need to rebuild the makefiles themselves you should be sure to disable the built-in rules that might try to do that. In fact, if you are just using make to run docker stuff you can disable all the built-in rules and speed things up a good bit. Just add this to your makefile:

MAKEFLAGS += -r

（请参见选项摘要有关此选项的详细信息。）

(see Options Summary for details of this option).

ETA

您不会说GNU的哪个版本使您正在使用或正在运行的操作系统。您没有显示所使用食谱的任何示例，因此我们可以看到它们的结构。

You don't say what version of GNU make you're using, or what operating system you're running on. You don't show any examples of the recipes you're using so we can see how they are structured.

问题是您的问题事情进展缓慢可行甚至定义的。例如，我每天使用的软件有41个makefile，其中包含22,500行（从cmake生成，这意味着它们效率不高：它们是通用的makefile，没有使用GNU make功能）。在没有任何实际要做的情况下运行我的构建所花费的时间（因此，基本上整个时间都是通过解析makefile来完成的），是0.35秒。

The problem is that your issue, "things are slow", is not actionable, or even defined. As an example, the software I work on every day has 41 makefiles containing 22,500 lines (generated from cmake, which means they are not as efficient as they could be: they are generic makefiles and not using GNU make features). The time it takes for my build to run when there is nothing to actually do (so, basically the entire time is taken by parsing the makefiles), is 0.35 seconds.

在您的评论中，您建议您有10个makefile和50个变量...我无法想象这种大小的makefile会导致任何可检测到的缓慢。给出这些信息，我并不感到惊讶， -r 并没有太大的不同。

In your comments you suggest you have 10 makefiles and 50 variables... I can't imagine how any detectable slowness could be caused by this size of makefile. I'm not surprised, given this information, that -r didn't make much difference.

因此，一定有一些东西关于导致缓慢的特定makefile文件：缓慢不是make固有的。显然，我们不能仅仅猜测可能是什么。您必须对此进行调查。

So, there must be something about your particular makefiles which is causing the slowness: the slowness is not inherent in make. Obviously we cannot just guess what that might be. You will have to investigate this.

使用 time make launch_app 。

现在需要使用 time make -n launch_app 。这将读取所有makefile，但实际上不会运行任何命令。

Now use time make -n launch_app. This will read all makefiles but not actually run any commands. How long does that take?

如果 make -n 不需要花费可辨别的时间，则问题不在于make，而在于您编写的食谱并切换到其他工具来运行相同的食谱将无济于事。

If make -n takes no discernible time then the issue is not with make, but rather with the recipes you've written and switching to a different tool to run those same recipes won't help.

如果 make -n 需要时间很长，那么makefile中的内容就会变慢。您应该检查它是否用于 $（shell ...）和可能的 $（通配符...）的使用；这些就是缓慢发生的地方。您可以在它们周围添加 $（info ...）语句以在它们运行之前和之后获取输出：也许它们运行了很多次。

If make -n takes a noticeable amount of time then something in your makefiles is slow. You should examine it for uses of $(shell ...) and possibly $(wildcard ...); those are where the slowness will happen. You can add $(info ...) statements around them to get output before and after they run: maybe they're running lots of times unexpectedly.

没有缓慢的具体例子，我们无能为力。

Without specific examples of things that are slow, there's nothing else we can do to help.

这篇关于许多Shell命令架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！