实施“制作检查"或“制作测试" [英] Implementing `make check` or `make test`

问题描述

如何使用Make实现简单的回归测试框架? (如果需要的话，我正在使用GNU Make.)

How can I implement a simple regression test framework with Make? (I’m using GNU Make, if that matters.)

我当前的makefile看起来像这样(为简单起见进行了编辑):

My current makefile looks something like this (edited for simplicity):

OBJS = jscheme.o utility.o model.o read.o eval.o print.o

%.o : %.c jscheme.h
    gcc -c -o $@ $<

jscheme : $(OBJS)
    gcc -o $@ $(OBJS)

.PHONY : clean

clean :
    -rm -f jscheme $(OBJS)

我想进行一组回归测试，例如 ，expr.in测试良好"表达式& unrecognized.in使用expr.cmp& unrecognized.cmp是每个的预期输出.手动测试如下所示:

I’d like to have a set of regression tests, e.g., expr.in testing a "good" expression & unrecognized.in testing a "bad" one, with expr.cmp & unrecognized.cmp being the expected output for each. Manual testing would look like this:

$ jscheme < expr.in > expr.out 2>&1
$ jscheme < unrecognized.in > unrecognized.out 2>&1
$ diff -q expr.out expr.cmp # identical
$ diff -q unrecognized.out unrecognized.cmp
Files unrecognized.out and unrecognized.cmp differ

我想向makefile添加一组规则，如下所示:

I thought to add a set of rules to the makefile looking something like this:

TESTS = expr.test unrecognized.test

.PHONY test $(TESTS)

test : $(TESTS)

%.test : jscheme %.in %.cmp
    jscheme < [something.in] > [something.out] 2>&1
    diff -q [something.out] [something.cmp]

我的问题:
•我要在[something]占位符中放什么?
•是否可以用消息测试expr失败"替换diff中的消息?

My questions:
• What do I put in the [something] placeholders?
• Is there a way to replace the message from diff with a message saying, "Test expr failed"?

推荐答案

如问题中所述，您的原始方法是最好的.您的每个测试都以一对预期的输入和输出的形式出现. Make有足够的能力遍历这些并运行测试.无需使用shell for循环.实际上，这样做会失去并行运行测试的机会，并且会为自己创建额外的工作来清理临时文件(不需要).

Your original approach, as stated in the question, is best. Each of your tests is in the form of a pair of expected inputs and outputs. Make is quite capable of iterating through these and running the tests; there is no need to use a shell for loop. In fact, by doing this you are losing the opportunity to run your tests in parallel, and are creating extra work for yourself in order to clean up temp files (which are not needed).

这是一个解决方案(以 bc 为例):

Here's a solution (using bc as an example):

SHELL := /bin/bash

all-tests := $(addsuffix .test, $(basename $(wildcard *.test-in)))

.PHONY : test all %.test

BC := /usr/bin/bc

test : $(all-tests)

%.test : %.test-in %.test-cmp $(BC)
    @$(BC) <$< 2>&1 | diff -q $(word 2, $?) - >/dev/null || \
    (echo "Test $@ failed" && exit 1)

all : test 
    @echo "Success, all tests passed."

该解决方案直接解决您的原始问题:

The solution directly addresses your original questions:

• 要查找的占位符分别是$<和$(word 2, $?)，分别对应于先决条件%.test-in和%.test-cmp.与@reinierpost注释临时文件相反.
• 差异消息被隐藏，并使用echo替换.
• 无论单个测试失败还是成功，都应使用make -k调用makefile来运行所有测试.
• make -k all仅在所有测试成功的情况下运行.

• The placeholders you're looking for are $< and $(word 2, $?) corresponding to the prerequisites %.test-in and %.test-cmp respectively. Contrary to the @reinierpost comment temp files are not needed.
• The diff message is hidden and replaced using echo.
• The makefile should be invoked with make -k to run all the tests regardless of whether an individual test fails or succeeds.
• make -k all will only run if all the tests succeed.

在避免定义all-tests变量时，我们避免利用文件命名约定(*.test-in)和GNU make 命令行后就会失效限制.

We avoid enumerating each test manually when defining the all-tests variable by leveraging the file naming convention (*.test-in) and the GNU make functions for file names. As a bonus this means the solution scales to tens of thousands of tests out of the box, as the length of variables is unlimited in GNU make. This is better than the shell based solution which will fall over once you hit the operating system command line limit.

这篇关于实施“制作检查"或“制作测试"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！