如何运行gem5单元测试? [英] How to run the gem5 unit tests?

问题描述

Gem5在源代码树中有多个测试，并且在以下位置有一些文档: http://www.gem5 .org/Regression_Tests ，但是这些文档不是很清楚.

Gem5 has several tests in the source tree, and there is some documentation at: http://www.gem5.org/Regression_Tests but those docs are not very clear.

有哪些测试以及如何运行?

What tests are there and how to run them?

推荐答案

单元测试与回归测试

gem5有两种测试:

Unit vs regression tests

gem5 has two kinds of tests:

• 回归:在整个模拟器上运行一些工作负载(完整的系统或syscall仿真)
• 单元:仅测试模拟器的一小部分，而不运行整个模拟器二进制文件

我们将在此答案中同时介绍这两者.

We will cover both on this answer.

2019年添加了新的测试框架，并在以下位置进行了记录: https://gem5.googlesource.com/public/gem5/+/master/TESTING.md

A new testing framework was added in 2019 and it is documented at: https://gem5.googlesource.com/public/gem5/+/master/TESTING.md

在发送补丁之前，您基本上要运行:

Before sending patches, you basically want to run:

cd tests
./main.py run -j `nproc` -t `nproc`

这将:

• 为受积极支持的ISA构建gem5:X86，ARM，由于j而具有nproc线程的RISCV
• 从gem5.org下载运行测试所需的二进制文件，例如 http://www.gem5.org/dist/current/arm/参见还: http://gem5.org/Download 当前无法下载或下载源代码树，如果您周围有一堆git worktree，这是不好的.
• 由于-t而在nproc线程上运行快速测试，该测试应在几分钟内完成

• build gem5 for the actively supported ISAs: X86, ARM, RISCV with nproc threads due to j
• download binaries required to run tests from gem5.org, e.g. http://www.gem5.org/dist/current/arm/ see also: http://gem5.org/Download It was not currently possible to download out or the source tree, which is bad if you have a bunch of git worktrees lying around.
• run the quick tests on nproc threads due to -t, which should finish in a few minutes

通过将tests/目录作为参数传递，可以在不使用cd的情况下实现与上一个命令相同的功能:

You can achieve the same as the previous command without cd by passing the tests/ directory as an argument:

./main.py run -j `nproc` -t `nproc` tests

但我希望两者都不是必需的: https://gem5.atlassian.net/browse/GEM5-397

but I wish neither were necessary: https://gem5.atlassian.net/browse/GEM5-397

这正是自动化的上游预提交测试的运行方式，如测试/詹金斯/presubmit.sh .

This is exactly what the automated upstream precommit tests are running as can be seen from tests/jenkins/presubmit.sh.

Stdout包含以下形式的清晰结果输出:

Stdout contains clear result output of form:

Test: cpu_test_DerivO3CPU_FloatMM-ARM-opt Passed                    
Test: cpu_test_DerivO3CPU_FloatMM-ARM-opt-MatchStdout Passed
Test: realview-simple-atomic-ARM-opt Failed                  
Test: realview-simple-atomic-dual-ARM-opt Failed

以及有关每个测试的详细信息可以在下面找到:

and details about each test can be found under:

tests/.testing-results/

例如:

.testing-results/SuiteUID:tests-gem5-fs-linux-arm-test.py:realview-simple-atomic-ARM-opt/TestUID:tests-gem5-fs-linux-arm-test.py:realview-simple-atomic-ARM-opt:realview-simple-atomic-ARM-opt/

尽管我们在那里只看到一些最小的stdout/stderr输出，甚至没有显示gem5 stdout.但是stderr文件确实包含完整的命令:

although we only see some minimal stdout / stderr output there which don't even show the gem5 stdout. The stderr file does however contain the full command:

CalledProcessError: Command '/path/to/gem5/build/ARM/gem5.opt -d /tmp/gem5outJtSLQ9 -re '/path/to/gem5/tests/gem5/fs/linux/arm/run.py /path/to/gem5/master/tests/configs/realview-simple-atomic.py' returned non-zero exit status 1

因此您可以删除-d和-re并重新运行以查看正在发生的事情，这可能很慢，但我看不到其他方法.

so you can remove -d and -re and re-run that to see what is happening, which is potentially slow, but I don't see another way.

请求简化从stdout直接获取原始运行命令的请求: https://gem5.atlassian.net/browse/GEM5-627

Request to make it easier to get the raw run commands from stdout directly: https://gem5.atlassian.net/browse/GEM5-627

要进一步对单个ISA进行压力测试，可以使用以下命令对一个ISA运行所有测试:

To further stress test a single ISA, you can run all tests for one ISA with:

cd tests
./main.py run -j `nproc` -t `nproc` --isa ARM --length long --length quick

每个测试都分类为long或quick，并且同时使用--length和--length进行测试.

Each test is classified as either long or quick, and using both --length runs both.

long测试通常与默认的quick测试非常相似，但是使用的是更详细的模型，因此速度较慢，例如

long tests are typically very similar to the default quick ones, but using more detailed and therefore slower models, e.g.

• tests/quick/se/10.mcf/ref/arm/linux/simple-atomic/使用更快的原子CPU很快
• tests/long/se/10.mcf/ref/arm/linux/minor-timing/很长，而次要CPU速度较慢

• tests/quick/se/10.mcf/ref/arm/linux/simple-atomic/ is quick with a faster atomic CPU
• tests/long/se/10.mcf/ref/arm/linux/minor-timing/ is long with a slower Minor CPU

待办事项:

• 从树中下载并使用测试二进制文件以在多个版本中共享.当前显示为硬编码: https://github.com/gem5/gem5/blob/bc26c0dd355f2bbaf2274dd4f549f11efcf84bba/tests/gem5/fs/linux/arm/test.py#L87 有--base-path，但是选择了gem5源的根目录.

• download and use the test binaries out of tree to share across multiple builds. Currently appears hardcoded: https://github.com/gem5/gem5/blob/bc26c0dd355f2bbaf2274dd4f549f11efcf84bba/tests/gem5/fs/linux/arm/test.py#L87 There is --base-path, but that selects the root directory of the gem5 source.

在gem5 69930afa9b63c25baab86ff5fbe632fc02ce5369中进行了测试.

Tested in gem5 69930afa9b63c25baab86ff5fbe632fc02ce5369.

2019年回归测试仅运行一项测试

列出所有可用的测试:

./main.py list --length long --length quick

这同时显示了套件和测试，例如:

This shows both suites and tests, e.g.:

SuiteUID:tests/gem5/cpu_tests/test.py:cpu_test_AtomicSimpleCPU_Bubblesort-ARM-opt

TestUID:tests/gem5/cpu_tests/test.py:cpu_test_AtomicSimpleCPU_Bubblesort-ARM-opt:cpu_test_AtomicSimpleCPU_Bubblesort-ARM-opt
TestUID:tests/gem5/cpu_tests/test.py:cpu_test_AtomicSimpleCPU_Bubblesort-ARM-opt:cpu_test_AtomicSimpleCPU_Bubblesort-ARM-opt-MatchStdout

现在您可以使用--uid进行一次测试:

And now you can run just one test with --uid:

./main.py run -j `nproc` -t `nproc` --isa ARM --uid SuiteUID:tests/gem5/cpu_tests/test.py:cpu_test_AtomicSimpleCPU_FloatMM-ARM

有点混乱，--uid必须指向SuiteUID，而不是TestUID.

A bit confusingly, --uid must point to a SuiteUID, not TestUID.

然后，当您运行测试时，如果其中任何一个失败了，而您只想运行失败的测试，则测试失败会显示以下行:

Then, when you run the tests, and any of them fails, and you want to run just the failing one, the test failure gives you a line like:

Test: cpu_test_DerivO3CPU_FloatMM-ARM-opt Passed

运行测试的唯一方法是在./main.py list的输出中针对该字符串进入grep，因为cpu_test_DerivO3CPU_FloatMM-ARM-opt不是完整的测试ID，这很烦人.

and the only way to run just the test is to grep for that string in the output of ./main.py list since cpu_test_DerivO3CPU_FloatMM-ARM-opt is not a full test ID, which is very annoyng.

2019年回归测试不在树上

默认情况下，tests/main.py将构建版本放置在源树内的gem5/build处.使用--build-dir可以测试树外构建:

By default, tests/main.py places the build at gem5/build inside the source tree. Testing an out of tree build is possible with --build-dir:

./main.py run -j `nproc` -t `nproc` --isa ARM --length quick --build-dir path/to/my/build

例如，将其放置在path/to/my/build/ARM/gem5.opt中.

如果您的构建已完成，请使用--skip-build选项也节省几秒钟:

If your build is already done, save a few scons seconds with --skip-build option as well:

./main.py run -j `nproc` -t `nproc` --isa ARM --length quick --build-dir path/to/my/build --skip-build

但是请注意，--skip-build还会跳过测试二进制文件的下载. TODO对此进行修补.

Note however that --skip-build also skips the downloading of test binaries. TODO patch that.

不建议使用这种运行测试的方式，并将其删除.

This way of running tests is deprecated and will be removed.

测试直接通过scons运行.

但是，由于测试命令有点长，因此甚至没有树内实用程序为您生成测试命令.

But since the test commands are a bit long, there is even an in-tree utility generate tests commands for you.

例如，要获取运行X86和ARM quick测试的命令，请运行:

For example, to get the command to run X86 and ARM quick tests, run:

./util/regress -n --builds X86,ARM quick

quick以外的其他选项是long或all，它们可以同时执行long和quick.

The other options besides quick are long or all to do both long and quick at the same time.

使用-n只会打印测试命令，而没有打印它实际​​上会运行它们.

With -n it just prints the test commands, and without it it actually runs them.

输出类似:

scons \
  --ignore-style \
  --no-lto \
  build/X86/gem5.debug \
  build/ARM/gem5.debug \
  build/X86/gem5.fast \
  build/ARM/gem5.fast \
  build/X86/tests/opt/quick/se \
  build/X86/tests/opt/quick/fs \
  build/ARM/tests/opt/quick/se \
  build/ARM/tests/opt/quick/fs 

TODO:为什么要构建gem5.debug和gem5.fast，然后运行/opt/测试?

TODO: why does it build gem5.debug and gem5.fast, but then runs an /opt/ test?

所以请注意这两者如何:

So note how this would both:

• 构建gem5可执行文件，例如build/X86/gem5.debug
• 运行测试，例如build/X86/tests/opt/quick/fs

• build the gem5 executables, e.g. build/X86/gem5.debug
• run the tests, e.g. build/X86/tests/opt/quick/fs

或获取命令以对所有拱门运行所有测试:

Or get the command to run all tests for all archs:

./util/regress -n all

然后，如果您只想运行这些类型的测试之一，例如您可以复制的quick X86粘贴用于该测试的scons:

Then, if you just want to run one of those types of tests, e.g. the quick X86 ones you can copy paste the scons just for that tests:

scons --ignore-style build/X86/tests/opt/quick/se

通过神奇地解析目标路径，以通常的方式运行测试以进行树外构建:

Running the tests with an out of tree build works as usual by magically parsing the target path: How to build gem5 out of tree?

scons --ignore-style /any/path/that/you/want/build/X86/tests/opt/quick/se

，或者您可以将--build-dir选项传递给util/regress:

or you can pass the --build-dir option to util/regress:

./util/regress --build-dir /any/path/that/you/want all

另一方面，引导Linux的测试需要M5_PATH中具有特定名称的Linux映像，这也很烦人.

The tests that boot Linux on the other hand require a Linux image with a specific name in the M5_PATH, which is also annoying.

但是这将非常慢，而不是每次提交后都可以运行的东西:您更有可能只想对感兴趣的ISA运行快速测试.

This would however be very slow, not something that you can run after each commit: you are more likely to want to run just the quick tests for your ISA of interest.

如果仅将源树中tests下的路径附加到测试命令，它将在给定目录下运行所有​​测试.

If you just append the path under tests in the source tree to the test commands, it runs all the tests under a given directory.

例如，我们有:

scons --ignore-style build/X86/tests/opt/quick/se

，我们注意到源树中tests下存在以下路径:

and we notice that the following path exists under tests in the source tree:

quick/se/00.hello/ref/x86/linux/simple-atomic/

所以我们通过删除ref来获得最终命令来按摩路径:

so we massage the path by removing ref to get the final command:

scons build/X86/tests/opt/quick/se/00.hello/x86/linux/simple-atomic

2019年前的回归测试:找出命令运行的确切gem5 CLI

运行测试时，它们会输出到stdout m5out路径.

Pre-2019 regression tests: Find out the exact gem5 CLI of the command run

When you run the tests, they output to stdout the m5out path.

在m5out路径中，有一个包含模拟器stdout的simout，其中包含所使用的完整gem5命令行.

Inside the m5out path, there is a simout with the emulator stdout, which contains the full gem5 command line used.

例如:

scons --ignore-style build/X86/tests/opt/quick/se

输出:

Running test in /any/path/that/you/want/build/ARM/tests/opt/quick/se/00.hello/arm/linux/simple-atomic.

和文件:

/any/path/that/you/want/build/ARM/tests/opt/quick/se/00.hello/arm/linux/simple-atomic

包含:

command line: /path/to/mybuild/build/ARM/gem5.opt \
  -d /path/to/mybuild/build/ARM/tests/opt/quick/fs/10.linux-boot/arm/linux/realview-simple-atomic \
  --stats-file 'text://stats.txt?desc=False' \
  -re /path/to/mysource/tests/testing/../run.py \
  quick/fs/10.linux-boot/arm/linux/realview-simple-atomic

2019年之前的回归测试:仅重新运行一项测试

如果您只运行了两次测试，例如与:

Pre-2019 regression tests: Re-run just one test

If you just run a test twice, e.g. with:

scons build/ARM/tests/opt/quick/fs/10.linux-boot/arm/linux/realview-simple-atomic
scons build/ARM/tests/opt/quick/fs/10.linux-boot/arm/linux/realview-simple-atomic

第二次运行不会真正重新运行测试，而只是比较前一次运行的统计信息.

the second run will not really re-run the test, but rather just compare the stats from the previous run.

要真正重新运行测试，必须先清除前一次运行生成的统计信息，然后才能重新运行:

To actually re-run the test, you must first clear the stats generated from the previous run before re-running:

rm -rf build/ARM/tests/opt/quick/fs/10.linux-boot/arm/linux/realview-simple-atomic

2019年之前的回归测试:获取测试结果

即使这很混乱... scons也不返回0成功和1失败，因此您必须解析日志.一种简单的查看方式:

Pre-2019 regression tests: Get test results

Even this is messy... scons does not return 0 success and 1 for failure, so you have to parse the logs. One easy way it to see:

scons --ignore-style build/X86/tests/opt/quick/se |& grep -E '^\*\*\*\*\* '

包含三种类型的结果:PASSSED，CHANGED或FAILED

which contains three types of results: PASSSED, CHANGED or FAILED

CHANGED主要用于统计数据比较，它们之间的差别很大，但是通常很难维护并且永久损坏，因此您应该专注于FAILED

CHANGED is mostly for stat comparisons that had a great difference, but those are generally very hard to maintain and permanently broken, so you should focus on FAILED

请注意，当前大多数测试都依赖 SPEC2000 ，除非您可以访问此非免费基准，否则它将失败...

Note that most tests currently rely on SPEC2000 and fail unless you have access to this non-free benchmark...

单元测试，将其编译为与gem5分开的可执行文件，并且仅测试一小段代码.

The unit tests, which compile to separate executables from gem5, and just test a tiny bit of the code.

目前有两种测试类型:

• UnitTest:已过时且已过时，应转换为GTest

• UnitTest: old and deprecated, should be converted to GTest

GTest:不错.使用 Google测试.

放置在他们要测试的课程旁边，例如:

Placed next to the class that they test, e.g.:

src/base/cprintf.cc
src/base/cprintf.hh
src/base/cprintftest.cc

编译并运行所有GTest单元测试:

Compile and run all the GTest unit tests:

scons build/ARM/unittests.opt

样本输出摘录:

build/ARM/base/cprintftest.opt --gtest_output=xml:build/ARM/unittests.opt/base/cprintftest.xml
Running main() from gtest_main.cc
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from CPrintf
[ RUN      ] CPrintf.Misc
[       OK ] CPrintf.Misc (0 ms)
[ RUN      ] CPrintf.FloatingPoint
[       OK ] CPrintf.FloatingPoint (0 ms)
[ RUN      ] CPrintf.Types
[       OK ] CPrintf.Types (0 ms)
[ RUN      ] CPrintf.SpecialFormatting
[       OK ] CPrintf.SpecialFormatting (0 ms)
[----------] 4 tests from CPrintf (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 4 tests.

仅编译并运行一个测试文件:

Compile and run just one test file:

scons build/ARM/base/cprintftest.opt
./build/ARM/base/cprintftest.opt

列出测试文件中可用的测试功能，然后仅运行其中之一:

List available test functions from a test file, and run just one of them:

./build/ARM/base/cprintftest.opt --gtest_list_tests
./build/ARM/base/cprintftest.opt SpecialFormatting

经过gem5 200281b08ca21f0d2678e23063f088960d3c0819测试，2018年8月.

Tested on gem5 200281b08ca21f0d2678e23063f088960d3c0819, August 2018.

从2019年开始，单元测试非常有限，因为开发人员尚未找到适当的方法来单独测试SimObjects，后者构成了模拟器的大部分，并与模拟器的其余部分紧密绑定.此未合并的补丁试图解决以下问题: https://gem5-review. googlesource.com/c/public/gem5/+/15315

As of 2019, the unit tests are quite limited, because devs haven't yet found a proper way to test SimObjects in isolation, which make up the bulk of the simulator, and are tightly bound to the rest of the simulator. This unmerged patch attempted to address that: https://gem5-review.googlesource.com/c/public/gem5/+/15315

可能已经可以使用已经存在于树中的Google Mock解决该问题，但是尚不清楚是否有人愿意耐心模拟出足够多的SimObject来实际进行此类测试.

It might be possible to work around it with Google Mock, which is already present in-tree, but it is not clear if anyone has the patience to mock out enough of SimObject to actually make such tests.

我相信唯一可行的解​​决方案是将所有测试嵌入到gem5.opt中，然后具有一个--test <testname>选项，该选项运行测试而不是运行模拟.这样，我们可以获得单个二进制文件而不会重复二进制文件的大小，但是仍然可以访问所有内容.

I believe the only practical solution is to embed all tests into gem5.opt, and then have a --test <testname> option that runs tests instead of running simulating. This way we get a single binary without duplicating binary sizes, but can still access everything.

相关问题: https://gem5.atlassian.net/browse/GEM5-433

在2019-04年左右，预维护CI在维护者给出+1之后的每个拉取请求之后运行.

Around 2019-04 a precommit CI that runs after every pull request after the maintainer gives +1.

它使用由Google提供的神奇的半内部Jenkins设置(称为Kokoro)，可降低配置可见性.

It uses a magic semi-internal Google provided Jenkins setup called Kokoro which provides low visibility on configuration.

例如: https://gem5-review. googlesource.com/c/public/gem5/+/18108 该服务器当前不夜间运行.入口点是tests/jenkins/presubmit.sh.

See this for example: https://gem5-review.googlesource.com/c/public/gem5/+/18108 That server does not currently run nightlies. The entry point is tests/jenkins/presubmit.sh.

夜色刚开始被禁用.

这是一台运行在某处的服务器，该服务器每晚对所有拱门进行快速测试，并将其发布到开发邮件列表中，这增加了该列表的无尽噪音:-)

here was a server running somewhere that runs the quick tests for all archs nightly and posts them on the dev mailing list, adding to the endless noise of that enjoyable list :-)

这是一个示例运行: https://www .mail-archive.com/gem5-dev @ gem5.org/msg26855.html

从2019年第一季度开始，gem5开发人员正在尝试设置自动魔术Google Jenkins以运行预提交测试，可以在以下网址找到原型的链接:

As of 2019Q1, gem5 devs are trying to setup an automated magic Google Jenkins to run precommit tests, a link to a prototype can be found at: https://gem5-review.googlesource.com/c/public/gem5/+/17456/1#message-e9dceb1d3196b49f9094a01c54b06335cea4ff88 This new setup uses the new testing system in tests/main.py.

截至2018年8月，许多测试已经CHANGED很长时间了.

As of August 2018, many tests have been CHANGED for a long time.

这是因为统计数据可能由于多种复杂因素而有所不同 因素.其中一些可能更准确，而其他人则不知道， 其他人只是臭虫.

This is because stats can vary due to a very wide number of complex factors. Some of those may be more accurate, others no one knows, others just bugs.

变更经常发生，以至于开发人员还没有找到适当的时间 理解并证明他们的合理性.

Changes happen so often that devs haven't found the time to properly understand and justify them.

如果您真的在乎它们为什么会发生变化，那么我最好的建议就是将它们一分为二.

If you really care about why they changed, the best advice I have is to bisect them.

但是通常最好的选择是只是在较新的gem5版本上重新运行您的旧实验，并在那里进行比较.

But generally your best bet is to just re-run your old experiments on the newer gem5 version, and compare everything there.

gem5不是周期精确的系统模拟器，因此绝对值或 一般而言，小的变化是没有意义的.

gem5 is not a cycle accurate system simulator, so absolute values or small variations are not meaningful in general.

这也告诉我们，以小幅度获得的结果是 由于噪声太大，通常对发布没有意义.

This also teaches us that results obtained with small margins are generally not meaningful for publication since the noise is too great.

该误差幅度是多少，我不知道.

What that error margin is, I don't know.

这篇关于如何运行gem5单元测试?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！