用java创建快速/可靠的基准测试？ [英] Create quick/reliable benchmark with java?
问题描述
我正在尝试使用java创建基准测试。目前我有以下简单方法：
I'm trying to create a benchmark test with java. Currently I have the following simple method:
public static long runTest(int times){
long start = System.nanoTime();
String str = "str";
for(int i=0; i<times; i++){
str = "str"+i;
}
return System.nanoTime()-start;
}
我目前在另一个发生多个循环的循环中多次出现此循环时间并获得运行此方法所需的最小/最大/平均时间。然后我在另一个线程上开始一些活动并再次测试。基本上我只是想得到一致的结果......如果我有1000万次runTest循环似乎非常一致：
I'm currently having this loop multiple times within another loop that is happening multiple times and getting the min/max/avg time it takes to run this method through. Then I am starting some activity on another thread and testing again. Basically I am just wanting to get consistent results... It seems pretty consistent if I have the runTest loop 10 million times:
Number of times ran: 5
The max time was: 1231419504 (102.85% of the average)
The min time was: 1177508466 (98.35% of the average)
The average time was: 1197291937
The difference between the max and min is: 4.58%
Activated thread activity.
Number of times ran: 5
The max time was: 3872724739 (100.82% of the average)
The min time was: 3804827995 (99.05% of the average)
The average time was: 3841216849
The difference between the max and min is: 1.78%
Running with thread activity took 320.83% as much time as running without.
但这似乎有点多，需要一些时间...如果我尝试较低的数字（ 100000）在runTest循环中......它开始变得非常不一致：
But this seems a bit much, and takes some time... if I try a lower number (100000) in the runTest loop... it starts to become very inconsistent:
Number of times ran: 5
The max time was: 34726168 (143.01% of the average)
The min time was: 20889055 (86.02% of the average)
The average time was: 24283026
The difference between the max and min is: 66.24%
Activated thread activity.
Number of times ran: 5
The max time was: 143950627 (148.83% of the average)
The min time was: 64780554 (66.98% of the average)
The average time was: 96719589
The difference between the max and min is: 122.21%
Running with thread activity took 398.3% as much time as running without.
我有没有办法像这样做一致，高效/快速的基准？
Is there a way that I can do a benchmark like this that is both consistent and efficient/fast?
顺便说一句，我没有测试开始和结束时间之间的代码。我正在以某种方式测试CPU负载（看看我是如何开始一些线程活动并重新测试的）。所以我认为我正在寻找的东西替代我在runTest中的代码，它将产生更快，更一致的结果。
I'm not testing the code that is between the start and end times by the way. I'm testing the CPU load in a way (see how I'm starting some thread activity and retesting). So I think that what I'm looking for it something to substitute for the code I have in "runTest" that will yield quicker and more consistent results.
谢谢
推荐答案
简而言之：
（Micro-）基准测试非常复杂，所以使用像Benchmarking框架这样的工具 http://www.ellipticgroup.com/misc/projectLibrary.zip - 并且仍然对结果持怀疑态度（将微信任放在微基准中，Cliff博士点击）。
(Micro-)benchmarking is very complex, so use a tool like the Benchmarking framework http://www.ellipticgroup.com/misc/projectLibrary.zip - and still be skeptical about the results ("Put micro-trust in a micro-benchmark", Dr. Cliff Click).
详细信息：
有很多因素可以对结果产生强烈影响：
There are a lot of factors that can strongly influence the results:
- System.nanoTime的准确性和精确度：最糟糕的情况是System.currentTimeMillis。
- 代码预热和类加载
- 混合模式：JVMs JIT编译（参见Edwin Buck的答案）仅在充分调用代码块之后十（1500或1000次）
- 动态优化：去优化，堆栈替换，死代码消除（您应该使用循环中计算的结果，例如：打印它）
- 资源回收：garbace集合（参见Michael Borgwardt的回答）和对象最终确定
- 缓存：I / O和CPU
- 整个操作系统：屏幕保护程序，电源管理，其他进程（索引器，病毒扫描......）
- The accuracy and precision of System.nanoTime: it is in the worst case as bad as of System.currentTimeMillis.
- code warmup and class loading
- mixed mode: JVMs JIT compile (see Edwin Buck's answer) only after a code block is called sufficiently often (1500 or 1000 times)
- dynamic optimizations: deoptimization, on-stack replacement, dead code elimination (you should use the result you computed in your loop, e.g. print it)
- resource reclamation: garbace collection (see Michael Borgwardt's answer) and object finalization
- caching: I/O and CPU
- your operating system on the whole: screen saver, power management, other processes (indexer, virus scan, ...)
Brent Boyer的文章健壮的Java基准测试，第1部分：问题（ http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html ）是对所有这些问题的良好描述，以及您是否可以对它们采取什么行动（例如，使用JVM选项或事先调用ProcessIdleTask。
Brent Boyer's article "Robust Java benchmarking, Part 1: Issues" ( http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html) is a good description of all those issues and whether/what you can do against them (e.g. use JVM options or call ProcessIdleTask beforehand).
您将无法消除所有这些因素，因此进行统计是一个好主意。但是：
You won't be able to eliminate all these factors, so doing statistics is a good idea. But:
- 而不是计算最大值和最小值之间的差值，你应该努力计算标准差（结果{ 1,1000次2,3}不同于{501次1,501次3}）。
- 通过产生置信区间（例如通过自举）来考虑可靠性。
上述基准框架（ http://www.ellipticgroup.com/misc/projectLibrary.zip ）使用这些技术。你可以在Brent Boyer的文章Robust Java benchmarkinging，Part 2：Statistics and solutions（ https://www.ibm.com/developerworks/java/library/j-benchmark2/ ）。
The above mentioned Benchmark framework ( http://www.ellipticgroup.com/misc/projectLibrary.zip) uses these techniques. You can read about them in Brent Boyer's article "Robust Java benchmarking, Part 2: Statistics and solutions" ( https://www.ibm.com/developerworks/java/library/j-benchmark2/).
这篇关于用java创建快速/可靠的基准测试？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
