Java HashMap Get基准测试（JMH vs循环） [英] Benchmarking Java HashMap Get (JMH vs Looping)

问题描述

我的最终目标是使用标准Java集合作为基线，为多个Java原始集合库创建一套全面的基准。在过去，我使用了编写这些微基准的循环方法。我把我在基准测试中的函数放在循环中并迭代100万次以上，这样jit就有机会进行预热。我获取循环的总时间，然后除以迭代次数，以估计单次调用我正在进行基准测试的函数所花费的时间。最近阅读了有关 JMH 项目的内容，特别是此示例： JMHSample_11_Loops 我看到了这种方法的问题。

My ultimate goal is to create a comprehensive set of benchmarks for several Java primitive collection libraries using the standard Java collections as a baseline. In the past I have used the looping method of writing these kinds of micro-benchmarks. I put the function I am benchmarking in a loop and iterate 1 million+ times so the jit has a chance to warmup. I take the total time of the loop and then divide by the number of iterations to get an estimate for the amount of time a single call to the function I am benchmarking would take. After recently reading about the JMH project and specifically this example: JMHSample_11_Loops I see the issue with this approach.

我的机器：

Windows 7 64-bit
Core i7-2760QM @ 2.40 GHz
8.00 GB Ram
jdk1.7.0_45 64-bit

这是上面描述的循环方法代码的简单示例：

Here is a stripped down simple example of the looping method code described above:

    public static void main(String[] args) {
    HashMap<Long, Long> hmap = new HashMap<Long, Long>();
    long val = 0;

    //populating the hashmap
    for (long idx = 0; idx < 10000000; idx++) {
        hmap.put(idx, idx);
    }


    Stopwatch s = Stopwatch.createStarted();
    long x = 0;
    for (long idx = 0; idx < 10000000; idx++) {
       x =  hmap.get(idx);
    }
    s.stop();
    System.out.println(s); //5.522 s
    System.out.println(x); //9999999

    //5.522 seconds / 10000000 = 552.2 nanoseconds
}

以下是我尝试使用JMH重写此基准测试：

Here is my attempt at rewriting this benchmark using JMH:

package com.test.benchmarks;

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.HashMap;
import java.util.concurrent.TimeUnit;


@State(Scope.Thread)
public class MyBenchmark {


    private HashMap<Long, Long> hmap = new HashMap<Long, Long>();
    private long key;

    @Setup(Level.Iteration)
    public void setup(){

        key = 0;

        for(long i = 0; i < 10000000; i++) {
            hmap.put(i, i);
        }
    }


    @Benchmark
    @BenchmarkMode(Mode.SampleTime)
    @OutputTimeUnit(TimeUnit.NANOSECONDS)
    public long testGetExistingKey() throws InterruptedException{

        if(key >= 10000000) key=0;
        return hmap.get(key++);
    }


    public static void main(String[] args) throws RunnerException {

        Options opt = new OptionsBuilder()
                .include(".*" + MyBenchmark.class.getSimpleName() + ".*")
                .warmupIterations(5)
                .measurementIterations(25)
                .forks(1)
                .build();

        new Runner(opt).run();

    }

}

这是结果：

 Result: 31.163 ±(99.9%) 11.732 ns/op [Average]
   Statistics: (min, avg, max) = (0.000, 31.163, 939008.000), stdev = 1831.428
   Confidence interval (99.9%): [19.431, 42.895]
  Samples, N = 263849
        mean =     31.163 ±(99.9%) 11.732 ns/op
         min =      0.000 ns/op
  p( 0.0000) =      0.000 ns/op
  p(50.0000) =      0.000 ns/op
  p(90.0000) =      0.000 ns/op
  p(95.0000) =    427.000 ns/op
  p(99.0000) =    428.000 ns/op
  p(99.9000) =    428.000 ns/op
  p(99.9900) =    856.000 ns/op
  p(99.9990) =   9198.716 ns/op
  p(99.9999) = 939008.000 ns/op
         max = 939008.000 ns/op


# Run complete. Total time: 00:02:07

Benchmark                                Mode   Samples        Score  Score error    Units
c.t.b.MyBenchmark.testGetExistingKey   sample    263849       31.163       11.732    ns/op

据我所知，JMH中的相同基准测试得到的hashmap 31 纳秒与 552 纳秒用于循环测试。 31纳秒对我来说似乎有点太快了。查看每位程序员应该知道的延迟数，主内存参考大约为100纳秒。 L2缓存引用大约为7纳秒，但具有1000万个长密钥和值的HashMap远远超过L2。 JMH的结果对我来说也很奇怪。 90％的get调用需要0.0纳秒？

As far as I can tell, the same benchmark in JMH has hashmap gets at 31 nanoseconds vs 552 nanoseconds for the looping test. 31 nanoseconds seems a little too fast for me. Looking at Latency Numbers Every Programmer Should Know a main memory reference is around 100 nanoseconds. L2 cache reference is roughly 7 nanoseconds, but the HashMap with 10 million Long keys and values well exceeds L2. Also the JMH results look strange to me. 90% of the get calls take 0.0 nanoseconds?

我假设这是用户错误。任何帮助/指针将不胜感激。谢谢。

I am assuming this is user error. Any help/pointers would be appreciated. Thanks.

更新

以下是执行<$ c $的结果c> AverageTime 运行。这更符合我的期望。谢谢@ oleg-estekhin！在下面的评论中，我提到我之前已经完成了 AverageTime 测试，结果与 SampleTime 类似。我相信在那次运行中我使用了一个HashMap，条目少得多，而且更快的查找确实有意义。

Here are the results from doing AverageTime run. This is much more inline with my expectations. Thanks @oleg-estekhin! In the comments below I mentioned that I had done the AverageTime test previously and had similar results as SampleTime. I believe doing that run I had used a HashMap with far fewer entries and that the faster lookups did make sense.

Result: 266.306 ±(99.9%) 139.359 ns/op [Average]
  Statistics: (min, avg, max) = (27.266, 266.306, 1917.271), stdev = 410.904
  Confidence interval (99.9%): [126.947, 405.665]


# Run complete. Total time: 00:07:17

Benchmark                                Mode   Samples        Score  Score error    Units
c.t.b.MyBenchmark.testGetExistingKey     avgt       100      266.306      139.359    ns/op

推荐答案

首先，循环测试测量平均时间，而JMH代码配置为采样时间。来自 Mode.SampleTime javadoc：

First, looping test measures average time, while your JMH code is configured for sample time. From the Mode.SampleTime javadoc:

采样时间：为每个采样时间操作。

Sample time: samples the time for each operation.

单个执行 Map.get（）非常快由于时间测量粒度，基础时间测量系统将为某些执行报告0的时间点（阅读 Nanotrimeing the Nanotime JMH作者的博客文章，了解更多信息）。

Individual executions of Map.get() are pretty fast to the point when the underlying time measurement system will report 0 for some of the executions due to time measurement granularity (read Nanotrusting the Nanotime blog post by the JMH author for more information).

在样本模式基准测试将单个采样时间收集到一个数组中，然后使用该数组计算平均值和百分位数。当超过一半的数组值为零时（在您的特定设置中，超过90％的数组值为零，如 p（90.0000）= 0.000 ns / op ）平均值肯定会很低但是当你看到 p（50）= 0 （尤其是 p（90）= 0 ）在你的输出中你可以得到的唯一结论是这些结果是垃圾，你需要找到另一种方法来测量该代码。

In the sample mode the benchmarks collects individual sample times into an array and then calculates averages and percentiles using that array. When more then half of the array values are zero (in your particular setup more than 90% of array values are zero, as indicated by the p(90.0000) = 0.000 ns/op) the average is bound to be pretty low but when you see p(50) = 0 (and especially p(90) = 0) in your output the only conclusion you can reliable make is that these results are garbage and you need to find another way to measure that code.

• 您应该使用 Mode.AverageTime （或 Mode.Throughput ）基准模式。对于个人调用需要相当长时间的情况，请保留 Mode.SampleTime 。

• You should use Mode.AverageTime (or Mode.Throughput) benchmark mode. Leave Mode.SampleTime for situations when individual invocation takes substantial time.

您可以添加基线 执行 if（）和 key ++ 的基准，以隔离所需的时间密钥簿记和实际的 Map.get（）时间，但你需要解释结果（上面链接的博客文章描述了缺陷和减法来自真实测量的基线。

You could add a "baseline" benchmark which executes the if () and key++ in order to isolate the time required for key bookkeeping and the actual Map.get() time, but you will need to explain the results (the blog post linked above describes pitfalls with subtracting "baselines" from "real" measurements).

您可以尝试使用 Blackhole.consumeCPU（）增加个人调用的执行时间（参见前面关于基线和相关陷阱的观点）。

You could try to use Blackhole.consumeCPU() to increase the execution time of individual invocation (see the previous point about "baselines" and associated pitfalls).

这篇关于Java HashMap Get基准测试（JMH vs循环）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！