JMH: How to setup and run a JMH benchmark

Daniel ShayaFebruary 6th, 2015Last Updated: February 6th, 2015

2 243 2 minutes read

Health Warning!

This post describes how to setup and run a simple JMH benchmark. Micro benchmarks are notoriously difficult to get right and even when you do get them right (by using tools such as JMH) they can still be misleading. Just because your code runs in a certain way in an extremely isolated artificial situation does not mean it will run in the same way inside your production code. To name but a few issues, in a real program the CPU caches will be subject to pressures from other parts of your code, any object creation will have a downstream effect on GC and the JIT may have inlined and compiled code from other parts of your code that conflict with the code you have benchmarked. Nevertheless micro benchmarks do have their place and if you are going to do them you might as well do them properly with JMH.

In a recent post I was asked to execute my tests as a JMH performance benchmark.

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM. See full documentation here.

Amongst other things JMH is great, because it takes care of warm up iterations, forking JVM processes so that benchmarks don’t interfere with each other, collating results and presenting then in a uniform manner. And there’s much much more.

I’d heard a lot about JMH and seen many JMH results but never actually run one myself. It was surprisingly easy! This is how I did it.

Add these dependencies to your Maven pom.xml file:

<dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-core</artifactId>
            <version>1.5.1</version>
        </dependency>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-generator-annprocess</artifactId>
            <version>1.5.1</version>
</dependency>

Then decide which methods you want benchmarked and add the annotation @Benchmark to them. If you need any initialisation code add it in a method which should be marked @Setup.

The easiest way to run the benchmark is by adding by adding this implementation into your main method. (See here for other ways to run your tests).

public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(MyBenchmark.class.getSimpleName())
                .forks(1)
                .build();

        new Runner(opt).run();
}

Then just run as you would any normal program and you will get all the JMH goodness!

As an example to see the format of a JMH benchmark, this is what my results looked like:

Benchmark                                         Mode  Cnt     Score    Error  Units
CompTestBenchmark.bmCustomComparator             thrpt   20  2598.617 ± 67.634  ops/s
CompTestBenchmark.bmJDKComparator                thrpt   20   751.110 ± 14.835  ops/s
CompTestBenchmark.bmNoVTLComparator              thrpt   20  1348.750 ± 30.382  ops/s
CompTestBenchmark.bmNoVTLOrAutoBoxingComparator  thrpt   20  2202.995 ± 43.673  ops/s

There are an enormous number of bells and whistles to fine tune your benchmarks which I’m not going into here but hopefully this will get you up and running.