JMH with Gradle — From Easy to Simple

JMH is a Java harness for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.

Gradle is my current build tool of choice.

This post tells how to setup JMH in gradle.

Starting with the simple approach. Just include and use this plugin: https://github.com/melix/jmh-gradle-plugin.

I used it quite a while and it worked out of the box for smaller projects. On medium sized ones i run into some problems, but could resolve them by making use of its configurations (like zip64 = true ). For a larger project with a complex dependency setup however, i run into some additional classpath problems.

So i took a step back, looked what the project was doing, also cast a glance at http://gvsmirnov.ru/blog/tech/2014/03/10/keeping-your-benchmarks-separate.html and figured… it’s not as complex… and can be even less complex… E.g. i don’t see why i need that shadow jar stuff!

Step by Step

Common practice is to have the JMH benchmarks separated, like instead of putting them e.g. in src/test/java , put them into src/jmh/java. Do that in gradle by adding another source-set:

sourceSets { 
  jmh 
}

In case your benchmarks using classes from your tests, do it like that:

sourceSets { 
    jmh { 
        compileClasspath += sourceSets.test.runtimeClasspath 
        runtimeClasspath += sourceSets.test.runtimeClasspath 
        } 
}

Next you wanna setup the dependencies for that new source-set called jmh. So everything your benchmarks depend on, which is usually the project itself (with all its dependencies) and the JMH jars:

dependencies { 
    jmhCompile project 
    jmhCompile 'org.openjdk.jmh:jmh-core:1.18' 
    jmhAnnotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.18' 
}

Note: jmhCompile is bound to the jmh source-set definition. You can choose the source-set name freely. If you call it e.g. benchmark , use benchmarkCompile in the dependencies section!

Anyone still using Eclipse ? This piece helps to set it up correctly:

eclipse { 
    classpath { 
        plusConfigurations.add(configurations.jmhCompile) 
        defaultOutputDir = file('build/classes-jmh-ide') 
    } 
}

The final piece is about the execution of the benchmark. In its simplest version it looks like that:

task jmh(type: JavaExec, description: 'Executing JMH benchmarks') { 
    classpath = sourceSets.jmh.runtimeClasspath 
    main = 'org.openjdk.jmh.Main' 
}

But almost certainly you wanna enhance that version, making some of JMH’s parameter, parameter of the Gradle task. E.g. the regex to select the Benchmarks + writing the result to a JSON file:

task jmh(type: JavaExec, description: 'Executing JMH benchmarks') { 
    classpath = sourceSets.jmh.runtimeClasspath 
    main = 'org.openjdk.jmh.Main' 

    def include = project.properties.get('include', ''); 
    def format = project.properties.get('format', 'json'); 
    def resultFile = file("build/reports/jmh/result.${format}") 
    resultFile.parentFile.mkdirs() 

    args include 
    args '-rf', format 
    args '-rff', resultFile 
}

Voila, now you can run your benchmarks with:

gw jmh2 -Pinclude=“.\*MyBenchmark.\*”

Note: There are probably still more parameters you wanna pass through… see the below complete version for some more i found useful.

Complete version

Putting everything together and adding some sugar and spice (jmhHelp task, more parameters, etc..) leaves us with that:

Closing Story

Few month back i discovered another Gradle plugin — think it was about annotation processing — and told my dear friend and colleague Mascha about it. I was enthusiastic, he was like ’yeah, i would check what the plugin is doing really, i’m preferring to do simple things on my own lately’.

I virtually raised my eyebrows in a way he couldn’t see it (i like re-using things), but didn’t confront.

Now weeks later, i figured he is right! This time… ;)

Final Words

I hope you enjoyed the post! Let me know if you think it’s worth making this a plugin… ;)

Actually, there seems to be a plugin doing the JMH integration in a similar way: https://github.com/blackboard/jmh-gradle-plugin. Checked it out, i haven’t!

---

Comments

[fsykop]

I tried your code but it wasn't working with latest Gradle (5.5) but I just change

jmhCompile 'org.openjdk.jmh:jmh-generator-annprocess:1.18'

to

jmhAnnotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.18'

and it worked perfectly.

[Johannes Zillmann]

Thanks, i've updated it!

[Cody Graham]

I'm currently using VSCode with the standard "Language Support for Java(TM) by Red Hat" extension installed.
AfFter following this tutorial, I can build and run benchmarks through Gradle without issue. However, the extension is unable to recognize any imports starting with "org.openjdk". This results in (a) incorrect error messages, and (b) no auto-completion for anything in openjdk. Does anyone know why this may happen?

[Yavuz Tas]

I think it's still the best option for using JMH with gradle. And still works in 2021 without any additional effort. Many thanks!