Measuring performance using BenchmarkDotNet - Part 2

Introduction

Previously we discussed what BenchmarkDotNet gives us and how to write simple benchmarks. As a quick reminder:

• We use benchmarks to find code performance
• BenchmarkDotNet is a nuget package
• We use console apps to host and run benchmarks

So what's next to do? We need to run it and get benchmarks as easily and frequently as we can.

---

Running Benchmarks locally

We have a sample .Net core console application coded up and ready to go in Github:

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Build and run

Once you've cloned the repo, just run a dotnet publish from the local repository's root folder:

dotnet publish -c Release -o publish

If you're unfamiliar with dotnet's CLI, dotnet publish will build and integrate the application, pushing the complete distributable application to the ./publish directory. You can read more here.

At this point, you've got a benchmarking console application in ./publish that's ready to use. Because I like my command line clean, I'm going to change the working folder:

cd publish

...and we're almost ready to start.

---

Before you run, prepare your machine

Whenever you're measuring CPU performance you've got to be mindful of what else is running on your machine. Even with a 64 core beast your OS may interrupt the benchmark execution and skew results. That skew is not easy to measure or counter: it's best to assume that the interrupts and switches always happen.

Whenever you run final benchmarks make sure the absolute minimum software and applications are running. Before you start, close down all other applications before running your benchmarks. Browsers, chat, video, everything!

For now, don't close down everything: we're just exploring BenchmarkDotNet here and you need a browser open to read. But, when capturing real results always remember to run on idle machines.

---

And now to get some benchmarks

To run them all we need to:

dotnet ./benchmarkdotnetdemo.dll -f *

-f * is a BenchmarkDotNet argument to selectively run benchmarks by their fully qualified namespace type. We've elected to select all of them with the wildcard *; if we want to run only selected benchmarks, I'd have to use -f benchmarkdotnetdemo.<pattern> as all these benchmarks fall in the benchmarkdotnetdemo namespace. For instance, -f benchmarkdotnetdemo.Simple* will run all the "Simple" benchmarks.

Each console application with BenchmarkDotNet has help automatically integrated. Just use --help as the arguments, and you will get a very comprehensive set of switches.

So now all we have to do is wait, and eventually your console will give you the good news:

// ***** BenchmarkRunner: End *****
// ** Remained 0 benchmark(s) to run **
Run time: 00:03:44 (224.56 sec), executed benchmarks: 3

Global total time: 00:08:03 (483.58 sec), executed benchmarks: 15
// * Artifacts cleanup *

All good! The result files will have been pushed to the BenchmarkDotNet.Artifacts folder:

    Directory: C:\...\benchmarking-performance-part-2\publish\BenchmarkDotNet.Artifacts


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          4/1/2021  11:50 AM                results
-a----          4/1/2021  11:50 AM         128042 BenchmarkRun-20210401-114253.log

The .log file is simply the benchmark console echoed to file.

Within the /results directory you'll find the actual reports:

    Directory: C:\...\benchmarking-performance-part-2\publish\BenchmarkDotNet.Artifacts\results


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----          4/1/2021  11:47 AM         109014 benchmarkdotnetdemo.FibonacciBenchmark-measurements.csv
-a----          4/1/2021  11:47 AM         103104 benchmarkdotnetdemo.FibonacciBenchmark-report-full.json
-a----          4/1/2021  11:47 AM           3930 benchmarkdotnetdemo.FibonacciBenchmark-report-github.md
-a----          4/1/2021  11:47 AM           6632 benchmarkdotnetdemo.FibonacciBenchmark-report.csv
-a----          4/1/2021  11:47 AM           4484 benchmarkdotnetdemo.FibonacciBenchmark-report.html
-a----          4/1/2021  11:50 AM          83537 benchmarkdotnetdemo.SimpleBenchmark-measurements.csv
-a----          4/1/2021  11:50 AM          53879 benchmarkdotnetdemo.SimpleBenchmark-report-full.json
-a----          4/1/2021  11:50 AM           1215 benchmarkdotnetdemo.SimpleBenchmark-report-github.md
-a----          4/1/2021  11:50 AM           2119 benchmarkdotnetdemo.SimpleBenchmark-report.csv
-a----          4/1/2021  11:50 AM           1881 benchmarkdotnetdemo.SimpleBenchmark-report.html

As you can see, it's a mix of CSV, HTML, markdown and pure JSON ready for publication and reading.

These formats are determined by either the benchmark code or the runtime arguments. I've included them all in the demo repo to give a feel of what's on offer.

---

Interpreting the results

We've previously discussed the various reports' contents. But suffice to say BenchmarkDotNet runs & reports benchmarks but does not evaluate them.

Evaluating these benchmarks and acting on them is a fairly complex problem: what analysis method to use? How do we run and capture results? Can we use benchmarks as a PR gateway? This will be the subject of a future post.

But before we run, we'd would like benchmarks running on all git pushes, right?

---

Running benchmarks in CI

Let's implement the simplest possible approach:

• build benchmarks
• run them
• capture the report files
• present for manual inspection

In short, benchmarks are built, run, and the results published as workflow artifacts. Anyone with access can download these artifacts.

Because our repo is in Github, and we want to show this in-the-flesh we'll be using Github Actions.

One day, Github Actions will support deep artifact linking and one-click reports, just like Jenkins and TeamCity have provided for years. But until that day dawns the tedium of download-extract-search is our lot :(

Here's a super-simple Github Actions workflow:

If you're unfamiliar with Action workflows, one of the best hands-on introductions is from Pete King.

This workflow file is in the sample Github repository, under ./.github/workflows/dotnet.yml.

Looking at the workflow, let's skip past the job's build steps as they're self explanatory.

---

Publish

- name: Publish
      run: dotnet publish -c Release --verbosity normal -o ./publish/

Here we prepare a fully publishable .Net core application.
We always need to build with Release configuration: BenchmarkDotNet will not adequately run without normal compiler optimisations. The application with its dependencies, including the code-under-test, is pushed to a ./publish/ directory within the job.

One glorious day, both Windows and linux will finally and completely converge on a single standard for directory path separators. Until that time, please be careful if you're writing these workflows on Windows!

---

Archive

- name: Archive 
      uses: actions/upload-artifact@v2
      with:
        name: benchmarkdotnetdemo
        path: ./publish/*

We're just arching the binaries here, in case we want to distribute and run locally.

---

Run Benchmarks

- name: Run Benchmarks    
      run: dotnet "./publish/benchmarkdotnetdemo.dll" -f "benchmarkdotnetdemo.*"

This is where we run the benchmarks.

As of now there are no Github Actions to support benchmark running, so all we do here is run the console application itself within the Github Actions job.

We're running all benchmarks in the benchmarkdotnetdemo namespace, and we expect the results to be pushed to the same working folder.

Note the double quotes! In windows you won't need to quote these arguments, but you will need to for Github Actions. If you don't, you'll see strange command line parsing errors.

Previously I remarked that you should only run benchmarks on an idle machine. Here we'll be running these on virtualised hardware, where OS interrupts are an absolutely unavoidable fact of life. Clearly we're trading precision for convenience here, and the code-under-test is simple enough not to worry too much about single-tick precision metrics.

---

Upload benchmark results

- name: Upload benchmark results
      uses: actions/upload-artifact@v2
      with:
        name: Benchmark_Results
        path: ./BenchmarkDotNet.Artifacts/results/*

This is where we present the results for inspection.

We just zip up the benchmark result files into a single artifact called Benchmark_Results.

---

And lastly...

That's it! Every time you push changes to this solution, benchmarks will be run. Performance degradations won't fail the build as we're not analysing the results, and we're certainly not applying quality gates in this solution. But you've got the minimum useful visibility, albeit very simply:

---

What have we learned?

Running benchmarks is very simple on face value, but there are considerations when doing so: you don't want to run while you're rendering videos!

Incorporating benchmark reporting into a CI pipeline is straight forward, although the lack of build reporting in Github Actions is a disappointment.

We've yet to act on those benchmarks' results. For instance, we don't yet fail the build if our code-under-test is underperforming.

---

Up next

How to fail the build if your code's underperforming.

---

Further Reading

• Github Actions
• Dotnet CLI
• Demo Github source & Actions

  NewDayTechnology / benchmarking-performance-part-2

  A simple demonstration of BenchmarkDotNet