You need to know about BenchmarkDotNet

BenchmarkDotNet is one of those free, open source tools that all .NET developers need to know how to use. This tool is useful if you want to:

• compare the performance of multiple ways of doing the same thing,
• decide which 3rd party library provides the fastest and most memory efficient implementation,
• compare the performance of code running on different versions of the .NET platform,
• monitor changes to your code base for changes that degrade performance.

What is a benchmark?

A benchmark is a program that measures the performance characteristics of some code (or hardware though this is not the focus of this article). Benchmarking is often used to compare the performance of different implementations but can also be used to show that you have achieved some performance goal (e.g. processing 1,000,000 operations per second).

Why use BenchmarkDotNet?

Don't use DateTime.Now

Hopefully you know not to use DateTime.Now for measuring the execution time of code. It isn't good for measuring sub-millisecond execution time and it can sometimes be affected by time synchronisation activities.

The pitfalls of using Stopwatch

You have probably used Stopwatch to measure execution times. However, taking a one shot value of one implementation and comparing it to another one shot value is unreliable. You will probably observe that if you do 5 runs of the same code, you will get 5 different results that vary wildly. So how can you compare multiple implementations that do the same thing? Using only the arithmetic mean might not be meaningful if there are outliers.

The JIT compiler can cause problems

Have you considered that the JIT compiler can dynamically re-compile code over time? It can do this to optimise hot paths of execution. The JIT compiler can also vary its optimisations based on the code it executed before the code you want to benchmark. How can you ensure the JIT compiler has done its optimisations and ensure isolation of the benchmarked code?

The overhead of running the benchmark can also cause problems

If you do realise that you need to do many iterations and perform some sort of statistical analysis, have you discounted the overhead in executing the benchmark? How many iterations do you need to do to get a stable result?

Enter BenchmarkDotNet

BenchmarkDotNet is a project supported by the .NET Foundation. It is used by many .NET projects (including those run by Microsoft) to analyse the performance of various libraries and detect regressions in performance as modifications are made.

It takes care of handling common pitfalls for you. It will:

• run your code multiple times and automatically detect when the results have stabilised,
• take into consideration the overhead used to run the benchmark and ensure the overhead does not interfere with the results,
• calculate all the statistics for you,
• produce nice graphs to help you visualise the results,
• warn you of potential issues when trying to reach conclusions based on the statistics (e.g. if it detects a bi-modal distribution, or if it detects and eliminates outliers).

Installing BenchmarkDotNet

Installation is simple ...

$ dotnet add package BenchmarkDotNet

C# Example

The main that runs the benchmark in this example is simple too ...

BenchmarkRunner.Run<MyBenchmark>();

Now let's look at the code to be benchmarked. Let's compare which is faster to look through a list - LINQ or a "for each" loop ...

[RankColumn]
[Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.FastestToSlowest)]
public class MyBenchmark
{
  [Params(10_000, 100_000, 1_000_000)]
  public int N { get; set; }

  private List<int> _list;

  [GlobalSetup]
  public void Setup() => _list = Enumerable.Range(0, N).ToList();        

  [Benchmark]
  public int LinqQuery() => _list.First(ii => ii == N - 1);

  [Benchmark(Baseline = true)]
  public int ForEachLoop()
  {
    foreach (var ii in _list)
    {
      if (ii == N - 1)
      {
        return ii;
      }
    }
    throw new Exception("Not found");
  }
}

The [RankColumn] and [Orderer] will rank and order the code according to what you configure in your Orderer.

The [Params] attribute will instruct the benchmarker to perform a benchmark run with (in this case) N set to that value. It will run the [GlobalSetup] once for each value of N before it runs a number of iterations of each of the benchmarks.

Using the Baseline attribute parameter will more clearly show how many times faster/slower the code is compared to a baseline.

The result summary from this example console application is shown below. Note that 1us is 1 microsecond. Also note that the Error is half of the 99.9% confidence interval (Note that I am not a statistician but I think of it as highly likely that the real mean is within the interval defined by the sampled Mean ± Error) ...

Method	N	Mean	Error	StdDev	Ratio	RatioSD	Rank
ForEachLoop	10000	11.80 us	0.229 us	0.225 us	1.00	0.00	1
LinqQuery	10000	86.12 us	1.568 us	1.466 us	7.30	0.13	2
ForEachLoop	100000	114.91 us	1.628 us	1.444 us	1.00	0.00	1
LinqQuery	100000	835.23 us	11.179 us	9.910 us	7.27	0.14	2
ForEachLoop	1000000	1,172.17 us	20.658 us	19.323 us	1.00	0.00	1
LinqQuery	1000000	8,446.51 us	166.337 us	163.366 us	7.21	0.16	2

So we can see that on my machine, LinqQuery is around 7 times slower (looking at the Ratio column) than the baseline ForEachLoop.

Other useful attributes

The attribute [RPlotExplorer] will create nice plots for you but you will need R and the rplotengine package installed on the machine running the benchmark.

The attribute [MemoryDiagnoser] will show the allocated memory so it can be useful for showing code that may be fast but uses more memory.

The attribute [SimpleJob(runtimeMoniker: RuntimeMoniker.Net70] will run each of the benchmarks with .NET 7. You can specify multiple platforms if you want to compare performance against other versions of .NET.