DEV Community: Tony Knight

Measuring performance using BenchmarkDotNet - Part 3 Breaking Builds

Tony Knight — Sat, 22 May 2021 00:25:13 +0000

Introduction

Previously we discussed the absolute bare minimum to run BenchmarkDotNet in your CI pipeline. Your code builds, benchmarks are taken, and you have to drill down into the numbers.

But what if bad code was committed? A small change sneaks in, probably under tight deadlines, that suddenly makes your nice fast code run like treacle?

How would you know about - and more importantly stop - such horrors? That's what we'll try and address in this post.

As before, we're only talking pure code here, that is, your class methods and algorithms. APIs, services, applications are much more complex, and we haven't considered I/O. So let's keep it simple and just focus on the performance of pure code.

What this post will cover

Discuss ways to stop builds upon degraded code performance
Installing tools in a sandbox environment
Collecting benchmark data for analysis
Analysing results and breaking builds

What you'll need for this post

.NET 7 SDK installed on your local machine
A BenchmarkDotNet solution
An IDE, such as VS, VS Code or Rider.

The simplest possible way to break a build...

...is, surprisingly, not a sledgehammer. It's even simpler than that.

For the vast majority of build platforms, to stop a build you normally need your script to return a non-zero return code. That age old trick is so simple and effective: it stops bad things dead in their tracks. So let's use that:

We want our benchmark analysis to return 0 on success and 1 on failure.

Easy! but this leaves us with a trickier problem.

How to detect performance has degraded?

You've got the stats from BenchmarkDotNet. You now need to monitor each build's performance results, or more accurately, detect deviance from accepted performance.

What is acceptable performance?

This is a very broad subject, and it's often difficult to put precise time limits on micro code performance. For much optimisation work, you'll be iteratively changing code so that performance should always improve with each commit. Therefore, you can stop optimising when the results are good enough.

So, as we often do not have absolute time requirements and we iteratively improve our performance as a matter of course, we'll take a broad view:

Accepted performance is the best recorded benchmark time

That leads us onto deviance.

Deviance from acceptable performance

Why do we want deviance and not absolutes? Because we cannot guarantee that repeated benchmark runs, even with a static codebase and the same infrastructure, will yield exactly the same time measurements over each iteration.

And as each build feeds many time-critical activities - user acceptance, security validation and the like - we don't want a tiny deviation to choke off this supply of new features.

How do we know what is an appropriate deviance is and how do we measure it? That's another very broad subject and depends entirely on your circumstances. For now, let's take a simple (& admittedly crude!) method just to illustrate the key point of stopping slow code getting into our main codebase.

Acceptable deviance, expressed as a percentage, falls in between [baseline measurement] < [new measurement] < [baseline measurement + deviance%]

Here we're simply allowing some slippage from the best recorded time.

Please remember: the subject is extremely broad and this article is just an introduction to the subject. But for now, the main take-away point is: whatever the current performance, keep improving it and never degrade!

It seems we need a tool for this

You could build your own, but here's something from our own stables: a dotnet tool to detect deviance in BenchmarkDotNet results:

NewDayTechnology / benchmarkdotnet.analyser

A tool for analysing BenchmarkDotNet results

BenchmarkDotNet.Analyser (BDNA) is a tool for iteratively collecting and analysing BenchmarkDotNet data. It's distributed as a dotnet tool, so you can use it locally and on almost any CI platform. You just need .NET 7 installed.

BDNA is in beta and we want to continually improve it. We welcome bug reports and feature suggestions.

Installing

The latest version is distributed via Nuget.

For the remainder of this section, I'll lead you through installing BDNA in a sandbox environment so if you do get into any problems you can simply destroy the directory and start again without any side effects.

Create a new sandbox

The sandbox will be a directory on your local drive.

We won't be pushing this directory to source control in this article. But the same steps are used in a cloned local repository.

mkdir c:\projects\scratch\tools
cd
cd projects\scratch\tools

Create a tools manifest

The tools manifest is simply a version list of the repo's tools, to ensure version consistency and stability: just like your own project's package dependencies. As we want these tools installed locally we'll create a new manifest in our directory:

dotnet new tool-manifest

Globally installed .NET tools are very convenient: you install it once on your machine and keep updating as necessary. But they place nasty dependencies on your build platform, and there's no guarantee your team members will use exactly the same version. Locally installed tools provide consistency, and are installed to the local repository.

Install BDNA

All that's left now is to download and install BDNA:

dotnet tool install bdna

This will install the latest non-preview version. If you want to install a specific version, just give the version, say:

dotnet tool install bdna --version 0.2.263

BDNA packages are listed here.

Check that BNDA is correctly installed:

dotnet tool list

and you will get a list of repo-local tools:

Package Id      Version              Commands      Manifest
-------------------------------------------------------------------------------------------------------
bdna            0.2.263              bdna          projects\scratch\tools\.config\dotnet-tools.json

Check that it's up and running:

dotnet bdna

and you should be greeted with a banner, like:

The installation is done!

You have successfully installed BDNA into your directory, and exactly the same steps will apply in a cloned git repository.

Checking benchmarks

What remains now is to get some benchmarks. If you've followed this series, you'll have some demonstration projects that generate benchmarks, such as

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Get some benchmarks

Clone the repo and start building:

dotnet clean
dotnet restore
dotnet build -c Release

cd src\benchmarkdotnetdemo\bin\Release\net7.0
dotnet Benchmarkdotnetdemo.dll -f *

The results will be found under **\BenchmarkDotNet.Artifacts\results.

Collect the data from your recent BenchmarkDotNet run

BDNA works by aggregating sequential benchmark runs. To aggregate (from the repo's root directory):

dotnet bdna aggregate -new ".\src\benchmarkdotnetdemo\bin\Release\net7.0\BenchmarkDotNet.Artifacts\results" -aggs ".\bdna" -out ".\bdna" -runs 30

To see all options try dotnet bdna aggregate -?

Repeatedly run benchmarks (dotnet Benchmarkdotnetdemo.dll -f *) and aggregate (dotnet bdna aggregate ...) to build a dataset.

When benchmarking you'll need points of reference for each datapoint. You can use --build %build_number% when aggregating each benchmark run to annotate with the build number. Tags are also supported.

Analyse the data

Now, we want to check the dataset for deviances. To see some errors we'll assume very strict deviance (0%) and allow no errors:

dotnet bdna analyse --aggregates ".\bdna" --tolerance 0 --maxerrors 0 --verbose

dotnet bdna analyse will send results to the console. If all is well you'll see a nice confirmatory message:

But if there are degraded benchmarks they'll be listed:

If too many errors are found the tool's return code will be 1: your CI script will need to watch for this return code and fail the build accordingly.

Reporting on the data

Console logs are often fine for CI pipelines. Wouldn't it be good to get some graphs of performance over time?

dotnet bdna report --aggregates ".\bdna"  --verbose -r csv -r json -f * -out ".\bdna_reports"

For help see dotnet bdna report --help

BDNA will build a CSV (and/or JSON) file containing selected benchmarks. Each benchmark is exported with namespace, class, method, parameters and annotations (build number, tags, etc).

Import the report file in your favourite BI tool and:

These measurements were taken from a machine with a lot of background procesing going on, and so you see peaks and troughs in the measurements. The general trend is flat. This is good, as the code didn't change between builds.

What have we learned?

We've discussed a very simple method of determining degraded performance where we compare results against a best-known result.

We've described how to set up local dotnet tools and nuget configurations.

We've introduced a tool that can collect, report & detect performance degradations, and how it can be used in a sandbox environment.

Measuring performance using BenchmarkDotNet - Part 2

Tony Knight — Thu, 01 Apr 2021 16:07:43 +0000

Introduction

Previously we discussed what BenchmarkDotNet gives us and how to write simple benchmarks. As a quick reminder:

We use benchmarks to find code performance
BenchmarkDotNet is a nuget package
We use console apps to host and run benchmarks

So what's next to do? We need to run it and get benchmarks as easily and frequently as we can.

Running Benchmarks locally

We have a sample .Net core console application coded up and ready to go in Github:

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Build and run

Once you've cloned the repo, just run a dotnet publish from the local repository's root folder:

dotnet publish -c Release -o publish

If you're unfamiliar with dotnet's CLI, dotnet publish will build and integrate the application, pushing the complete distributable application to the ./publish directory. You can read more here.

At this point, you've got a benchmarking console application in ./publish that's ready to use. Because I like my command line clean, I'm going to change the working folder:

cd publish

...and we're almost ready to start.

Before you run, prepare your machine

Whenever you're measuring CPU performance you've got to be mindful of what else is running on your machine. Even with a 64 core beast your OS may interrupt the benchmark execution and skew results. That skew is not easy to measure or counter: it's best to assume that the interrupts and switches always happen.

Whenever you run final benchmarks make sure the absolute minimum software and applications are running. Before you start, close down all other applications before running your benchmarks. Browsers, chat, video, everything!

For now, don't close down everything: we're just exploring BenchmarkDotNet here and you need a browser open to read. But, when capturing real results always remember to run on idle machines.

And now to get some benchmarks

To run them all we need to:

dotnet ./benchmarkdotnetdemo.dll -f *

-f * is a BenchmarkDotNet argument to selectively run benchmarks by their fully qualified namespace type. We've elected to select all of them with the wildcard *; if we want to run only selected benchmarks, I'd have to use -f benchmarkdotnetdemo.<pattern> as all these benchmarks fall in the benchmarkdotnetdemo namespace. For instance, -f benchmarkdotnetdemo.Simple* will run all the "Simple" benchmarks.

Each console application with BenchmarkDotNet has help automatically integrated. Just use --help as the arguments, and you will get a very comprehensive set of switches.

So now all we have to do is wait, and eventually your console will give you the good news:

// ***** BenchmarkRunner: End *****
// ** Remained 0 benchmark(s) to run **
Run time: 00:03:44 (224.56 sec), executed benchmarks: 3

Global total time: 00:08:03 (483.58 sec), executed benchmarks: 15
// * Artifacts cleanup *

All good! The result files will have been pushed to the BenchmarkDotNet.Artifacts folder:

    Directory: C:\...\benchmarking-performance-part-2\publish\BenchmarkDotNet.Artifacts


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          4/1/2021  11:50 AM                results
-a----          4/1/2021  11:50 AM         128042 BenchmarkRun-20210401-114253.log

The .log file is simply the benchmark console echoed to file.

Within the /results directory you'll find the actual reports:

    Directory: C:\...\benchmarking-performance-part-2\publish\BenchmarkDotNet.Artifacts\results


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----          4/1/2021  11:47 AM         109014 benchmarkdotnetdemo.FibonacciBenchmark-measurements.csv
-a----          4/1/2021  11:47 AM         103104 benchmarkdotnetdemo.FibonacciBenchmark-report-full.json
-a----          4/1/2021  11:47 AM           3930 benchmarkdotnetdemo.FibonacciBenchmark-report-github.md
-a----          4/1/2021  11:47 AM           6632 benchmarkdotnetdemo.FibonacciBenchmark-report.csv
-a----          4/1/2021  11:47 AM           4484 benchmarkdotnetdemo.FibonacciBenchmark-report.html
-a----          4/1/2021  11:50 AM          83537 benchmarkdotnetdemo.SimpleBenchmark-measurements.csv
-a----          4/1/2021  11:50 AM          53879 benchmarkdotnetdemo.SimpleBenchmark-report-full.json
-a----          4/1/2021  11:50 AM           1215 benchmarkdotnetdemo.SimpleBenchmark-report-github.md
-a----          4/1/2021  11:50 AM           2119 benchmarkdotnetdemo.SimpleBenchmark-report.csv
-a----          4/1/2021  11:50 AM           1881 benchmarkdotnetdemo.SimpleBenchmark-report.html

As you can see, it's a mix of CSV, HTML, markdown and pure JSON ready for publication and reading.

These formats are determined by either the benchmark code or the runtime arguments. I've included them all in the demo repo to give a feel of what's on offer.

Interpreting the results

We've previously discussed the various reports' contents. But suffice to say BenchmarkDotNet runs & reports benchmarks but does not evaluate them.

Evaluating these benchmarks and acting on them is a fairly complex problem: what analysis method to use? How do we run and capture results? Can we use benchmarks as a PR gateway? This will be the subject of a future post.

But before we run, we'd would like benchmarks running on all git pushes, right?

Running benchmarks in CI

Let's implement the simplest possible approach:

build benchmarks
run them
capture the report files
present for manual inspection

In short, benchmarks are built, run, and the results published as workflow artifacts. Anyone with access can download these artifacts.

Because our repo is in Github, and we want to show this in-the-flesh we'll be using Github Actions.

One day, Github Actions will support deep artifact linking and one-click reports, just like Jenkins and TeamCity have provided for years. But until that day dawns the tedium of download-extract-search is our lot :(

Here's a super-simple Github Actions workflow:

If you're unfamiliar with Action workflows, one of the best hands-on introductions is from Pete King.

This workflow file is in the sample Github repository, under ./.github/workflows/dotnet.yml.

Looking at the workflow, let's skip past the job's build steps as they're self explanatory.

`Publish`

- name: Publish
      run: dotnet publish -c Release --verbosity normal -o ./publish/

Here we prepare a fully publishable .Net core application.
We always need to build with Release configuration: BenchmarkDotNet will not adequately run without normal compiler optimisations. The application with its dependencies, including the code-under-test, is pushed to a ./publish/ directory within the job.

One glorious day, both Windows and linux will finally and completely converge on a single standard for directory path separators. Until that time, please be careful if you're writing these workflows on Windows!

`Archive`

- name: Archive 
      uses: actions/upload-artifact@v2
      with:
        name: benchmarkdotnetdemo
        path: ./publish/*

We're just arching the binaries here, in case we want to distribute and run locally.

`Run Benchmarks`

- name: Run Benchmarks    
      run: dotnet "./publish/benchmarkdotnetdemo.dll" -f "benchmarkdotnetdemo.*"

This is where we run the benchmarks.

As of now there are no Github Actions to support benchmark running, so all we do here is run the console application itself within the Github Actions job.

We're running all benchmarks in the benchmarkdotnetdemo namespace, and we expect the results to be pushed to the same working folder.

Note the double quotes! In windows you won't need to quote these arguments, but you will need to for Github Actions. If you don't, you'll see strange command line parsing errors.

Previously I remarked that you should only run benchmarks on an idle machine. Here we'll be running these on virtualised hardware, where OS interrupts are an absolutely unavoidable fact of life. Clearly we're trading precision for convenience here, and the code-under-test is simple enough not to worry too much about single-tick precision metrics.

`Upload benchmark results`

- name: Upload benchmark results
      uses: actions/upload-artifact@v2
      with:
        name: Benchmark_Results
        path: ./BenchmarkDotNet.Artifacts/results/*

This is where we present the results for inspection.

We just zip up the benchmark result files into a single artifact called Benchmark_Results.

And lastly...

That's it! Every time you push changes to this solution, benchmarks will be run. Performance degradations won't fail the build as we're not analysing the results, and we're certainly not applying quality gates in this solution. But you've got the minimum useful visibility, albeit very simply:

What have we learned?

Running benchmarks is very simple on face value, but there are considerations when doing so: you don't want to run while you're rendering videos!

Incorporating benchmark reporting into a CI pipeline is straight forward, although the lack of build reporting in Github Actions is a disappointment.

We've yet to act on those benchmarks' results. For instance, we don't yet fail the build if our code-under-test is underperforming.

Up next

How to fail the build if your code's underperforming.

Measuring performance using BenchmarkDotNet - Part 1

Tony Knight — Mon, 15 Mar 2021 18:15:31 +0000

Introduction

We all must build fast software, right? Right? It’s true that microservices tend to introduce latencies - stateless functions mean a whole lot more network calls, and you can wave goodbye to data locality. But a microservice is still dependent on its own code being fast, and at least fast enough.

In the past we’ve relied on profilers, stopwatches, dedicated performance teams, and sometimes plain old complaints from the field. All of these methods require some form of measurement; unfortunately they tend to be “big picture” performance that lacks detail - and often without concrete scenarios. This gets very expensive very quickly.

Very often, you just want to measure the code’s performance without the baggage of dependencies. You might have a critical piece of code that absolutely must meet certain performance criteria. Measuring such microcode cam obviously be done with profilers - dotTrace, ANTS to name just two. The problem is they bring their own baggage as well, and worse can’t be easily relied upon in a CI pipeline. So how can you measure microcode performance in CI? Unit tests are a terrible idea, what else is there? Step forward BenchmarkDotNet.

TL;DR

Measure your code’s performance with benchmarks at near zero cost and. All you need are:

.NET 7 SDK
VS/VSCode
BenchmarkDotNet from Nuget

We’ll talk about how to write simple benchmarks, how to run them and how to interpret the results.

What is BenchmarkDotNet?

BenchmarkDotNet does what it says on the tin: benchmark .net code. It’s available as a Nuget packaged library for inclusion into your .net console applications. It is very widely used by all major players in the .net world, including the dotnet core runtime project itself.

What does a HelloWorld benchmark look like?

Let’s say you have a very basic Fibonacci implementation - and you want to measure its resource usage growth as more numbers are generated.

By “resource usage” I mean time and memory consumed per method call.

In other words, you'd want to know how it scales. Here's an implementation of "get the first N Fibonacci numbers":



public static IEnumerable<int> GetFibonacci(this int count)
{
    var w = 1;
    var x = 1;

    yield return x;
    foreach (var _ in Enumerable.Range(1, count - 1))
    {
        var y = w + x;
        yield return y;
        w = x;
        x = y;
    }
}

No prizes are sought for best efficiency here. Please do not take this as a reference implementation of Fibonacci!

To answer the scaling question, we would implement a benchmark, run it and analyse the results. Skipping forward a rendered benchmark report would look something like the below:

What do all the headers actually mean?

The column	What it means
Method	The name of the code-under-test; a single benchmark may have several methods under test for, e.g. scenarios. This value is lifted directly from your benchmark code.
Count	An arbitrary parameter: in this case the number of Fibonacci numbers generated by the method under test.
Mean/Error/StdDev/StdError	Execution time statistics. Note that these can be given down to nanoseconds, depending on how fast your code is. Low is best.
Min/Q1/Median/Q3/Max	Quartile execution time statistics: note the time units. Low is best.
Ops/sec	The number of operations executed per second for the method/parameter combination. High is good.
Rank	The fastest performing method/parameter combination. Low is best.
Gen 0/1/2	The total number of collections per generation
Allocated	Total bytes allocated against all generations

Note the header information in the report! It’ll give details on the OS, CPU, .Net version, JIT method and GC configuration. Always benchmark like-for-like!

OK… what do those numbers really mean?

Let’s look at each iteration of Count, and we’re using it here to get the first Count numbers of the Fibonacci sequence.

Where Count is 1 the mean execution time is 103.4 nanoseconds. That’s 0.1 microseconds, or 0.0001 milliseconds. I like that: nice and fast.

Where Count is 13 (yes, the parameters themselves follow Fibonacci!) the mean time is 407.2 ns: four times what Count=1 is, yet the Count is 13 times bigger. I’ll take that, for now.

Where Count is 34 the mean time is 1,077.9 ns, or 1.077 microseconds, or just over 0.0001 milliseconds. That’s taking 2.6 times more time than Count = 13. Let’s compare against Count = 1: Count is 34 times bigger , yet takes 10 times the time. I’ll take that too.

If we plot Count against the time ratio we see this:

In other words, time used does not grow as Count grows. If it did, the lines would be parallel.

So the benchmarks are showing that the implementation has reasonably acceptable scaling. It's not constant time, but it’s better than O(n) time: a pleasant surprise.

If you're not satisfied with the performance results, simply make your changes, re-run the benchmarks & re-analyse. That's it.

You haven’t mentioned the memory yet, have you?

Trust me, I’m getting to that.

Pay particular attention to memory usage. Garbage collections and memory allocations are as important as sheer speed!

Count=1 used 128 bytes.
Count=13 used 312 bytes
Count=34 used 744 bytes.

If we plot Count against the allocation growth ratios, we see this:

This means the used memory isn’t constant either: the memory used for Count=34 is greater than the memory used for Count=1. Again it's better than O(n). To my mind this is OK, but not great: we need more investigation. It's probably incurred with yield return, but do we want to sacrifice the readability? Probably not, but in any case we’re getting new perspectives on our code. This is a good thing.

What other rendered reports can you get?

You can output a markdown version of your report and many other formats; Markown output is GitHub inspired.

You can use the following attributes to output the many different types of rendered reports:



[JsonExporterAttribute.Full()]
[CsvMeasurementsExporter()]
[CsvExporter(CsvSeparator.Comma)]
[HtmlExporter()]
[MarkdownExporterAttribute.GitHub()]

An example of the GitHub Markdown report:

Charting is supported through the R project. As R is a world in itself, I’m going to skip the subject.

If you want charts, consider importing the rendered JSON into Excel. The CsvExporter attribute will generate a CSV with the data you need.

Full code example

What does the benchmark code look like using BenchmarkDotNet? It might surprise you to see how simple it is.

BenchmarkDotNet relies on declarative code over which it will reflect. Leaving aside the class attributes (more on those later), note the [Params] attribute over Count from the report above, likewise [Benchmark] and Fibonacci().



using System.Linq;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class FibonacciBenchmark
    {
        [Params(1, 2, 3, 5, 8, 13, 21, 34)]
        public int Count { get; set; }

        [Benchmark]
        public void Fibonacci()
        {
            var xs = Count.GetFibonacci().ToList();
        }
    }
}

You’ll notice that the benchmarks have a return type of void and do not have any assertions. Remember: we’re not proving functional correctness here, we’re measuring resource usage.

Show me the code!

I’ve created a simple BenchmarkDotNet implementation here:

NewDayTechnology / benchmarking-performance-part-1

A simple demonstration of BenchmarkDotNet

There’s only the one C# project in there - benchmarkdotnetdemo.csproj - that contains the minimal files.

BenchmarkDotNet will only work if the console project is built with a Release configuration, that is with code optimisations applied. Running in Debug will result in a run-time error.

Setup

This is the Program.cs file, and like all C# console apps you need an entry point:



using System;
using BenchmarkDotNet.Running;

namespace benchmarkdotnetdemo
{
    class Program
    {
        static int Main(string[] args)
        {
            try
            {
                BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
                    .Run(args);

                return 0;
            }
            catch (Exception ex)
            {
                Console.ForegroundColor = ConsoleColor.Red;
                Console.WriteLine(ex.Message);
                Console.ResetColor();
                return 1;
            }
        }
    }
}

Aside from the standard method entry point let’s go over it bit by bit.



using BenchmarkDotNet.Running;

For bootstrapping BenchmarkDotNet, this is the only import you need.



BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

This one-line-to-rule-them-all will perform all command line parsing, all help, all benchmark execution and all report generation.

One point here is .FromAssembly(typeof(Program).Assembly) - this informs BenchmarkDotNet of its benchmark search scope. Benchmarks are internally discovered by reflection - you’ll see soon enough.

NOTE: If you were to run the project without any command line arguments, BenchmarkDotNet will assume an interactive CLI.

.Run(args) returns a sequence of report objects comprised of the same data used for rendered reports: I’ve excluded them for simplicity. If you want to run benchmarks and fail CI builds if performance dips they are your first place to look.

Create a new benchmark

There is a file called SimpleBenchmark.cs. Let’s have a look.



using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class SimpleBenchmark
    {
        [Benchmark]
        public void NoopTest() { }

        [Benchmark]
        public int AddTest() => int.MaxValue + int.MinValue;

        [Benchmark]
        public int MultiplyTest() => 11 * 3;
    }
}

FibonacciBenchmark.cs

Just for completeness: note the similar declarations as SimpleBenchmarks.cs. In this case, we’re adding a [Params] parameter to support benchmark permutations.



using System.Linq;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class FibonacciBenchmark
    {
        [Params(1, 2, 3, 5, 8, 13, 21, 34)]
        public int Count { get; set; }

        [Benchmark]
        public void Fibonacci()
        {
            var xs = Count.GetFibonacci().ToList();
        }
    }
}

How are benchmarks executed?

Without going into too much detail, BenchmarkDotNet will attempt to run your benchmarks many many times over to settle on mean and median values.

When you run the benchmarks you may first be confused by just how many iterations are involved, so let’s give a simplistic explanation. Modern OSs are preemptive multitaskers, CPUs have pipeline caches as well as instruction reordering features. .NET itself has the JIT compiler. This means that no single execution of code can be relied upon to give a canonical result.

This is part of the reason why unit tests are terrible for benchmarking! They only run once and incur their own (unaccounted) overheads.

BenchmarkDotNet will run warm up iterations before it can take representative values. These show up as various stages: OverheadJitting & WorkloadJitting, WorkloadPilot, OverheadWarmup, OverheadActual.

JIT comes at a cost: the first time any .NET code executes it must first be JIT compiled. The more complex the code the higher the JIT cost, usually showing as CPU and time costs. As we’re interested only in runtime performance, these steps eliminate JIT costs from measurements.

In the same vein other warmup steps are run to eliminate other “once only” costs, for instance to warm up pipelining caches.

After these steps have completed, BenchmarkDotNet will iterate these operations to yield the final statistics; these are shown as WorkloadActual steps.

If you want more detail, please refer to BenchmarkDotNet’s own documentation. In these code samples we’re using the default Throughput strategy for microbenchmarking.

How long does it take?

It depends ;) Simple calculations, such as in the demo project, will run in under a minute. Adding permutations (such as with [Params]) will linearly increase the benchmarking time, as each parameter will be benchmarked in its own right.

With that in mind, it’s quite clear that resource hungry algorithms, benchmarked with a large variety of parameters, will take a considerable amount of time.

Don’t expect to parallelise BenchmarkDotNet: it runs benchmarks sequentially. Thread context switching is itself a cost and extremely difficult to compensate for.

What have we learned?

We’ve seen how to get BenchmarkDotNet
We’ve seen how to integrate it in a simple console application
We’ve seen the minimum work needed to build benchmarks
We’ve had a taste of the reports and inferences we can gain from BenchmarkDotNet

Next Steps

How to incorporate into CI?

More Information

What is benchmarking - Wiki
BenchmarkDotNet
BenchmarkDotNet on Github
A real world use case of BenchmarkDotNet
GitHub repository:
NewDayTechnology / benchmarking-performance-part-1

A simple demonstration of BenchmarkDotNet
Measuring performance with BenchmarkDotNet part 1

A simple demonstration of the superlative BenchmarkDotNet and its integration into Github Actions.

Measuring code performance is self evidently a vital discipline to software engineering and yet is so often skipped, usually for false economies. BenchmarkDotNet makes this essential task simplicity itself, with a syntax and style that's immediately intuitive to anyone versed in unit testing.

Just exercise your code in a declarative way, include it in your CI pipeline, and enjoy the results.

This project just demonstrates the basics: the .net project, the CI pipeline and the resultant reports.

The Benchmarks
- NoopTest The absolute minimum function that can be benchmarked - it does nothing.
- AddTest A simple addition metric, again of minimal complexity.
- MultiplyTest A simple multiplication metric, again of minimal complexity.
- Fibonacci Benchmarking a Fibonacci implementation, measuring the computation time for the first N Fibonacci numbers.
Builds

Builds are managed with love…
View on GitHub

DEV Community: Tony Knight

Measuring performance using BenchmarkDotNet - Part 3 Breaking Builds

Introduction

What this post will cover

What you'll need for this post

The simplest possible way to break a build...

How to detect performance has degraded?

What is acceptable performance?

Deviance from acceptable performance

It seems we need a tool for this

NewDayTechnology / benchmarkdotnet.analyser

A tool for analysing BenchmarkDotNet results

Installing

Create a new sandbox

Create a tools manifest

Install BDNA

The installation is done!

Checking benchmarks

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Get some benchmarks

Collect the data from your recent BenchmarkDotNet run

Analyse the data

Reporting on the data

What have we learned?

More reading

NewDayTechnology / benchmarkdotnet.analyser

A tool for analysing BenchmarkDotNet results

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Measuring performance using BenchmarkDotNet - Part 2

Introduction

Running Benchmarks locally

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Build and run

Before you run, prepare your machine

And now to get some benchmarks

Interpreting the results

Running benchmarks in CI

Publish

Archive

Run Benchmarks

Upload benchmark results

And lastly...

What have we learned?

Up next

Further Reading

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Measuring performance using BenchmarkDotNet - Part 1

Introduction

TL;DR

What is BenchmarkDotNet?

What does a HelloWorld benchmark look like?

What do all the headers actually mean?

OK… what do those numbers really mean?

You haven’t mentioned the memory yet, have you?

What other rendered reports can you get?

Full code example

Show me the code!

NewDayTechnology / benchmarking-performance-part-1

A simple demonstration of BenchmarkDotNet

Setup

Create a new benchmark

FibonacciBenchmark.cs

How are benchmarks executed?

How long does it take?

What have we learned?

Next Steps

More Information

NewDayTechnology / benchmarking-performance-part-1

A simple demonstration of BenchmarkDotNet

Measuring performance with BenchmarkDotNet part 1

The Benchmarks

Builds

`Publish`

`Archive`

`Run Benchmarks`

`Upload benchmark results`