BenchmarkGate, Part 2: Why a Baseline Isn't Enough

In part one, I introduced
BenchmarkGate — a local-first
performance regression gate for BenchmarkDotNet. v0.1.0-alpha.1 could do
one thing: compare a benchmark's mean time against a committed baseline,
using a single global threshold passed on the command line.

That was enough to catch a regression. It wasn't enough to trust the tool.

v0.2.0-alpha.1 is live now, and this is the part of the story where the
tool stopped being "diff two numbers" and started being a gate you can
actually build a CI contract on.

The run that made the case for itself

Here's real output from check, run against CedarRecon's benchmark suite:

# Benchmark Gate — CedarRecon

**Overall: ❓ Unstable**

| Total | Improved | Passed | Warning | Regressed | Missing | New | Unstable |
|---|---|---|---|---|---|---|---|
| 27    | 0        | 6      | 0       | 0         | 0       | 0   | 21       |

21 of 27 benchmarks came back Unstable. Not passed, not failed —
unstable. Every one of them has the same explanation:

[Unstable] ClassifierPhaseBenchmark.BuildDictionaries|job=Job-ZDAPPP|N=10000:
Only 3 measurements were taken, below the configured minimum of 5.

Those benchmarks ran with only 3 iterations — not enough for the tool to
trust what it was looking at. A naive diff-the-means tool would have
happily compared 3-sample noise against a baseline and reported a
confident Passed or Regressed. BenchmarkGate refused. That refusal
is the feature.

The other 6 benchmarks, from a different BenchmarkDotNet job with a real
iteration count, evaluated cleanly:

| Benchmark                                                          | Metric          | Baseline    | Current     | Delta  | Status |
|---|---|---|---|---|---|
| ExceptionClassifierBenchmark.DictionaryClassifier|job=Job-SNYTAA|N=10000 | meanNanoseconds | 4.946 ms | 4.946 ms | +0.00% | Passed |
| ExceptionClassifierBenchmark.IndexedClassifier|job=Job-SNYTAA|N=1000000 | meanNanoseconds | 472.444 ms | 472.444 ms | +0.00% | Passed |

Same suite, same check invocation, two entirely different verdicts —
because the tool actually looked at how each result was produced before
deciding whether it meant anything.

What changed since v0.1

v0.1 had exactly one lever: --threshold-percent, applied to mean time,
globally, with no way to say "trust this benchmark less" or "allocation
matters here too." v0.2 replaces that with three things.

1. A policy file, not a flag

benchmark-gate check \
  --results ./BenchmarkDotNet.Artifacts/results \
  --baseline ./benchmarks/baseline.json \
  --policy ./benchmarks/policy.json

policy.json:

{
  "schemaVersion": 1,
  "stability": {
    "minimumMeasurements": 5,
    "maximumCoefficientOfVariation": 0.10
  },
  "metrics": {
    "meanNanoseconds": {
      "direction": "lower-is-better",
      "warningPercent": 5,
      "failurePercent": 10,
      "minimumAbsoluteChange": 0
    }
  }
}

This is a committed, reviewable file — a change to your performance budget
shows up in a pull-request diff, same as any other code change. That was
true in spirit in v0.1; in v0.2 it's actually true, because there's a real
document to diff instead of a CLI flag buried in a CI YAML file.

2. A stability gate that runs before anything else

Every benchmark is checked against stability first — measurement count,
coefficient of variation — before a single metric comparison happens. Fail
that check, and the benchmark is Unstable, full stop. No metric verdict
gets attached to a sample the tool doesn't trust.

This is what produced the 21 Unstable results above. It's also, I'd
argue, the most important thing v0.2 added — not because "unstable" is an
exciting status, but because not having it means every noisy CI runner
eventually produces a false regression, and false regressions are how
people learn to ignore the gate entirely.

3. Warning, not just pass/fail

A metric crossing warningPercent but not failurePercent is Warning —
visible in every report, but it only affects the process exit code if you
explicitly ask for that with --fail-on-warning:

benchmark-gate check \
  --results ./BenchmarkDotNet.Artifacts/results \
  --baseline ./benchmarks/baseline.json \
  --policy ./benchmarks/policy.json \
  --fail-on-warning

Without the flag: a suite with only warnings exits 0, but the Markdown,
JSON, and console reports all still show the underlying Warning status —
they never silently upgrade it to Regressed just because the exit code
happened to be zero. With the flag: the same suite exits non-zero, and
every report format — including a new JUnit XML output for CI test-result
UIs — agrees.

benchmark-gate check \
  --results ./BenchmarkDotNet.Artifacts/results \
  --baseline ./benchmarks/baseline.json \
  --policy ./benchmarks/policy.json \
  --markdown ./report.md \
  --json ./decision.json \
  --junit ./junit.xml

That consistency — CLI exit code, JSON's exitCode field, and JUnit's
<failure> elements all telling the same story — mattered enough that I
wrote tests specifically to pin it down, not just eyeballed it once and
moved on.

Multi-metric, not just mean time

policy.json isn't hardcoded to mean time. Allocation tracking is a
second first-class metric:

"metrics": {
  "meanNanoseconds": {
    "direction": "lower-is-better",
    "warningPercent": 5,
    "failurePercent": 10
  },
  "allocatedBytesPerOperation": {
    "direction": "lower-is-better",
    "warningPercent": 1,
    "failurePercent": 5,
    "minimumAbsoluteChange": 1024
  }
}

If your benchmarks run with [MemoryDiagnoser], both metrics get
evaluated independently, and a benchmark's overall status is whichever
metric did worst — a mean-time pass doesn't hide an allocation
regression. Under the hood this isn't two hardcoded fields; it's an
extensible metric dictionary, so a future metric (throughput, GC
collections, whatever) is a policy entry, not a schema migration.

What v0.2 cost: two breaking changes

Being pre-1.0 meant I could make the calls that actually made the tool
better instead of carrying compatibility baggage for an audience that
doesn't exist yet:

• Baseline schema bumped v1 → v2. The old single meanNanoseconds field couldn't hold a metrics dictionary. Old baselines are rejected outright with a message pointing at capture — no migration shim, no silent reinterpretation.
• --threshold-percent/--minimum-absolute-change-ns are gone. --policy replaces them entirely.

Both are documented, both are deliberate, and both happened before anyone
but me depended on this tool. That's the whole point of shipping early
alphas.

What's next

Two commands are next, in order, per the actual roadmap:

• v0.3.0 — benchmark-gate validate. Validates a policy, baseline, or BenchmarkDotNet input file on its own — catches malformed data (an inverted warning/failure threshold, a missing required field, an unsupported schema version) before it ever reaches check.
• v0.4.0 — benchmark-gate compare. Pulls the delta calculation out of the evaluator entirely: compare describes what changed between a baseline and a current run, independent of any policy; check then applies policy to that same comparison instead of computing it inline. That split is what makes everything after it — history, and eventually explaining why something regressed — possible without re-plumbing the evaluator again. Further out, once compare exists: history-aware baselines (v0.5.0), which is also where environment compatibility checking lives — filtering out comparisons across incompatible machines/runtimes before they can produce a misleading result. That's real and coming, just not the immediate next step.

Separately, and not version-gated to any of the above: net8.0
multi-targeting, so the tool installs on machines with only the .NET 8
LTS runtime, not just .NET 10.

Full sequencing, including what's deliberately parked past v1.0.0
(diagnostic explanation, EventPipe integration), is in the repo's
ROADMAP.md.

If you want to try it:

dotnet tool install --global Bijecta.BenchmarkGate.Tool --version 0.2.0-alpha.1

Repo, README, and the full roadmap are at
github.com/Bijecta/BenchmarkGate.