DEV Community: Youness Aamiri

BenchmarkGate, Part 4: benchmark-gate compare, and Why Order Matters

Youness Aamiri — Mon, 10 Aug 2026 11:35:10 +0000

In part three,
v0.3.0-alpha.1 added validate — catching a broken policy or baseline
file before check ever saw it. That release didn't touch what check
actually does, though: match benchmarks, match metrics, calculate
deltas, apply thresholds, decide pass or fail — all inside one evaluator,
one pass, no way to get "what changed" without also asking "is that
acceptable."

v0.4.0-alpha.1 splits that in two. benchmark-gate compare answers the
first question. check still answers the second — but it answers it by
applying policy on top of the same comparison compare produces,
instead of computing its own.

A real diff

Here's compare against a real baseline from this project's own
benchmark suite:

benchmark-gate compare --baseline ..\baseline.json --results .\BenchmarkDotNet.Artifacts\results\CedarRecon.Tests.Performance.ClassifierPhaseBenchmark-report-full-compressed.json

Suite: CedarRecon  Comparable: 21  Added: 0  Removed: 6

Benchmark                                Metric               Reference    Candidate    Abs Delta      % Delta    Direction      Status
ClassifierPhaseBenchmark.BuildDictionar. allocatedBytesPerOp. 5.465 MB     5.465 MB     0 B            +0.00%     Unchanged      Comparable
                                         meanNanoseconds      4.998 ms     4.998 ms     0.000 ns       +0.00%     Unchanged      Comparable
[... 19 more Comparable benchmarks, all Unchanged ...]

Removed benchmarks:
  ExceptionClassifierBenchmark.DictionaryClassifier|job=Job-SNYTAA|N=10000
  ExceptionClassifierBenchmark.DictionaryClassifier|job=Job-SNYTAA|N=100000
  ExceptionClassifierBenchmark.DictionaryClassifier|job=Job-SNYTAA|N=1000000
  ExceptionClassifierBenchmark.IndexedClassifier|job=Job-SNYTAA|N=10000
  ExceptionClassifierBenchmark.IndexedClassifier|job=Job-SNYTAA|N=100000
  ExceptionClassifierBenchmark.IndexedClassifier|job=Job-SNYTAA|N=1000000

Nothing in the 21 Comparable benchmarks moved. But an entire benchmark
class — ExceptionClassifierBenchmark, six parameterized entries — is
gone from the current run. No policy was involved in producing that
output. compare doesn't know whether a benchmark disappearing is a
deliberate rename, a benchmark someone deleted on purpose, or a sign the
project no longer builds that code path. It just reports the fact: this
existed in the baseline, it doesn't exist now. Whether that's fine is a
judgment call for a human, or for check with a policy attached — not
for compare.

The same fact, as data

--format json writes the identical comparison as a versioned,
machine-readable document — same command, --format json --output compare.json instead of printing a table. This isn't just another report
format: it's the same comparison facts, serialized as a versioned
artifact that another tool — or a future BenchmarkGate command — can
consume without rerunning the comparison. The JSON preserves the
comparison engine's canonical ordering exactly; reporters don't re-sort
the results.

{
  "schemaVersion": 1,
  "suite": "CedarRecon",
  "comparable": 21,
  "added": 0,
  "removed": 6,
  "benchmarks": [
    {
      "identity": "ExceptionClassifierBenchmark.DictionaryClassifier|job=Job-SNYTAA|N=10000",
      "status": "Removed",
      "metrics": [
        {
          "metricName": "allocatedBytesPerOperation",
          "status": "MissingCandidateMetric",
          "reference": { "value": 5087065, "unit": "bytes" }
        },
        {
          "metricName": "meanNanoseconds",
          "status": "MissingCandidateMetric",
          "reference": { "value": 4945967.65625, "unit": "ns" }
        }
      ]
    }
  ]
}

Removed means "present in the reference baseline but absent from the
candidate run." It does not imply an error.

Two more things worth noticing here. First, there's no candidateStability
field on this benchmark at all — not null, genuinely absent. A removed
benchmark has no current-run observation, so there's nothing to source
stability facts from; the field is omitted rather than filled with a
placeholder. Second, each metric's reference value is still the exact
number from the baseline, full precision, un-rounded — compare's JSON
output never rounds a value for display the way the console table does.

Why order matters

The title isn't about the order benchmarks appear in the output. It's
about the order two things happen in.

Before this release, the pipeline was:

baseline + current + policy → evaluation

One evaluator performed benchmark matching, metric matching, delta
calculation, and threshold evaluation in a single pass. There was no way
to ask "what changed" without also asking "is that acceptable" — the two
questions were the same function call.

Now it's:

baseline + current → comparison → (+ policy) → evaluation

BenchmarkComparisonEngine is the only place benchmark matching, metric
matching, and delta calculation happen. It doesn't know what a policy is.
compare calls it directly and reports the result. check calls the
exact same engine, then hands its output to RegressionEvaluator,
which applies policy on top — never recomputing a delta, never
re-matching a benchmark, just interpreting facts that already exist. In
other words: arithmetic belongs to comparison; judgment belongs to
evaluation.

That ordering is the whole point of this release. history (v0.5.0)
and anything after it can depend on "what changed" being a real, callable
thing on its own — not something baked one level deep inside an
evaluator that also happens to know about warning thresholds.

What v0.4.0 deliberately didn't do

MetricComparisonStatus has a UnitMismatch value. compare never
produces it.

That's not a bug — it's a gap in the data, not the logic. Neither a
baseline entry nor a current observation carries a per-value unit
today; a metric's unit comes entirely from MetricCatalog, keyed by
metric name. Both sides currently derive their unit from the same
catalog entry, so a mismatch can never be observed. The status stays in
the schema, documented as reserved, because it's part of the intended
shape; it just needs baseline and candidate values to carry real
source-unit metadata before it can mean anything. That's separate,
future work, not something to fake with a check that can never fire.

The bigger picture

The important part isn't the new command. It's that "what changed" is
now a first-class concept. Once comparison exists independently of
policy, other features — history, trend analysis, dashboards, and
explanation — can all build on the same artifact instead of
reimplementing comparison logic.

What's next

v0.5.0 — filesystem history. check --history <path> as an alternative
to --baseline: a plain directory of snapshots instead of one file,
aggregated (median of the last N, by default) into a baseline on the fly.
That only becomes possible because comparison is now a standalone
artifact instead of an internal evaluator step — history needs something
stable to compare against, and now it has one. Full sequencing is in
ROADMAP.md.

dotnet tool install --global Bijecta.BenchmarkGate.Tool --version 0.4.0-alpha.1

Repo, README, and the full roadmap:
github.com/Bijecta/BenchmarkGate.

BenchmarkGate, Part 3: benchmark-gate validate

Youness Aamiri — Mon, 03 Aug 2026 09:27:08 +0000

In part two,
v0.2.0-alpha.1 added a stability gate and a policy file. Both are good
things. Both are also new ways to write a file that's subtly wrong — a
threshold typo, an inverted warning/failure pair, a baseline entry missing
its identity — and until now, the only way to find that out was to run
check and read whatever exception came back first.

v0.3.0-alpha.1 adds benchmark-gate validate. It doesn't evaluate
anything. It just tells you, exhaustively, what's wrong with a file —
before that file ever gets near a real check run.

The problem with fail-fast

Here's policy.json with two unrelated mistakes in it:

{
  "schemaVersion": 1,
  "stability": { "minimumMeasurements": 0, "maximumCoefficientOfVariation": 0.1 },
  "metrics": {
    "meanNanoseconds": { "direction": "lower-is-better", "warningPercent": 10, "failurePercent": 10 }
  }
}

minimumMeasurements: 0 is invalid. warningPercent equal to
failurePercent is also invalid. Before this release, PolicyFile.Load
threw on the first problem it found — you'd fix minimumMeasurements,
re-run, and then find out about the threshold. Two mistakes, two
round-trips.

validate finds both in one pass:

$ benchmark-gate validate --policy bad-policy.json
bad-policy.json
  ERROR BGV105 /stability/minimumMeasurements: Value must be greater than zero; actual value was 0.
  ERROR BGV117 /metrics/meanNanoseconds: Metric 'meanNanoseconds' has warningPercent (10) >= failurePercent (10). warningPercent must be strictly less than failurePercent for the policy to be meaningful.

$ echo $LASTEXITCODE
12

Both errors, one invocation, no guessing what else might be wrong once
you fix the first thing.

One command, three file types, together

benchmark-gate validate --policy policy.json
benchmark-gate validate --baseline baseline.json
benchmark-gate validate --results ./BenchmarkDotNet.Artifacts/results

All three flags are optional individually, but at least one is required,
and you can combine them — validate your whole input surface in one call
before wiring anything into CI:

$ benchmark-gate validate --policy bad-policy.json --baseline bad-baseline.json
bad-policy.json
  ERROR BGV105 /stability/minimumMeasurements: Value must be greater than zero; actual value was 0.
  ERROR BGV117 /metrics/meanNanoseconds: Metric 'meanNanoseconds' has warningPercent (10) >= failurePercent (10). warningPercent must be strictly less than failurePercent for the policy to be meaningful.

bad-baseline.json
  ERROR BGV203 /benchmarks/1: Duplicate benchmark identity 'MyApp.Bench.Sort|job=Default'.

Two independently broken files, two different validators (PolicyValidator
and SnapshotValidator), one invocation — each grouped under its own
source heading, with a blank line separating artifacts so a multi-file
run stays scannable instead of turning into a wall of undifferentiated
error text.

Console output groups by source file. Errors print in red when you're
looking at a real terminal; piped or redirected output stays plain, so
scripts and CI logs get deterministic text.

Structured output for machines

--json writes a versioned report covering every artifact you validated
in one call, instead of just printing to the console. Here's a baseline
with a duplicate benchmark entry:

benchmark-gate validate --baseline bad-baseline.json --json validation-report.json

{
  "schemaVersion": 1,
  "isValid": false,
  "errorCount": 1,
  "warningCount": 0,
  "artifacts": [
    {
      "kind": "Baseline",
      "source": "bad-baseline.json",
      "isValid": false,
      "errorCount": 1,
      "warningCount": 0,
      "diagnostics": [
        {
          "code": "BGV203",
          "severity": "Error",
          "title": "Duplicate benchmark identity",
          "path": "/benchmarks/1",
          "message": "Duplicate benchmark identity 'MyApp.Bench.Sort|job=Default'."
        }
      ]
    }
  ]
}

Top-level isValid/errorCount/warningCount let a CI step decide
pass/fail without walking every artifact; each artifact still carries its
own counts and full diagnostic list for anything that wants the detail —
a future benchmark-gate explain BGV203 command, an IDE annotation, a
Markdown summary, whatever ends up consuming this.

Where the checks actually live

This is the part I think is more interesting than the command itself:
validate doesn't contain a single validation rule. Every check it runs
already existed — I extracted it from check's own load path, rather than
writing a second implementation next to the first.

PolicyFile.Load used to do this:

if (warningPercent >= failurePercent)
    throw new PolicyFileException(path, "warningPercent must be strictly less than failurePercent...");

Fail fast, one exception, done. That check now lives in PolicyValidator,
Core-side, returning a ValidationDiagnostic instead of throwing — and
PolicyFile.Load calls that same validator, same as it always did, just
collecting every finding instead of stopping at the first one:

public static GatePolicy Load(string path)
{
    var document = Deserialize(path);
    var validation = PolicyValidator.Validate(document);

    if (!validation.IsValid)
        throw PolicyFileException.FromValidationResult(path, validation);

    return PolicyCompiler.CompileValidated(document);
}

validate calls the same PolicyValidator.Validate and just... returns
what it finds, without compiling anything:

public static ValidationResult Validate(string path)
{
    var document = Deserialize(path);
    return PolicyValidator.Validate(document);
}

Same rule, same code path, two different callers with two different
needs. check needs a compiled, guaranteed-valid GatePolicy or nothing.
validate needs the findings themselves — including ones check would
never show you, because a file with only warnings still loads fine as far
as check is concerned.

The same split applies to baseline files (SnapshotValidator, which is
what caught the duplicate identity above) and to BenchmarkDotNet result
JSON (ObservationValidator, plus a second ObservationSetValidator for
catching the same benchmark identity duplicated across two different
result files in a directory — a genuinely different scope from
"duplicated within one file," and worth its own diagnostic).

Diagnostic codes are not implementation detail

Every finding has a stable, documented code — BGV1xx for policy,
BGV2xx for baseline, BGV3xx for BenchmarkDotNet input. The codes are
numbered by what they mean, not by the order I happened to write the
checks in — BGV100/BGV101 are "missing schema version" and
"unsupported schema version" as two separate codes, not one, because a
missing field and an explicitly-wrong value have different fixes. Once a
code ships, it means what it means; a future version can add codes, never
repurpose one.

What v0.3.0 didn't fix

BdnParameterStringParser — the thing that turns
"N=1000000,Distribution=Canonical" into a parameter dictionary — still
silently drops a malformed fragment ("N1000000", no =) instead of
reporting it. I know, because I went looking for exactly this kind of gap
while building the observation validator, and found it already documented
in a two-year-old code comment as "a future version could surface this as
a diagnostic."

I didn't fix it here, on purpose. By the time any validator runs, the
parser has already thrown the malformed fragment away — there's no
evidence left to build a diagnostic from. Fixing it means changing what
the parser returns, not adding a check on top of what it currently
discards. That's real, separate, tracked work, not a validate-command
diagnostic I could bolt on this release.

What's next

v0.4.0 — benchmark-gate compare. Pulls the delta calculation out of
the evaluator: compare describes what changed between a baseline and a
current run, independent of any policy; check applies policy to that
same comparison instead of computing it inline. Full sequencing is in
ROADMAP.md.

dotnet tool install --global Bijecta.BenchmarkGate.Tool --version 0.3.0-alpha.1

Repo, README, and the full roadmap are at
github.com/Bijecta/BenchmarkGate.

BenchmarkGate, Part 2: Why a Baseline Isn't Enough

Youness Aamiri — Wed, 29 Jul 2026 09:04:36 +0000

In part one, I introduced
BenchmarkGate — a local-first
performance regression gate for BenchmarkDotNet. v0.1.0-alpha.1 could do
one thing: compare a benchmark's mean time against a committed baseline,
using a single global threshold passed on the command line.

That was enough to catch a regression. It wasn't enough to trust the tool.

v0.2.0-alpha.1 is live now, and this is the part of the story where the
tool stopped being "diff two numbers" and started being a gate you can
actually build a CI contract on.

The run that made the case for itself

Here's real output from check, run against CedarRecon's benchmark suite:

# Benchmark Gate — CedarRecon

**Overall: ❓ Unstable**

| Total | Improved | Passed | Warning | Regressed | Missing | New | Unstable |
|---|---|---|---|---|---|---|---|
| 27    | 0        | 6      | 0       | 0         | 0       | 0   | 21       |

21 of 27 benchmarks came back Unstable. Not passed, not failed —
unstable. Every one of them has the same explanation:

[Unstable] ClassifierPhaseBenchmark.BuildDictionaries|job=Job-ZDAPPP|N=10000:
Only 3 measurements were taken, below the configured minimum of 5.

Those benchmarks ran with only 3 iterations — not enough for the tool to
trust what it was looking at. A naive diff-the-means tool would have
happily compared 3-sample noise against a baseline and reported a
confident Passed or Regressed. BenchmarkGate refused. That refusal
is the feature.

The other 6 benchmarks, from a different BenchmarkDotNet job with a real
iteration count, evaluated cleanly:

| Benchmark                                                          | Metric          | Baseline    | Current     | Delta  | Status |
|---|---|---|---|---|---|
| ExceptionClassifierBenchmark.DictionaryClassifier|job=Job-SNYTAA|N=10000 | meanNanoseconds | 4.946 ms | 4.946 ms | +0.00% | Passed |
| ExceptionClassifierBenchmark.IndexedClassifier|job=Job-SNYTAA|N=1000000 | meanNanoseconds | 472.444 ms | 472.444 ms | +0.00% | Passed |

Same suite, same check invocation, two entirely different verdicts —
because the tool actually looked at how each result was produced before
deciding whether it meant anything.

What changed since v0.1

v0.1 had exactly one lever: --threshold-percent, applied to mean time,
globally, with no way to say "trust this benchmark less" or "allocation
matters here too." v0.2 replaces that with three things.

1. A policy file, not a flag

benchmark-gate check \
  --results ./BenchmarkDotNet.Artifacts/results \
  --baseline ./benchmarks/baseline.json \
  --policy ./benchmarks/policy.json

policy.json:

{
  "schemaVersion": 1,
  "stability": {
    "minimumMeasurements": 5,
    "maximumCoefficientOfVariation": 0.10
  },
  "metrics": {
    "meanNanoseconds": {
      "direction": "lower-is-better",
      "warningPercent": 5,
      "failurePercent": 10,
      "minimumAbsoluteChange": 0
    }
  }
}

This is a committed, reviewable file — a change to your performance budget
shows up in a pull-request diff, same as any other code change. That was
true in spirit in v0.1; in v0.2 it's actually true, because there's a real
document to diff instead of a CLI flag buried in a CI YAML file.

2. A stability gate that runs before anything else

Every benchmark is checked against stability first — measurement count,
coefficient of variation — before a single metric comparison happens. Fail
that check, and the benchmark is Unstable, full stop. No metric verdict
gets attached to a sample the tool doesn't trust.

This is what produced the 21 Unstable results above. It's also, I'd
argue, the most important thing v0.2 added — not because "unstable" is an
exciting status, but because not having it means every noisy CI runner
eventually produces a false regression, and false regressions are how
people learn to ignore the gate entirely.

3. Warning, not just pass/fail

A metric crossing warningPercent but not failurePercent is Warning —
visible in every report, but it only affects the process exit code if you
explicitly ask for that with --fail-on-warning:

benchmark-gate check \
  --results ./BenchmarkDotNet.Artifacts/results \
  --baseline ./benchmarks/baseline.json \
  --policy ./benchmarks/policy.json \
  --fail-on-warning

Without the flag: a suite with only warnings exits 0, but the Markdown,
JSON, and console reports all still show the underlying Warning status —
they never silently upgrade it to Regressed just because the exit code
happened to be zero. With the flag: the same suite exits non-zero, and
every report format — including a new JUnit XML output for CI test-result
UIs — agrees.

benchmark-gate check \
  --results ./BenchmarkDotNet.Artifacts/results \
  --baseline ./benchmarks/baseline.json \
  --policy ./benchmarks/policy.json \
  --markdown ./report.md \
  --json ./decision.json \
  --junit ./junit.xml

That consistency — CLI exit code, JSON's exitCode field, and JUnit's
<failure> elements all telling the same story — mattered enough that I
wrote tests specifically to pin it down, not just eyeballed it once and
moved on.

Multi-metric, not just mean time

policy.json isn't hardcoded to mean time. Allocation tracking is a
second first-class metric:

"metrics": {
  "meanNanoseconds": {
    "direction": "lower-is-better",
    "warningPercent": 5,
    "failurePercent": 10
  },
  "allocatedBytesPerOperation": {
    "direction": "lower-is-better",
    "warningPercent": 1,
    "failurePercent": 5,
    "minimumAbsoluteChange": 1024
  }
}

If your benchmarks run with [MemoryDiagnoser], both metrics get
evaluated independently, and a benchmark's overall status is whichever
metric did worst — a mean-time pass doesn't hide an allocation
regression. Under the hood this isn't two hardcoded fields; it's an
extensible metric dictionary, so a future metric (throughput, GC
collections, whatever) is a policy entry, not a schema migration.

What v0.2 cost: two breaking changes

Being pre-1.0 meant I could make the calls that actually made the tool
better instead of carrying compatibility baggage for an audience that
doesn't exist yet:

Baseline schema bumped v1 → v2. The old single meanNanoseconds field couldn't hold a metrics dictionary. Old baselines are rejected outright with a message pointing at capture — no migration shim, no silent reinterpretation.
--threshold-percent/--minimum-absolute-change-ns are gone. --policy replaces them entirely.

Both are documented, both are deliberate, and both happened before anyone
but me depended on this tool. That's the whole point of shipping early
alphas.

What's next

Two commands are next, in order, per the actual roadmap:

v0.3.0 — benchmark-gate validate. Validates a policy, baseline, or BenchmarkDotNet input file on its own — catches malformed data (an inverted warning/failure threshold, a missing required field, an unsupported schema version) before it ever reaches check.
v0.4.0 — benchmark-gate compare. Pulls the delta calculation out of the evaluator entirely: compare describes what changed between a baseline and a current run, independent of any policy; check then applies policy to that same comparison instead of computing it inline. That split is what makes everything after it — history, and eventually explaining why something regressed — possible without re-plumbing the evaluator again. Further out, once compare exists: history-aware baselines (v0.5.0), which is also where environment compatibility checking lives — filtering out comparisons across incompatible machines/runtimes before they can produce a misleading result. That's real and coming, just not the immediate next step.

Separately, and not version-gated to any of the above: net8.0
multi-targeting, so the tool installs on machines with only the .NET 8
LTS runtime, not just .NET 10.

Full sequencing, including what's deliberately parked past v1.0.0
(diagnostic explanation, EventPipe integration), is in the repo's
ROADMAP.md.

If you want to try it:

dotnet tool install --global Bijecta.BenchmarkGate.Tool --version 0.2.0-alpha.1

Repo, README, and the full roadmap are at
github.com/Bijecta/BenchmarkGate.

BenchmarkDotNet Measures Performance—But Who Enforces the Budget?

Youness Aamiri — Fri, 24 Jul 2026 11:13:13 +0000

You add [MemoryDiagnoser] and [Benchmark] to a class, run dotnet run -c Release, and BenchmarkDotNet hands you back a beautifully formatted table:
mean, median, standard deviation, allocated bytes per operation. It's one of
the best measurement tools in any ecosystem.

And then what?

You read the numbers. You compare them, in your head, to what you remember
from last week. Maybe you screenshot the table and paste it into a Slack
message. Maybe you don't, because nothing regressed today, so there's
nothing to say.

That's the gap. BenchmarkDotNet is a measurement tool. It has no opinion
about whether 4.5ms is acceptable or a regression. It doesn't remember what
the number was yesterday. It doesn't fail your build. It doesn't stop a pull
request. Every one of those decisions is left as an exercise for the reader
— which in practice means it doesn't happen, until someone notices
production got slower and goes spelunking through git blame.

A budget is not a measurement

The distinction I kept running into while working on
CedarRecon, a reconciliation
engine where I'm doing a lot of comparative benchmarking (dictionary-based
vs. indexed classification strategies, that kind of thing): a measurement
tells you what happened. A budget is a policy — a committed, reviewable
statement that says "this operation must not take more than X, and here's
what we do if it does."

Those are different artifacts with different lifecycles. A measurement is
disposable — you take it, you look at it, you move on. A budget needs to be:

Committed — visible in a pull request diff, not a number someone remembers
Reviewable — a change to the budget is itself a decision someone signs off on, the same way you'd review a change to a config file
Enforced — a CI failure, not a Slack message someone might read

BenchmarkDotNet gives you excellent measurements. It was never trying to be
the second thing.

Why not just eyeball it in CI logs?

The honest answer is: you can, for a while, on a small project with one or
two people watching closely. It stops working the moment either of those
conditions changes.

I found this out concretely, not hypothetically. While dogfooding an early
build of a tool I've been writing specifically to close this gap (more on
that below), I ran a real benchmark suite from CedarRecon, deliberately
introduced a regression into one method to confirm the tool would catch it
— and it did, correctly, twice. But on the same run, an unrelated method I
hadn't touched showed a 28% regression at the largest input size.

If I'd been eyeballing a CI log, that's the kind of number that either gets
ignored ("probably just noise, I'm busy") or gets chased for an hour before
someone realizes the machine was probably running warm. What actually
happened: I reran just that benchmark in isolation, and it came back
clean — comfortably within a few percent of baseline, no regression at all.
It was noise. A shared CI runner, or a laptop that hadn't idled down between
runs, produced one bad sample.

That's not a knock on BenchmarkDotNet — its statistics are good. It's a
demonstration of why "look at the number and decide" doesn't scale as a
process, even when the underlying measurement tool is solid. You need
something that applies a consistent rule every time, and — just as
importantly — something that treats a single suspicious result as a
hypothesis to confirm, not a verdict to act on immediately.

What a performance contract actually needs

Turning "here are some numbers" into "here is an enforced budget" means a
few concrete things have to exist that BenchmarkDotNet doesn't try to
provide:

A committed baseline. Not a number in someone's head — a file in the
repo. JSON, reviewable, diffable. When a baseline changes, that's a
deliberate act someone did on purpose, visible in the PR.

A threshold policy. How much regression is acceptable before it's a
failure? 5%? 15%? Does a tiny absolute change (1ns → 1.2ns) count, even
though it's "20%"? These are decisions, and they should live in
configuration, not in someone's judgment call at 4pm on a Friday.

A pass/fail exit code. The single most useful thing a CI step can
produce. Not a report someone might read — a build that goes red.

No SaaS, no hosted database. This one's a deliberate design choice more
than a hard requirement, but it matters to me: baselines and policies as
plain repo-local JSON mean no external account, no network dependency to
evaluate a benchmark, and — critically — the same command produces the same
decision locally and in CI. Nothing hidden behind a dashboard only CI can
reach.

Where this is going

That's the shape of the tool I've been building —
BenchmarkGate,
a local-first performance regression gate designed specifically for
BenchmarkDotNet output. It doesn't run your benchmarks. It doesn't replace
BenchmarkDotNet's statistics. It reads the JSON BenchmarkDotNet already
produces, compares it against a committed baseline under a policy you
control, and returns an exit code CI can act on — plus a Markdown summary
that explains why, not just that.

v0.1.0-alpha.1 is live now, built and dogfooded against CedarRecon's real
benchmark suite — including that 28% false alarm, caught and correctly
dismissed by rerunning rather than reacting.

Next up: the architecture — normalized benchmark identities, why baselines
and policies are deliberately separate documents, and the exit-code
contract that makes this usable as an actual CI gate and not just a fancier
report.

From dictionary to columnar: how we made a .NET financial reconciliation engine 2x faster

Youness Aamiri — Tue, 14 Jul 2026 19:00:00 +0000

When you're building financial reconciliation software, correctness is the
first requirement. Performance is the second. At small scale the two rarely
conflict. At a million transactions per run, they start to fight each other —
and the data structures you chose at the beginning of the project start to
look like liabilities.

This is the story of how we took CedarRecon's exception classification engine
from a dictionary-based implementation running at 1,368ms per million
transactions to a columnar engine running at 646ms — a 2.1x improvement —
and what we learned along the way, including four approaches that made things
worse, not better.

What Exception Classification Does

Reconciliation engines do two things: match transactions and classify the ones
that couldn't be matched. Matching is interesting in its own right (see our
earlier post on the hash-based matching engine), but classification is what
produces the output that compliance teams actually act on.

The classifier takes three inputs:

Unmatched source transactions (what we sent)
Unmatched target transactions (what we received)
Matched pairs (transactions already resolved by the matching engine)

And assigns every unmatched transaction one of eight discrepancy types:

DuplicateInSource     — multiple source rows share the same reference
DuplicateInTarget     — multiple target rows share the same reference
SplitPayment          — one source, multiple targets for the same reference
ConsolidatedPayment   — multiple sources, one target for the same reference
AmountMismatch        — closest target has a different amount
DateMismatch          — closest target has the same amount, different date
MissingInTarget       — no target exists for this source
MissingInSource       — no source claimed this target

These eight types are assigned in a strict cascade — earlier phases take
priority and a row classified in phase 1 is never reclassified in phase 6.

The classification is a pure function: same inputs, same outputs, every time.
That constraint turns out to be important for how we structured the
optimization work.

The Starting Point: Dictionary-Based Classification

The original implementation built four dictionaries from the input data:

var sourceByRef = unmatchedSource
    .GroupBy(tx => tx.NormalizedReference.Value)
    .ToDictionary(g => g.Key, g => g.ToList());

var targetByRef = unmatchedTarget
    .GroupBy(tx => tx.NormalizedReference.Value)
    .ToDictionary(g => g.Key, g => g.ToList());

var matchedSourceLegCounts = matchedPairs
    .GroupBy(p => p.Source.NormalizedReference.Value)
    .ToDictionary(g => g.Key, g => g.Count());

var matchedTargetLegCounts = matchedPairs
    .GroupBy(p => p.Target.NormalizedReference.Value)
    .ToDictionary(g => g.Key, g => g.Count());

Then the 8-phase cascade ran lookups against those dictionaries. Every
lookup hashed a NormalizedReference.Value string — O(string length) per
lookup, called millions of times across the cascade at N=1M.

The benchmark numbers were not good:

N	Mean	ms/1K rows	Growth
10K	42ms	4.2ms/1K	—
100K	632ms	6.3ms/1K	1.5x
1M	8,400ms	8.4ms/1K	2.0x

Super-linear scaling at 2.0x from N=10K to N=1M. At N=1M, 8.4 seconds per
classification run. For a batch reconciliation system running overnight,
that's acceptable. For anything approaching real-time, it isn't.

We set a benchmark discipline before touching anything: every optimization
claim must be backed by BenchmarkDotNet evidence, hypotheses that turn out
wrong get documented as negative results, and we never batch multiple changes
into one unbenchmarked commit. That last rule saved us several times.

First Wins: DictionaryBuilder and Parallel Batching

DictionaryBuilder

The first obvious improvement was the dictionary construction itself.
GroupBy().ToDictionary() makes two passes and allocates intermediate
IGrouping objects. A single-pass approach using
CollectionsMarshal.GetValueRefOrAddDefault builds the same structure in
one pass with roughly half the allocations:

public static Dictionary<string, List<Transaction>> Build(
    IReadOnlyList<Transaction> transactions)
{
    var dict = new Dictionary<string, List<Transaction>>(
        transactions.Count, StringComparer.Ordinal);

    foreach (var tx in transactions)
    {
        ref var list = ref CollectionsMarshal.GetValueRefOrAddDefault(
            dict, tx.NormalizedReference.Value, out _);
        list ??= new List<Transaction>();
        list.Add(tx);
    }

    return dict;
}

Result: 2.4–3.7x faster dictionary construction, ~50% less allocation,
and the super-linear curve for this specific phase flattened to roughly
linear. This was the clearest win of the entire investigation, and it cost
about 20 lines of code.

Parallel Batching

The cascade's more expensive phases (Split/Consolidated, Mismatch) could
run in parallel across reference groups. We added Parallel.ForEach with
ConcurrentBag collection and a merge step, with a configurable
ParallelThreshold (default 10,000) below which parallel overhead isn't
worth it.

Combined effect of both fixes: 2.02x faster at N=1M.

But the whole-method scaling curve still climbed — 0.816ms/1K at N=10K
rising to 1.196ms/1K at N=1M, a 1.46x growth ratio. The dictionary
structure itself was still the problem.

Dead End #1: FrozenDictionary

The steepest-growing phase was SplitConsolidatedPhases — 3.06x growth
from N=10K to N=1M. The obvious hypothesis: FrozenDictionary<TKey, TValue>,
introduced in .NET 8, offers faster lookups than Dictionary<TKey, TValue>
for read-only scenarios. We'd build the dictionaries once and freeze them
before the cascade.

The benchmark:

N	Dictionary	FrozenDictionary	Δ
10K	1.21 µs/1K	0.95 µs/1K	−22%
100K	1.87 µs/1K	2.23 µs/1K	+19%
1M	3.09 µs/1K	4.35 µs/1K	+41%

FrozenDictionary won at N=10K and lost badly at N=100K and N=1M — the
opposite of what we needed.

The lesson: the problem wasn't lookup speed. It was scan pattern. At large
N, the dictionary's bucket array and the heap-allocated List<Transaction>
objects it pointed to were scattered across memory with no spatial locality.
Every "lookup" was really a cache miss chain: hash → bucket → list pointer
→ list contents, none of it co-located. A faster hash function doesn't fix
a cache miss problem.

This distinction — lookup speed vs access pattern — shaped everything that
came after.

The Indexed Engine: Sorting Instead of Hashing

If hashing was the wrong model, what was the right one? For sequential
access over a known key set, sorted arrays.

A sorted array of transactions, sorted by normalized reference, has all
transactions for the same reference in a contiguous block. Iterating over
them is a sequential memory access — exactly the pattern CPU prefetchers
are built for. A dictionary's entries, by contrast, are accessed in hash
order, which for large tables approximates random access.

ReferenceInterner

The first building block: map reference strings to dense sequential integers
(0, 1, 2, ...) once, upfront, so every subsequent operation works on ints
instead of strings.

public sealed class ReferenceInterner
{
    private readonly Dictionary<string, int> _ids = new(StringComparer.Ordinal);

    public int GetOrAdd(string reference)
    {
        if (_ids.TryGetValue(reference, out var id))
            return id;
        id = _ids.Count;
        _ids.Add(reference, id);
        return id;
    }

    public int Count => _ids.Count;
}

After interning, every lookup in the cascade is an array index operation
on a dense integer — O(1), no hashing, no string comparison. The string
dictionary pays its cost once at build time and never appears on the hot
path again.

RefGroup

Instead of Dictionary<string, List<Transaction>>, we built a RefGroup
struct that points into sorted arrays:

public struct RefGroup
{
    public int ReferenceId;
    public int SourceStart;      // first position in sorted source array
    public int SourceCount;      // number of source rows for this reference
    public int TargetStart;
    public int TargetCount;
    public int MatchedSourceCount;  // legs already matched
    public int MatchedTargetCount;

    public int TotalSourceLegs => SourceCount + MatchedSourceCount;
    public int TotalTargetLegs => TargetCount + MatchedTargetCount;
}

One flat RefGroup[] array, one flat source IndexedTransaction[] array,
one flat target IndexedTransaction[] array. All sorted by ReferenceId.
All contiguous in memory. All accessible via array index, never via hash
lookup.

The cascade phases became sequential scans over RefGroup[], with inner
loops over the contiguous row ranges each group described. No dictionaries,
no hash lookups, no pointer chasing.

First indexed-engine results

Phase benchmark, SplitConsolidatedScan (the worst-scaling phase):

N	Dictionary µs/1K	Indexed µs/1K	Δ
10K	1.27	0.82	−35%
100K	1.87	0.64	−66%
1M	3.09	0.91	−71%

Growth ratio: 2.43x → 1.11x. Nearly flat. The cache-locality hypothesis
was confirmed.

Whole-method at N=1M: 0.52x vs dictionary (1.92x faster).

But a new problem had appeared. MismatchScan — which had previously been
cheap — was now the most expensive scan phase and was showing 1.6x growth:

N	MismatchScan µs/1K
10K	7.76
100K	10.79
1M	12.57

We had fixed the scan pattern problem and exposed a different problem: the
arithmetic inside the inner loop.

Item 1: decimal → long (The Instruction-Count Insight)

MismatchScan's inner loop found the closest unmatched target by amount:

var diff = Math.Abs(targetRows[ti].Amount - sourceRow.Amount);

Amount was decimal. On .NET, decimal arithmetic is software-emulated —
there is no native CPU instruction for it. Math.Abs on a decimal is a
method call that executes multiple instructions. At N=1M with the mismatch
phase touching every source row that reached it, this added up.

The fix: store amounts as long AmountMinor (scaled by 10^4) at index-build
time. Math.Abs on a long is a single CPU instruction.

Scale factor choice: 10^4, not 10^2. Most currencies use 2 decimal places,
but ISO 4217 includes currencies with 3 (KWD, BHD, OMR). Scaling by 10^2
would silently truncate those — a data loss bug that would only surface on
specific currency combinations. 10^4 is a strict superset of every realistic
precision in this domain.

var diff = Math.Abs(targetBatch.AmountMinor[ti] - sourceAmountMinor);

Result:

N	Before µs/1K	After µs/1K	Δ
10K	7.76	5.20	−33%
100K	10.79	6.86	−36%
1M	12.57	6.34	−50%

Growth ratio: 1.6x → 1.22x. Half the cost at N=1M, nearly flat scaling.
Same fact class as the FrozenDictionary lesson — understanding what kind
of work is expensive (software-emulated arithmetic vs native instructions)
matters more than optimizing at the wrong level.

Dead Ends #2, #3, #4: Three More Negative Results

Before reaching the columnar design, we tested three more approaches that
didn't work. Documenting them is as important as documenting what did.

Dead End #2: BuildGroups allocation

Hypothesis: List<RefGroup> + [.. groups] spread in BuildGroups
allocated two arrays — the list's backing array and a second array from
the spread. Replacing with a pre-sized RefGroup[] + count cursor would
eliminate the second allocation.

Result: Allocated column unchanged at every N. The realistic dataset
always has some matched-only references (contributing to internedCount
but not to count), so count < internedCount on every run, meaning
Array.Resize fired just as often as the old spread. The "double
allocation" only exists when count == internedCount exactly — a case
the realistic dataset never produces.

Mean time improved slightly (~17% at N=100K) but inconsistently. The change
was kept (not worse) but the hypothesis was wrong.

Dead End #3: ReferenceInterner reverse list

Hypothesis: ReferenceInterner built a List<string> for reverse
lookup (GetValue). Since GetValue is diagnostics-only and never called
during classification, making it optional should reduce allocation at no
time cost.

Result: Allocation down 10–16%. Time consistently +5–34% worse at
all N.

This is also where we learned that 3 benchmark iterations is not enough for
effect sizes in the 5–35% range. The first run (3 iterations, noisy) showed
an apparent win on both allocation and time. A rerun with 15 iterations
(tighter error bars) revealed the time regression clearly. We reverted the
change and adopted warmupCount: 3, iterationCount: 15 as the standing
benchmark convention from that point forward.

Dead End #4: Index-sort (sort int[] instead of structs)

Hypothesis: Array.Sort on an IndexedTransaction[] moves whole
~20-byte structs on every swap. Sorting a parallel int[] rowOrder by
ReferenceId would move only 4-byte ints — smaller element, cheaper sort.

Result: The sort did get cheaper. But every scan phase then needed one
extra array dereference to read a row: rows[rowOrder[i]] instead of
rows[i]. At N=1M, that extra dereference for every row read in every
phase cost more than the sort savings.

Phase (N=1M)	Struct-sort µs/1K	Index-sort µs/1K	Δ
SplitConsolidatedScan	2.81	4.53	+61%
DuplicateScan	5.55	7.34	+32%

The pattern from FrozenDictionary reappeared: indirection costs more than
it saves at production scale. Cache misses from pointer chasing dominate
any algorithmic savings from working with smaller elements.

The Columnar Leap: Struct-of-Arrays

The index-sort investigation clarified the real problem. Even after
compacting IndexedTransaction to ~20 bytes, array-of-structs still had
a structural inefficiency: every cache line loaded for any phase contained
all fields, even ones that phase never read.

DuplicateScan only checks whether multiple rows share the same reference.
It never looks at amounts or dates. But in an array-of-structs layout, the
cache line containing a row's ReferenceId also contains AmountMinor and
DayNumber. That's wasted bandwidth at every cache line boundary.

Struct-of-arrays separates the columns:

public sealed class ColumnarTransactionBatch
{
    public int Count { get; init; }
    public required int[]  OriginalIndex   { get; init; }
    public required int[]  MatchKeyId      { get; init; }
    public required long[] AmountMinor     { get; init; }
    public required int[]  DayNumber       { get; init; }
    public required byte[] ProcessingState { get; init; }
}

DuplicateScan touches only ProcessingState. MismatchScan touches only
AmountMinor, DayNumber, and ProcessingState. MatchKeyId is read
only during index build. When a phase iterates over AmountMinor, every
byte in every cache line is an amount value — 100% useful data.

ProcessingState: the generic execution buffer

One design decision worth explaining: ProcessingState is a byte[], not
a ClassificationState[]. The enum exists and works fine for external
APIs, but on the hot path we use byte constants:

internal static class ClassificationStateBytes
{
    public const byte None                = 0;
    public const byte DuplicateInSource   = 1;
    // ...
    public const byte MissingInSource     = 8;
}

The reason: ProcessingState is an execution buffer, not a typed domain
field. The same array can be reused by a future matching operator with
completely different byte semantics — no reallocation, just a different
interpretation. This is how database execution engines handle per-row status:
one generic buffer, operator-local meaning. Translate to stable domain types
(DiscrepancyType) before exposing results; the buffer stays internal.

Columnar scan phase results (N=1M)

Phase	Array-of-structs µs/1K	Columnar µs/1K	Δ
MismatchScan	6.34	4.54	−28%
MissingSweep	2.96	1.45	−51%
SplitConsolidatedScan	2.81	3.45	+23%
DuplicateScan	5.55	5.12	−8%

Most phases improved. SplitConsolidatedScan regressed — unexpected, but
it's a phase that barely touches column data and is dominated by RefGroup
iteration, so the columnar layout didn't help it as much.

The IndexBuild regression

There was a catch. The initial columnar builder used sort-then-scatter:

// Sort a parallel int[] by MatchKeyId
Array.Sort(sortOrder, (a, b) => tmpMatchKeyId[a].CompareTo(tmpMatchKeyId[b]));

// Scatter each column
for (var i = 0; i < n; i++)
    originalIndex[i] = tmpOriginalIndex[sortOrder[i]];  // random read

At N=1M, tmpOriginalIndex[sortOrder[i]] is a random read — sortOrder[i]
is an arbitrary position in a large array that doesn't fit in cache. Same
problem as the index-sort approach in dead end #4, just in the build phase
instead of the scan phases.

IndexBuild at N=1M: +18% worse than array-of-structs.

The whole-method benchmark was still positive (0.50x vs dictionary, 2x
faster) because scan-phase wins outweighed the build regression. But we
had a known problem.

Histogram Sort: The Final Piece

MatchKeyId is a dense integer — it's an interned reference ID assigned
sequentially from 0. Dense integer keys are the ideal case for counting
sort (also called histogram sort).

Instead of a comparison sort (O(n log n)) and a random-read scatter, we do:

Pass 1: Count rows per key in one sequential scan:

for (var i = 0; i < n; i++)
    counts[tmpMatchKeyId[i]]++;

Pass 2: Prefix sum to get start offsets:

var running = 0;
for (var k = 0; k < keyCount; k++)
{
    starts[k] = running;
    running += counts[k];
}

Pass 3: Place rows using a cursor per key:

var cursor = starts.ToArray(); // copy of starts
for (var i = 0; i < n; i++)
{
    var key = tmpMatchKeyId[i];
    var pos = cursor[key]++;    // cursor advances sequentially within each key
    finalAmountMinor[pos] = tmpAmountMinor[i];
    // ... other columns
}

The reads from tmp* arrays are sequential (we iterate input order).
The writes to final* arrays are contiguous within each key bucket
(cursor[key]++ means consecutive rows for the same key go to consecutive
positions). No random reads anywhere.

Total: O(n + k) instead of O(n log n). For typical reconciliation data
with high reference reuse, k ≪ n and this is effectively O(n).

The merge-join disappears

There's a bonus: RefGroup[] construction used to require a merge-join
walk over both sorted arrays to discover group boundaries. With histogram
sort, starts[k] and counts[k] are already exactly SourceStart and
SourceCount for every reference k. RefGroup[] is built in a single
O(k) pass over the key space, with no traversal of row arrays at all:

for (var k = 0; k < internedCount; k++)
{
    var sourceCount = source.Counts[k];
    var targetCount = target.Counts[k];
    if (sourceCount == 0 && targetCount == 0) continue;

    groups[count++] = new RefGroup
    {
        ReferenceId  = k,
        SourceStart  = source.Starts[k],
        SourceCount  = sourceCount,
        TargetStart  = target.Starts[k],
        TargetCount  = targetCount,
        // ...
    };
}

Histogram sort results

IndexBuild benchmark (µs/1K rows):

N	Array-of-structs	Scatter sort	Histogram	vs AoS
10K	241	212	139	−42%
100K	448	322	206	−54%
1M	395	464	296	−25%

The N=1M regression (+18% with scatter) became a −25% improvement with
histogram. No trade-offs, no reversals at any N.

Final Results

Whole-method benchmark, DictionaryClassifier vs ColumnarClassifier:

N	Dictionary	Columnar+Histogram	Ratio
10K	8.06 ms	2.52 ms	0.31x
100K	108.6 ms	51.8 ms	0.48x
1M	1,368 ms	646 ms	0.47x

The columnar engine with histogram sort is 2.1x faster than the
dictionary engine at N=1M.

The full journey for MismatchScan specifically — from the original
dictionary baseline through every optimization:

Stage	N=1M µs/1K	vs original
Dictionary engine	12.57	baseline
Indexed engine (item 1, decimal→long)	6.34	−50%
Columnar engine (item 6, AoS→SoA)	4.54	−64%

What We Learned

1. Know which kind of slowness you have. FrozenDictionary failed because
the problem was cache miss patterns, not lookup speed. Index-sort failed
because the problem was read indirection cost, not sort element size. Every
time we correctly identified the bottleneck type, the fix worked. Every time
we guessed, it didn't.

2. Benchmark discipline is non-negotiable. Three of our negative results
(FrozenDictionary, index-sort, scatter sort) would have been hard to
predict analytically. The fourth (reverse list removal) showed an apparent
win with 3 iterations that reversed with 15 iterations. Intuition about
what should be faster is unreliable enough that "benchmark it" isn't just
good practice — it's the only reliable epistemology for performance
engineering at this level.

3. Document the dead ends. Every negative result in this investigation
tells you something about the system that a positive result doesn't.
FrozenDictionary's failure taught us the problem was scan pattern, not
lookup speed — which directly led to the sorted-array design. Index-sort's
failure taught us that indirection costs more than element size saves at
scale — which directly influenced the histogram sort design. If we'd
hidden those results, we'd have lost the reasoning.

4. Dense integer keys change the game. Once ReferenceInterner existed
— once every string key was a dense sequential integer — a whole class of
algorithms became available. Counting sort. Direct array indexing. RefGroup
construction from prefix sums. The interning step was inexpensive (one
dictionary lookup per distinct reference, upfront) and unlocked everything
that came after.

5. The database execution model scales. ProcessingState as a generic
byte buffer with operator-local semantics, ColumnarTransactionBatch as a
shared execution context, refgroup-based range access rather than
per-row object navigation — these patterns exist in database engines for
the same reasons they work here. Financial reconciliation data at scale
has more in common with a database's internal representation than with a
collection of domain objects.

What's Next

The engine still has known costs worth addressing. The temporary arrays in
ColumnarIndexBuilder (~20 bytes/row) are the remaining source of
super-linear growth at N=1M — eliminating them would require a two-pass
encode (first pass counts, second pass places directly into final arrays).

The ProcessingState design also sets up a future build strategy selector:
when the matching engine and quality gates share the same
ColumnarTransactionBatch, those operators may have different key
characteristics. Dense keys get histogram sort. Sparse integer keys get
radix sort. Arbitrary comparison keys fall back to struct-sort. The
selector doesn't exist yet — it will when the second operator needs it.

CedarRecon is open source. The full benchmark suite, all negative results,
and the complete investigation history are in the repository.