BenchmarkGate, Part 3: benchmark-gate validate

In part two,
v0.2.0-alpha.1 added a stability gate and a policy file. Both are good
things. Both are also new ways to write a file that's subtly wrong — a
threshold typo, an inverted warning/failure pair, a baseline entry missing
its identity — and until now, the only way to find that out was to run
check and read whatever exception came back first.

v0.3.0-alpha.1 adds benchmark-gate validate. It doesn't evaluate
anything. It just tells you, exhaustively, what's wrong with a file —
before that file ever gets near a real check run.

The problem with fail-fast

Here's policy.json with two unrelated mistakes in it:

{
  "schemaVersion": 1,
  "stability": { "minimumMeasurements": 0, "maximumCoefficientOfVariation": 0.1 },
  "metrics": {
    "meanNanoseconds": { "direction": "lower-is-better", "warningPercent": 10, "failurePercent": 10 }
  }
}

minimumMeasurements: 0 is invalid. warningPercent equal to
failurePercent is also invalid. Before this release, PolicyFile.Load
threw on the first problem it found — you'd fix minimumMeasurements,
re-run, and then find out about the threshold. Two mistakes, two
round-trips.

validate finds both in one pass:

$ benchmark-gate validate --policy bad-policy.json
bad-policy.json
  ERROR BGV105 /stability/minimumMeasurements: Value must be greater than zero; actual value was 0.
  ERROR BGV117 /metrics/meanNanoseconds: Metric 'meanNanoseconds' has warningPercent (10) >= failurePercent (10). warningPercent must be strictly less than failurePercent for the policy to be meaningful.

$ echo $LASTEXITCODE
12

Both errors, one invocation, no guessing what else might be wrong once
you fix the first thing.

One command, three file types, together

benchmark-gate validate --policy policy.json
benchmark-gate validate --baseline baseline.json
benchmark-gate validate --results ./BenchmarkDotNet.Artifacts/results

All three flags are optional individually, but at least one is required,
and you can combine them — validate your whole input surface in one call
before wiring anything into CI:

$ benchmark-gate validate --policy bad-policy.json --baseline bad-baseline.json
bad-policy.json
  ERROR BGV105 /stability/minimumMeasurements: Value must be greater than zero; actual value was 0.
  ERROR BGV117 /metrics/meanNanoseconds: Metric 'meanNanoseconds' has warningPercent (10) >= failurePercent (10). warningPercent must be strictly less than failurePercent for the policy to be meaningful.

bad-baseline.json
  ERROR BGV203 /benchmarks/1: Duplicate benchmark identity 'MyApp.Bench.Sort|job=Default'.

Two independently broken files, two different validators (PolicyValidator
and SnapshotValidator), one invocation — each grouped under its own
source heading, with a blank line separating artifacts so a multi-file
run stays scannable instead of turning into a wall of undifferentiated
error text.

Console output groups by source file. Errors print in red when you're
looking at a real terminal; piped or redirected output stays plain, so
scripts and CI logs get deterministic text.

Structured output for machines

--json writes a versioned report covering every artifact you validated
in one call, instead of just printing to the console. Here's a baseline
with a duplicate benchmark entry:

benchmark-gate validate --baseline bad-baseline.json --json validation-report.json

{
  "schemaVersion": 1,
  "isValid": false,
  "errorCount": 1,
  "warningCount": 0,
  "artifacts": [
    {
      "kind": "Baseline",
      "source": "bad-baseline.json",
      "isValid": false,
      "errorCount": 1,
      "warningCount": 0,
      "diagnostics": [
        {
          "code": "BGV203",
          "severity": "Error",
          "title": "Duplicate benchmark identity",
          "path": "/benchmarks/1",
          "message": "Duplicate benchmark identity 'MyApp.Bench.Sort|job=Default'."
        }
      ]
    }
  ]
}

Top-level isValid/errorCount/warningCount let a CI step decide
pass/fail without walking every artifact; each artifact still carries its
own counts and full diagnostic list for anything that wants the detail —
a future benchmark-gate explain BGV203 command, an IDE annotation, a
Markdown summary, whatever ends up consuming this.

Where the checks actually live

This is the part I think is more interesting than the command itself:
validate doesn't contain a single validation rule. Every check it runs
already existed — I extracted it from check's own load path, rather than
writing a second implementation next to the first.

PolicyFile.Load used to do this:

if (warningPercent >= failurePercent)
    throw new PolicyFileException(path, "warningPercent must be strictly less than failurePercent...");

Fail fast, one exception, done. That check now lives in PolicyValidator,
Core-side, returning a ValidationDiagnostic instead of throwing — and
PolicyFile.Load calls that same validator, same as it always did, just
collecting every finding instead of stopping at the first one:

public static GatePolicy Load(string path)
{
    var document = Deserialize(path);
    var validation = PolicyValidator.Validate(document);

    if (!validation.IsValid)
        throw PolicyFileException.FromValidationResult(path, validation);

    return PolicyCompiler.CompileValidated(document);
}

validate calls the same PolicyValidator.Validate and just... returns
what it finds, without compiling anything:

public static ValidationResult Validate(string path)
{
    var document = Deserialize(path);
    return PolicyValidator.Validate(document);
}

Same rule, same code path, two different callers with two different
needs. check needs a compiled, guaranteed-valid GatePolicy or nothing.
validate needs the findings themselves — including ones check would
never show you, because a file with only warnings still loads fine as far
as check is concerned.

The same split applies to baseline files (SnapshotValidator, which is
what caught the duplicate identity above) and to BenchmarkDotNet result
JSON (ObservationValidator, plus a second ObservationSetValidator for
catching the same benchmark identity duplicated across two different
result files in a directory — a genuinely different scope from
"duplicated within one file," and worth its own diagnostic).

Diagnostic codes are not implementation detail

Every finding has a stable, documented code — BGV1xx for policy,
BGV2xx for baseline, BGV3xx for BenchmarkDotNet input. The codes are
numbered by what they mean, not by the order I happened to write the
checks in — BGV100/BGV101 are "missing schema version" and
"unsupported schema version" as two separate codes, not one, because a
missing field and an explicitly-wrong value have different fixes. Once a
code ships, it means what it means; a future version can add codes, never
repurpose one.

What v0.3.0 didn't fix

BdnParameterStringParser — the thing that turns
"N=1000000,Distribution=Canonical" into a parameter dictionary — still
silently drops a malformed fragment ("N1000000", no =) instead of
reporting it. I know, because I went looking for exactly this kind of gap
while building the observation validator, and found it already documented
in a two-year-old code comment as "a future version could surface this as
a diagnostic."

I didn't fix it here, on purpose. By the time any validator runs, the
parser has already thrown the malformed fragment away — there's no
evidence left to build a diagnostic from. Fixing it means changing what
the parser returns, not adding a check on top of what it currently
discards. That's real, separate, tracked work, not a validate-command
diagnostic I could bolt on this release.

What's next

v0.4.0 — benchmark-gate compare. Pulls the delta calculation out of
the evaluator: compare describes what changed between a baseline and a
current run, independent of any policy; check applies policy to that
same comparison instead of computing it inline. Full sequencing is in
ROADMAP.md.

dotnet tool install --global Bijecta.BenchmarkGate.Tool --version 0.3.0-alpha.1

Repo, README, and the full roadmap are at
github.com/Bijecta/BenchmarkGate.