Testing Against State Drift: Guarding a Config Tool Whose Source of Truth Lives Somewhere Else

TL;DR — I maintain a browser-based configuration tool for ESA's Pyxel detector-simulation framework. Its whole job is to help people write configs that a separate, independently-versioned project will accept. That makes drift — my tool's idea of "valid" quietly falling out of sync with Pyxel's — the central failure mode. This post is about the testing and automation I built to catch drift at three different layers, and the one principle that ties them together: test each invariant at the seam where it actually lives.

What you'll get out of it

• A concrete way to think about state drift when your correctness depends on an external source of truth
• Three drift-defense layers — bundled-schema freshness, real-schema bounds tests, and API-introspection checks — with the actual code
• Why I split the suite into automated deterministic logic vs manually-verified rendered output, and how to draw that line
• How per-file coverage gates in CI turn "we should keep this tested" into "the pipeline won't let us not"

---

Contents

1. The problem: correctness you don't own
2. Three kinds of drift
3. Layer 1 — Is the bundled schema still fresh?
4. Layer 2 — Do the real schema's bounds still hold?
5. Layer 3 — Do the docs still match the actual API?
6. Testing at the seam
7. Making the gates non-optional
8. What's deliberately not automated
9. Takeaways

---

The problem: correctness you don't own

Most testing advice quietly assumes you own the definition of "correct." Your code, your tests, your rules. When a test fails, something in your repo changed.

Config tooling breaks that assumption. My tool exists to help scientists produce YAML that Pyxel — a large, actively-developed simulation framework on its own release cadence — will accept and run. Pyxel is the source of truth for what a valid config is. My tool is a helpful intermediary that holds a copy of that truth: a bundled JSON schema, a set of tutorial examples, a mapping of detector types to parameter bounds.

The instant Pyxel ships a new model, renames an argument, or tightens a numeric range, my copy is wrong — and nothing in my repo changed to tell me. The tests still pass. The build still goes green. The tool now confidently blesses configs that Pyxel will reject, or flags valid ones as broken. That's state drift: two systems that are supposed to agree, silently diverging, with no failing assertion at the moment of divergence.

You can't unit-test your way out of that with fixtures alone, because a fixture is also a frozen copy of the truth. If Pyxel drifts, your fixture drifts with it and your green test is lying to you. Guarding against drift means deliberately reaching for the real, external thing at test time — carefully, at the right layer.

---

Three kinds of drift

Once I framed the tool this way, the drift surface split cleanly into three, each needing a different defense:

Layer	The copy that can drift	The truth it must match	How I detect it
Schema freshness	bundled pyxel_schema.json	Pyxel's published schema	live diff of definition names
Argument bounds	my bounds-extraction logic	real values inside the bundled schema	tests run against the real schema file
Tutorial / doc accuracy	70+ tutorial .md examples	the installed Pyxel Python API	introspect Pyxel, check every example

The rest of the post is these three, in order.

---

Layer 1 — Is the bundled schema still fresh?

The tool ships a bundled copy of Pyxel's schema so validation works offline and instantly. The risk is obvious: the day Pyxel adds a model, the bundle is stale and can't validate configs that use it.

Rather than pretend that never happens, the tool checks at runtime and tells the user the truth. It fetches Pyxel's published schema and diffs the set of defined model names against the bundle:

export async function checkSchemaFreshness(bundledSchemaText) {
  try {
    const res = await fetch(PYXEL_SCHEMA_URL, { cache: "no-store" });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    const upstream = JSON.parse(await res.text());
    const bundled = JSON.parse(bundledSchemaText);

    const bundledNames = definitionNames(bundled);
    const upstreamNames = definitionNames(upstream);
    const missing = [...upstreamNames].filter((n) => !bundledNames.has(n));

    if (missing.length === 0) return { state: "current" };
    return { state: "stale", missing, messages: [ /* human-readable summary */ ] };
  } catch (err) {
    return { state: "unknown", messages: [ /* "couldn't reach Pyxel…" */ ] };
  }
}

Three states, not two. current and stale are the happy and sad paths — but unknown is the one that matters for trust. If the user is offline, or Pyxel's host is down, or the response is HTML instead of JSON, the tool must not claim freshness it can't verify. It says so plainly: validation reflects the bundled schema only.

That third branch is exactly the kind of thing that rots untested, so it's pinned down hard. The freshness suite mocks fetch and drives every path — superset-is-still-current, singular-vs-plural messaging, five-name truncation with an ellipsis, non-OK HTTP, a rejected promise (offline/CORS), unparseable HTML, and an assertion that the request actually sets cache: "no-store" so a cached response can't produce a false "current":

it("reports 'unknown' when upstream returns unparseable JSON", async () => {
  vi.stubGlobal("fetch", mockFetchResolving("<!doctype html><html>"));
  const result = await checkSchemaFreshness(BUNDLED);
  expect(result.state).toBe("unknown");
});

The invariant being protected isn't "the schema is fresh" — I can't guarantee that. It's "the tool never lies about whether the schema is fresh." That's a property I fully own, so it gets full coverage.

---

Layer 2 — Do the real schema's bounds still hold?

The tool reads numeric argument bounds (min / max, inclusive / exclusive) out of the schema so it can warn when a value is out of range. That extraction logic has ordinary unit tests against a small synthetic schema — the fast, focused kind you'd expect.

But synthetic fixtures have the drift problem baked in: they prove my parser works, not that it still works against Pyxel's actual data. So a second, smaller set of tests runs the same extractor against the real bundled pyxel_schema.json:

describe("extractArgBounds — against the real bundled pyxel_schema.json", () => {
  const realSchema = JSON.parse(readFileSync(schemaPath, "utf8"));

  it("reads cutoff_wavelength bounds for dark_current_rule07", () => {
    expect(extractArgBounds(realSchema, "dark_current_rule07")).toMatchObject({
      cutoff_wavelength: { min: 1.7, max: 15.0 },
    });
  });

  it("reads an exclusive-minimum arg for usaf_illumination", () => {
    expect(extractArgBounds(realSchema, "usaf_illumination")).toMatchObject({
      multiplier: { min: 0, exclusiveMin: true },
    });
  });
});

These assert on specific values from the shipped schema. If a future schema refresh changes cutoff_wavelength's range, or flips an inclusive bound to exclusive, this test fails loudly at refresh time — which is precisely when I want to find out, not weeks later from a confused user. It turns a silent data change into a red pipeline. The synthetic tests prove the logic; these pin the logic to reality.

---

Layer 3 — Do the docs still match the actual API?

This is the layer I'm most attached to, because docs are where drift hides best. The tool ships 70+ tutorial files, each with YAML examples showing how to configure a Pyxel model. Prose examples don't get executed, so they rot invisibly: an argument gets renamed upstream, and the tutorial keeps cheerfully showing the old name forever.

So the CI pipeline installs a pinned real version of Pyxel and introspects it — no fixtures, the actual package — to discover the ground truth, then checks every tutorial against it:

def discover_pyxel_models() -> dict[str, dict[str, set[str]]]:
    """Return {group: {model_name: {valid_argument_names}}} from installed Pyxel."""
    catalog = {}
    for group in PIPELINE_GROUPS:
        module = importlib.import_module(f"pyxel.models.{group}")
        for name in dir(module):
            obj = getattr(module, name)
            if not inspect.isfunction(obj):
                continue
            sig = inspect.signature(obj)
            args = { p for p, param in sig.parameters.items()
                     if p != "detector" and param.kind not in (VAR_POSITIONAL, VAR_KEYWORD) }
            catalog.setdefault(group, {})[name] = args
    return catalog

Every YAML block in every tutorial is then parsed and checked against that live catalog. A tutorial fails if it references a model that doesn't exist, puts a real model under the wrong pipeline stage, or passes an argument the actual function signature doesn't accept:

for key in args:
    if key not in valid:
        problems.append(
            f"func '{func}': invalid argument '{key}' "
            f"(valid: {sorted(valid) or 'none'})"
        )

There are two details here I'd happily defend in review. First, there's a guard test asserting introspection discovered more than 40 models — so if an import path changes and discover_pyxel_models() silently returns almost nothing, the suite fails on that rather than falsely reporting every tutorial as clean. A drift check that can quietly discover nothing is worse than no check, because it's green and wrong.

Second, because these tutorials are partly authored with LLM help, the checker also greps for stray assistant chatter — "Certainly!", "Would you like me to…", "Here are the N Mermaid diagrams" — that has no business in a published tutorial. That's a different kind of drift (content contamination rather than API mismatch), but it lives at the same seam, so it's caught in the same pass.

The Pyxel version is pinned in CI (PYXEL_VERSION: "2.17.1"). Bumping it is a deliberate, visible act — and when I bump it, this job tells me immediately which tutorials the new version just invalidated. Drift becomes a code review, not a surprise.

---

Testing at the seam

Across all three layers, the same principle keeps showing up: test each invariant at the layer where it actually lives, and mock the layer below it.

The freshness checker, the schema loader, and the live telemetry sender all touch the network. None of them hit the network in tests. fetch (and the YAML parser) are replaced with mocks, so the suite exercises the logic around the request — cache hit/miss, HTML-instead-of-YAML detection, missing-models guards, offline fallbacks, the OTLP body shape, the span rate limiter — deterministically, with zero flakiness:

function mockFetchResolving(bodyText, { ok = true, status = 200 } = {}) {
  return vi.fn().mockResolvedValue({ ok, status, text: async () => bodyText });
}

This is the same move as mocking the schema loader when testing pure validation logic, and it's why the JS suite runs in a plain Node environment with no browser and no live calls. The network is a seam — a clean boundary I can substitute at — and testing right at that seam is what makes otherwise-flaky, environment-dependent branches into boring, reliable regression guards that run on every commit.

The pattern generalizes: find the boundary between the logic you own and the world you don't, and put your test double exactly there. Above it, assert everything. Below it, don't pretend to.

---

Making the gates non-optional

A regression guard nobody enforces is a suggestion. The parts of this codebase that guard against drift — the diagnostics that turn noisy validator errors into friendly messages, the bounds extractor, the shape analysis — are the parts most worth protecting from silent decay, so CI enforces per-file coverage thresholds, not a single blurry project-wide number:

thresholds: {
  "scripts/configShape.js":        { statements: 95, branches: 80, functions: 95, lines: 95 },
  "scripts/modelDiscriminator.js": { statements: 95, branches: 85, functions: 95, lines: 95 },
  "scripts/schemaBounds.js":       { statements: 95, branches: 90, functions: 90, lines: 95 },
  // …
}

Per-file gates matter because a project-wide average lets a well-tested module subsidize a rotting one. Pinning each critical file just under its measured coverage means the pipeline fails the moment someone adds an untested branch to that specific file — the drift-sensitive logic can't quietly lose its safety net behind a healthy-looking global percentage.

One threshold is deliberately low: the module that wires a real AJV instance to the live schema sits at 35%, because most of it is genuinely exercised only in the browser. Rather than fake coverage with brittle DOM tests, the gate is set honestly to what the pure logic covers, and the rest is documented as manual. Which brings me to the last piece.

---

What's deliberately not automated

Not everything belongs in CI, and pretending it does produces flaky suites that teams learn to ignore. This project draws an explicit line — a method-to-layer split:

• Automated: the deterministic logic — parsing, shape derivation, bounds extraction, error de-duplication, freshness diffing, the network seam. Pure functions with clear inputs and outputs. These run on every commit and gate the merge.
• Manually verified: the modules that render to or read from the live DOM — schema renderers, validation overlays, the dynamic model loader, the UI components. These are checked by hand in the browser with devtools, where a human eye catches visual and perceptual problems a headless assertion would miss.

The important part is that this line is written down, in the test README, with the reasoning and the next step named: if the rendered layer ever warrants automation, a headless-browser harness like Playwright (whose auto-waiting removes most timing flakiness) is the natural tool. Until then, "manual" is a documented decision, not an accidental gap. A reader — or future me — can see exactly what's covered, what isn't, and why.

That honesty is itself a drift defense. The most dangerous test suite is the one that looks more complete than it is.

---

Takeaways

1. When you don't own the source of truth, drift is the primary bug. Frame the whole test strategy around "where can my copy silently disagree with the real thing?" before writing a single assertion.
2. Fixtures freeze the truth; sometimes you must test against the real thing. Run a subset of tests against the real schema and a pinned real dependency, so an upstream change turns into a red pipeline instead of a confused user.
3. A check that can silently discover nothing is worse than no check. Guard your introspection with a sanity assertion (">40 models found"), or a broken import will report everything as clean.
4. Test at the seam. Mock the network (or DOM, or external API) right at its boundary; assert fully on the logic above it. Deterministic branches become reliable guards instead of flaky ones.
5. Enforce per-file, not project-wide, coverage. An average lets healthy modules hide rotting ones. Pin the drift-sensitive files individually.
6. Write down what you don't automate. A documented manual-test surface is a decision; an undocumented one is a gap. The suite that overstates its own completeness is the one that eventually bites you.

Drift is quiet by nature — it's the bug that arrives with no failing test and no commit to blame. The only defense is to build the checks that go looking for it on purpose, put each one where it can actually see, and let CI insist on them every single time.