Three ways our automated quality checks passed while the work was broken
Our production line runs in stages with quality gates between them. Research,
build, review, publish. A stage cannot advance until its gate returns green. That
architecture is correct and I would build it again. It also gave me the three
most expensive failures of the year, all of the same shape: the gate was green
and the work was wrong.
One: the gate could not see the data
We had a check that verified our published archive for anachronism, meaning an
article dated before a tool existed cannot mention that tool. It returned clean.
Clean meant nothing, because I had not verified that the check could match
anything at all. When I instrumented it, the term table did match: 26 terms and
over 340 occurrences across the corpus, with no violations. That was a real pass.
But it was luck that I checked, and a green light from a rule that cannot see the
data is worse than no rule, because it stops you looking.
The gate now warns when zero terms match. A check that finds nothing has to say
whether it found nothing or saw nothing.
Two: the gate had a rule that rewarded gaming
An early quality gate asked whether the output existed and had the required
sections. Output always existed. Sections always appeared. The gate was
satisfiable by producing the shape of the work rather than the work.
When a model writes its own gate, this is the default outcome, not the edge case.
The gate gets written to be passable, because passing is the visible objective.
We changed how we test gates. Every gate now gets a deliberate violation injected
before we trust it. If the red light does not come on, the gate does not exist.
The anachronism check above was verified that way: a fabricated article dated
January 2024 mentioning a model released in August 2025, plus a link pointing
forward in time. Both rules fired, exit code 1.
Three: the counter said there was capacity
Our pipeline tracks engine quota locally so it can pick a lane. The counter said
capacity was available. The engine had been failing silently for some time, and
the pipeline kept assigning work into a lane that could not run it.
A local counter is a model of a remote system, and the model drifts. Any counter
maintained on your side of a network boundary needs to be reconciled against the
thing it claims to describe, or it will report health for a dead component.
What these have in common
None of these failures were visible in the output at the moment they happened.
They were visible only in the distance between what the check claimed and what
the check could actually observe.
If you take one thing from this: a rule being written down is not the same as a
rule being enforced, and the only way to know which one you have is to break it
on purpose and watch what the system does.
Top comments (0)