Every one of these had a green suite. 161 tests on the portfolio, 77 on the budget proxy, 53 on the
conformance checker, 78 on the retrieval system. All passing. Each had a benchmark, a published
result, and a live deployment.
Then I wrote scenarios that hit the live URLs instead of the code, and one of them was overrunning
its central guarantee by 303% in production.
This is what that exercise found, including the four times my own harness accused a product of a
bug it did not have.
The rule
A scenario had to be something a person would plausibly do, described in their words. "A runaway
agent loop burns through its budget overnight" is a scenario. "Assert reserve() returns False" is
a unit test.
And nothing in the scenario suite may import product source code. It gets what a stranger gets: a
URL over the public internet.
That second rule is the one that mattered.
What it found
A spend ceiling that did not hold. 25 concurrent workers against a budget with room for two.
Twenty were admitted. Overrun: 303%.
The cause was 25 serverless instances, each holding its own in-memory ledger, each enforcing the
$0.05 ceiling perfectly against its own slice of traffic. There were 25 ceilings instead of one.
The uncomfortable part is where the answer already was. The store's own docstring read "correct for
one replica and useless for two — which is precisely measured failure #1". The deployment doc said
not to run it behind a load balancer. The baseline benchmark had measured the single-replica ledger
as the first of four ways budget enforcement fails open.
Three layers said the right thing and the product was still wrong, because none of them ran
against the deployment.
The first version of the test passed
Worth dwelling on. My scenario asked the service how much it had spent:
✓ 12 workers fired concurrently — 4 allowed, 8 refused
✓ spend after the burst $0.008400 — ceiling $0.05
Both reads landed on an instance that had served one request. The suite was green while the product
overran by 233%.
A test that asks the system under test whether it is behaving cannot detect a system that is
wrong about itself.
Spend is now derived from what a provider would bill — admitted calls times observed cost per call
— which does not depend on the service being right. The ledger's own figure is recorded beside it,
and the disagreement between the two is the finding.
Twelve workers also was not enough: it fanned across fewer instances and passed on some runs.
Twenty-five made it deterministic. A test that fails intermittently gets rerun until it passes.
Four times the harness was wrong
The honest ratio is one real defect to four false accusations, and every false one was mine.
It claimed a portfolio route was broken. It assumed /work, /about and /writing existed,
got a 404, and reported the site as broken. There is no /work index. Routes now come from
sitemap.xml. A scenario that invents the interface measures its author's assumptions.
It claimed the citation mechanism was undeployed. It scraped work pages for citation links and
found zero. Citations are not static links — the site ships a build-time allowlist of pre-verified
passages, and the chatbot may only cite from that set. So a model cannot invent a citation even
when it invents a claim. A stronger design than I had assumed, and verifying the allowlist verifies
every citation the site can ever produce.
It claimed 2 of 34 citations were unresolved. The passages contained apostrophes rendered as
', and my tag-strip did not decode HTML entities. The text was on the page and would
highlight correctly in a browser.
It claimed a side channel in a project about side channels. It asserted that "no token" and
"unknown token" must return byte-identical responses. But an attacker already knows whether they
sent a credential, so telling them carries no information. The property that matters is that two
rejected credentials are indistinguishable — verified against five plausible guessed principal
names, all returning the same body.
Each correction is documented in the code rather than quietly amended, because a test that flags
correct behaviour costs more than no test: the next real finding gets ignored alongside it.
What passing scenarios are worth
Eighteen now pass. A few are worth more than the rest.
One tests that a documented weakness is still real. The retrieval project publishes a
limitation: padded results share no query terms, so the recipient can spot the filler and recover
what was withheld. The scenario passes when that reproduces against production. A published
weakness that turned out to be overstated would be its own kind of dishonesty.
One tests that the baseline still leaks. The comparison only means something if the unfixed
path is genuinely exploitable. Counts across nine identities come back as 0, 2 and 4, and the
deficit tracks permission — the identity with least access infers the most, the CEO infers nothing.
If the baseline did not leak, the enforced result would be measuring nothing.
One tests that the good work stays findable. Three projects were added to the portfolio with
live URLs and reproducible benchmarks. The failure mode that would quietly undo that is not a 404 —
it is the pages existing while being absent from the sitemap, the chat corpus and the MCP tool
list. The site would keep answering, from the generically described employer work, and nobody would
notice the verifiable material had dropped out.
What I would tell someone starting this
Write the scenario against the URL, not the module. The import is what makes a test blind to
deployment.
Compute the thing you care about independently of the system that is supposed to be computing it.
If you are checking a spend ceiling, calculate the bill yourself.
Turn the concurrency up until the failure is deterministic. An intermittent failure is a failure
you will explain away.
And expect your harness to be wrong more often than the product. Mine was, four to one. That ratio
is not a reason to skip it — one of the five was a 303% overrun on the exact guarantee the product
existed to provide.
Scenario suite, findings and reproduction: the scenarios/ directory. Every number here traces to
a benchmark or a scenario result; a launch post that states an unmeasured figure fails its own
build gate.
Top comments (0)