Two and a half weeks ago I posted that vibe coding is not a level but an axis: how much you cede to the model per unit of work. Better models mean you can cede more without disaster. Same continuum, different points on it.
That was half the picture. The other half decides whether the code ships or you gambled.
Cession is one axis. Verification is the other. Better models raise the ceiling on the first for free. Nothing about a better model raises the ceiling on the second.
| Low verification | High verification | |
|---|---|---|
| Low cession | Small-script tinkering | Manual coding with tests |
| High cession | Vibe roulette | Real AI leverage |
The expensive failure mode is not the cell everyone flags. It is the drift. Cession climbs generation by generation with each new model release. Verification does not move. GPT-5-codex, Sonnet 5, Fable (Anthropic's latest tier) each raised the first ceiling. Nobody's release notes said "you can now verify 30% less because the model is more articulate."
That is what makes the trap operational. It looks like adoption. Cession went up, defect rate did not immediately spike, so nothing screamed. What actually happened is you moved off the diagonal into vibe roulette and the failures now hit downstream: PRs that pass tests and break in production, migrations that read clean against schemas that shifted, generated functions that echo the shape of a bug that was fixed six months ago.
Three verification patterns that scale differently with cession
Most teams call the first one "tests" and stop. The three that matter separately:
Reproducible check. The kind of test where the same input produces the same failure signal every time. Unit tests, property tests, snapshot diffs. Cheap. This is what CI already runs. It does not scale with cession the way people assume. A model that writes twice as much code produces twice as many places where the reproducible check has to be reproducible, which is not the same as twice as many bugs found.
Adversarial check. A test written to break the specific class of failure the model tends to produce. Fuzzing, mutation testing, chaos injection, prompt-injection harnesses on any tool-using agent. Fewer teams do this. It scales sublinearly with cession because the harness itself is the reusable artifact: one well-designed prompt-injection suite catches the same shape of failure across every new agent feature the model produces, without a new harness per feature.
Independent check. A verification whose authority does not come from the same actor that produced the work. External review by someone who did not write the prompt, staging with real users, canary error rate over threshold, support ticket delta by feature, rollback threshold hit inside the first release window. This is the expensive one. It scales with the stakes, not with the code volume. It is the only class of check that catches "the code passed everything the author knew to test for."
The traps are consistent. Teams add generated code to a codebase that has good reproducible checks and treat that as verification going up. It is not. The reproducible layer stayed flat. The cession layer went up. The gap widened silently.
One diagnostic that surfaces vibe roulette early
Ask two questions of any code the model wrote recently:
- [ ] Was any check on this code authored by someone whose incentive is different from shipping it?
- [ ] Was any check on this code executed by a process that does not know what the author was trying to do?
Two unchecked boxes is the pattern. The code may still be right. Every witness to its correctness is inside the loop that produced it. Self-attestation scales with how wrong you are. The fluency that makes the output look right is what convinces the author their own review found nothing.
If you run an agent long enough to see this in the wild, the pattern shows up in threads, not files. The agent produces the migration and the migration test. The tests pass. The reviewer trusts the tests because tests are the artifact reviewers were trained to trust. The migration ships. Two weeks later a value that was supposed to be non-null is null in production. Nothing in the loop that produced the migration knew that constraint mattered. Nothing in the loop was outside it.
A rule for when a model release lands
The moment you decide to raise cession because a new model is out, the verification budget has to move with it. Not the reproducible check budget. That already exists. The adversarial or independent one. If the release cycle is "adopt the new model on day one, revisit verification quarterly," verification will lag by definition.
The concrete version: for any new model that becomes the default for a class of work, list one adversarial check that will be added because of the cession increase, and one independent check that will run against the first week of its output. Neither has to be big. Both authored before the model runs unattended.
Worked example: if GPT-5-codex becomes the default for migrations in your team, add one schema-drift adversarial test that fuzzes generated migrations against a snapshot of last month's production schema, and route the first week's generated migrations through a human DBA review or a staged replay against production traffic before merge. That is the whole budget update. Boring on purpose.
Otherwise every release moves the free ceiling up. Verification stays. The diagonal rotates without you.
Deeper version on Substack: Vibe Coding Has Two Axes. The Second One Doesn't Get Cheaper.
Top comments (10)
Red flag is your model is producing 100s of tests and they are always green all the time the time. something stinks and its #1 fail is testing the function of existence and not the existence of function.
There is no harm in establishing whats there thats all good. The harm is caused when the model says all clear, your the man, tests=1097638/1097638, code green go go go. It can really mess up the rest of your day having to go back and gut something you didn't check returns anything because all it ever was is a == b.
do for meaningful tests after the model goes into test mode:.
-Tests must fail first.
-Measure with coverage + mutation
-Test for properties and behavior,
-If you only test asserts, your only asserting tests.
That line, testing the function of existence not the existence of function, is the whole failure in six words, and I'm stealing it.
It's the same shape as the reviewer who inherits the ship-mandate. A test that has never failed is a check the producer authored and that agreed with itself. Green means nothing until you've watched it go red for the right reason. Mutation testing is just the write-hop version of that: you make the code wrong and demand the suite notice. If it doesn't, the suite was asserting its own existence, not the code's behavior.
Tests-must-fail-first is the cheapest independence you can buy. No new infrastructure, just refusing to trust a check you've never seen reject anything.
"If you only test asserts you're only asserting tests" is yours, I just set it up. Take it.
Read the ablation table. T1 to T4 holding the model constant is the honest part most memory evals skip, and "gullible" as a third verdict next to correct and wrong is the one that earns its place: a system confidently wrong from stale state is a different failure than a system that just missed. That's exactly the class L4 is aimed at.
One thing I'd poke at, since poking is the deal: L4 tests whether the system detects a fact past expiry. Naive-age versus expiry-aware, 75/50 to 100/100, that split is the right one. But the recompute script makes a run reproducible, not longitudinal. It re-runs the same probe, it doesn't tell you whether the probe's own notion of "expired" still means the same thing six months from now. Stale-by-expiry is a capability you test. Definition-drift in the expiry rule is a staleness the harness itself can carry. Same shape, one level up, and the level the eval sits on.
I'm adding a meta-level expiry-definition invariant: versioned semantics, frozen boundary cases, and a drift detector that distinguishes implementation regression from rule-definition change. I’ll fold it into the work already underway.
I wonder how mad people would get if I just thier OSS system and actually do my best to
The drift detector splitting implementation-regression from rule-definition-change is the whole ballgame, that's the bi-temporal cut made operational: did the code get worse, or did the meaning move. One thing worth wiring in from the start: those two fire different alarms to different owners. A regression pages the engineer. A definition-change pages whoever owns the rule. Collapse them into one alert and every legitimate meaning-shift reads as a bug and someone helpfully fixes it back to the stale definition. Same detector, two exits.
The matrix is good, but "the verification budget has to move with cession" has a reason it doesn't happen, and it isn't technical: cession is visible in sprint metrics (features close faster) and verification isn't. An adversarial harness closes zero tickets. As long as verification is invisible work, its budget loses every planning meeting to the next feature, no matter how right the matrix is.
What worked for us: make the harness a shipped artifact with a name and an owner - "prompt-injection suite v2" sitting in the release notes next to features. Once verification had an artifact you could point at, it stopped being a tax and started winning planning. Your two-checkbox diagnostic is a ready-made definition-of-done candidate for exactly that reason: a checkbox is visible.
Agreed, and I'd go further: the artifact only wins planning once it has a failure mode someone else can point to. A named harness sitting in release notes is visible, but if it never actually blocks a ship or gets skipped under deadline pressure, it decays into decoration fast. The two-checkbox diagnostic works as definition-of-done only if failing a checkbox is a real gate, not a suggestion. Curious whether "prompt-injection suite v2" has ever actually stopped a release, that's the test of whether it bought real budget or just visibility.
Fair test, and yes - twice, which is why it kept the budget. The more useful detail is what the first block did politically: the feature it stopped (user-supplied markdown rendered into model context) got a postmortem, and that postmortem sold the harness better than any planning argument ever did. Nobody debated the second block.
Agreed on the decay risk though - a gate that has never fired is indistinguishable from decoration. Our answer to that is a canary: a known-bad case planted in the suite, run quarterly. If the gate stops catching the canary, that pages someone. A gate needs its own heartbeat, because "green forever" means either nothing bad shipped or the gate died, and from the outside those look identical.
A gate that has never fired is indistinguishable from decoration is the line, and the canary is the right instrument, green-forever means nothing-bad-shipped or the-gate-died and from outside those are identical. One thing I'd add before you trust the heartbeat: a fixed canary proves the gate is breathing, not that it still matches the threat. The gate can be alive and blind at once, it catches the planted known-bad every quarter and misses the real thing because the real thing drifted past the predicates the canary was cut from. Liveness and coverage are different properties and one canary only tests the first.
So the canary wants to age with the threat model, not sit frozen. A known-bad drawn from your recent near-misses tests does-the-gate-still-match-what's-actually-coming-at-us, where the frozen one only tests is-the-gate-conscious. Keep the frozen canary as the liveness heartbeat, it's cheap and it pages when the wiring breaks. But rotate a second canary from the last quarter's escapes, because the gate that catches a two-year-old bug and nothing newer is breathing and still decorative.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.