Intent — the spec says what correct means.
Verification — tests check whether you got it.
Correction — a loop closes the gap.
This post is what happened when I stopped running those as three separate practices and wired them into one system.
The layers were the easy part.
The hard part is where they join, and there are exactly two joints.
The Real Problem: I Was the Wiring
Running three methodologies "concurrently" looked like this:
- The spec lived in GitHub Issues.
- The tests lived in the repo.
- The loop lived in a skill that re-ran an agent until a task list emptied.
All three were real.
The connection between them was me, noticing things.
I noticed that a failing test meant a PRD assumption was wrong.
I noticed that slices five through nine came from a version of the spec that no longer existed.
I noticed that an agent had "fixed" a failing test by making the test weaker.
That's not a system.
That's three components and an operator, and the operator is the part that doesn't scale.
Each layer had a trigger, an input and an output.
None of them had a defined interface to the layer next door, so the interface defaulted to my attention.
So the goal isn't:
"Adopt all three."
It's:
Give the handoffs between layers a name, a trigger and an artifact, so they happen without me.
The Flow
The system looks like this:
SPEC
(what correct means)
|
v
+---------+---------+
| |
v v
work slices tests written
and FROZEN first
| |
v |
agent implements |
| |
+---------+---------+
|
v
run the tests
|
+---------+---------+
| |
v v
code is wrong spec looks wrong
INNER LOOP OUTER LOOP
retry N times STOP, no retries
agent fixes it human decides
| |
v v
human gate, amend the spec,
by blast radius re-derive the rest
Two loops.
The difference between them is the whole design:
The inner loop fixes the code.
Runs unsupervised, on a retry budget, allowed to act.
The outer loop fixes the spec.
Runs on escalation, no retry budget, allowed only to propose.
Changing the spec means changing intent, and intent isn't something an agent gets to change by itself.
A loop that can edit its own target will move the target instead of solving the problem.
Every time.
The rest of this post is those two joints.
Joint One: Your Tests Can't Just Be a Copy of Your Spec
Part two ended on this claim, and it's harder than it sounds:
A feedback loop with a bad sensor is worse than no loop, because it converges, confidently, at machine speed, on the wrong thing.
The obvious move is to generate your tests from your spec.
That's the spec-driven pitch, and it works.
But it buys you exactly one thing:
It catches the code disagreeing with the spec.
Now notice what it can't catch.
If the spec is wrong about the real world, a test generated from that spec is wrong in the same direction.
It passes.
Green checkmark.
Everyone goes home.
Safety engineering has a name for this: common-cause failure.
Two backups don't help if they share a design flaw, which is why aircraft use sensors built on different physical principles instead of just duplicating one.
Your spec-generated test suite is one channel, and its design is the spec.
What Can the Test Actually Catch?
| What went wrong | Example | Can a spec-generated test catch it? |
|---|---|---|
| Code doesn't match the spec | PRD says retry 3 times, code retries once | Yes. This is its job |
| Spec contradicts itself | Requires strict ordering on a queue that doesn't guarantee it | Yes, cheaply |
| Spec is silent | Says nothing about ordering, so the agent picks one | No. No test exists for an unstated rule |
| Spec is wrong about the world | "The provider's charge endpoint is idempotent." It isn't | No. The test asserts the same falsehood |
| Spec is right, the requirement was bad | Flawless build of the wrong feature | No, and no test ever will |
Rows three and four are where money gets lost.
The fix isn't a better prompt.
It's a second kind of test.
Two Kinds of Tests, and Only One Can Challenge the Spec
I now split tests by what they're measured against, not by unit/integration/e2e.
Spec Tests
Generated from the spec.
Measured against intent.
Property tests over the stated invariants, assertions on the out-of-scope list.
Cheap, generated, and I make a lot of them.
They ask:
Does the code do what we said?
Reality Tests
Not generated from the spec.
Measured against the world.
Provider sandbox checks, fault injection, replays of real incidents, reconciliation against the provider's ledger.
Expensive, mostly hand-written, they accumulate slowly.
They ask:
Does this survive contact?
The line that reorganized my thinking:
Only a test that didn't come from the spec can prove the spec wrong.
Anything derived from the target can confirm the target.
Nothing derived from it can contradict it.
Which gives a hard rule:
A failing spec test is never grounds for changing the spec. It means the code is wrong. Only a failing reality test is allowed to say the spec is wrong.
That sounds like bureaucracy until the first time an agent responds to a hard failing test by suggesting you amend the PRD.
Which it will, because that's the cheapest path to green.
Three Habits That Keep the Spec Tests Honest
1. Write the Tests Before the Code — and Freeze Them
Not the same session.
Not after there's an implementation to look at.
Sequencing buys independence almost free.
The agent writing the code may not edit its own tests.
If a slice genuinely needs a test changed, that's not a code change.
It's a claim about intent, and it escalates.
This one rule killed my worst failure mode.
2. Hunt the Silences
The clarifying-questions pass from /to-prd was aimed at my rough prompt.
I now run it again on the finished spec, asking only:
What does this not say that an implementer would have to guess?
Each answer becomes a spec edit or a recorded "don't care."
Both are durable.
An agent's undocumented assumptions are the most dangerous thing in the system, because they're the only part that isn't written down anywhere.
Joint Two: What Happens When the Loop Decides the Target Was Wrong
The Failure This Exists to Prevent
A loop with a retry budget and no escape hatch doesn't stop when the spec is wrong.
It keeps going, because that's what loops do.
And since it can't reach the spec, it reduces the error the only other way available:
by weakening the test.
Relax the assertion.
Widen the tolerance.
Skip the case with a plausible comment.
Special-case the failing input.
Each move is locally reasonable, and each one lies to you afterward.
A loop that can weaken its own tests will always converge. That's not a feature, that's the bug.
So the outer loop isn't mainly a correction mechanism.
It's a stop button.
And stopping is its most important capability.
The Stop, and the Four Verdicts
When a reality test fails in a way the inner loop can't fix, the loop halts that branch.
Zero retries.
Because retrying a spec error just buys more attempts at the wrong problem.
Then it files a spec challenge that blocks the parent PRD.
For example:
spec_challenge:
spec: PRD-412 @ a3f19c2
assumption: "provider charge endpoint is idempotent on our key"
contradicted_by:
test: reality/provider-sandbox-conformance
# must be a reality test
observed: "duplicate charge on retry with identical key"
reproduced: "5 of 5 runs"
blast_radius:
slices_blocked: [4, 5, 7]
slices_already_merged_on_this_assumption: [1, 2]
smallest_fix_proposed: |
key on provider_ref only; add 15m reconciliation sweep
against provider ledger.
Four fields do the work.
1. Test Class
Because it decides who's even allowed to file this.
2. Slices Already Merged
Because that's the expensive number and I want it before I decide anything.
3. Smallest Fix
Because "smallest" is load-bearing.
An unconstrained agent will propose rewriting the spec around its discovery.
4. Reproduced
Because "failed once" and "failed five out of five" are different conversations.
Then a human decides.
There are exactly four answers, and naming them turned a judgment call into triage:
- The spec was wrong. Amend it, cascade downstream. The intended path.
- The test was wrong. Fix the test. If it was a spec test, that usually means the requirement was ambiguous, so the spec gets touched too.
- The world is wrong. The provider's docs lie. The spec gets a workaround with an expiry date and a ticket, because undated workarounds become permanent architecture.
- The requirement was wrong. Not the wording, the requirement. This leaves engineering entirely. Good systems just surface it early and cheaply.
Stamp Everything With the Spec Version
An amendment describes what changed, not the whole world again.
Then it propagates, in three tiers with very different costs.
Not Started Yet
Regenerate.
Free.
This is the entire payoff of spec-driven work.
In Flight
Halt, throw away, re-derive.
Costs tokens and nothing else.
Implementations are disposable; that was the bet.
Already Merged
The one nobody warns you about.
It isn't automatically wrong, but it is now unverified and has to be re-checked against the amended spec.
So every derived artifact carries the spec revision it came from:
- Issues
- Generated tests
- Commit trailers
For example:
Derived-From: PRD-412@a3f19c2
Unglamorous, and it converts the worst question in the system from an archaeology dig into a search.
Which merged work came from a spec that has since changed?
Without stamps, the honest answer after three amendments is:
"Some of it, let me read the git log."
With them, you get a list.
And lists can be worked.
One More Guardrail
If the same assumption gets amended twice in one feature, I stop.
Two amendments mean I'm not correcting my model of the system.
I'm searching for one.
That's a spike, run deliberately in a throwaway branch, with nothing derived from it until it finishes.
A target that moves every time a test complains isn't a target.
What Got Worse
It's worth being honest about what this architecture doesn't solve.
Reality Tests Are Expensive
Reality tests are expensive and don't generate well.
That's the honest asymmetry here.
The channel I need most for correctness is the one that can't be derived from the artifact I have, by definition.
Fault injection.
Real sandboxes.
Recorded incidents.
It accumulates at human speed.
Anyone selling fully generated verification is selling you one channel and calling it two.
Escalation Fatigue
Which is the same bottleneck again.
If the loop escalates too readily, spec challenges pile up and get rubber-stamped exactly the way oversized diffs do.
I've moved the constraint from reviewing code to deciding about intent.
That's a better place for it, far more leverage per decision, but part one's point holds and it's recursive:
You don't eliminate a bottleneck, you relocate it. Then you go find it again.
More Artifacts to Keep True
Spec.
Spec tests.
Reality tests.
Version stamps.
Part two warned that a hybrid fails as a stale spec that agents still trust.
Version stamps make that detectable, not impossible.
What I Still Don't Have
Provenance Tooling
No tooling for any of the provenance.
Commit trailers and issue labels, which is to say conventions and discipline, which is to say it will decay.
This feels like something that should exist.
If it does, tell me.
The Next Generation of Senior Engineers
No answer to part one's open question.
If the mechanical work is where judgment used to get manufactured, and that work is now agentic, where do the next senior engineers come from?
Deciding spec challenges is excellent practice for exactly the skill that matters, and it's also the task I'd hand to the most experienced person in the room.
Which means it isn't a training ground.
I'm suspicious of anyone who claims to have solved this.
Where This Leaves the Series
Part one: your job moved from building the system to designing the system that builds the system.
Part two: sort the methodologies by where truth lives between runs, and you get three layers.
Part three: the layers were the easy part, and a hybrid is defined by its joints, not its parts.
If one thing survives out of all of it:
Both joints are the same problem stated twice.
Independence has to be engineered in.
It is never the default.
A test derived from the spec can't correct the spec, and a loop that can edit its target will move the target instead of doing the work.
And part two's caveat applies here more than anywhere.
A lot of this is model-specific error correction with a good name on it.
The rule about tests exists because today's models will happily weaken one to reach green.
The zero-retry rule exists because they'll happily rewrite a target to hit it.
If a future model reliably refuses both, some of this becomes scar tissue and should be cut.
Notice which of your rituals are load-bearing.
Retune when you change models.
And if you've built the same shape and hit a different joint, tell me, because I'm fairly sure I've only found two of them.
Part three of a series on agentic engineering in production. Part one covered the three-skill workflow and where the bottleneck goes when code stops being the constraint. Part two mapped the methodology landscape by asking where truth lives between agent runs.
Top comments (1)
The two-joints framing is useful because agents usually break at handoffs: intent to plan, plan to action, action to verification. Naming the joint makes the failure easier to debug.