The Problem Isn't Tests—It's Authority
Talk about AI-generated code long enough, and you'll hit this question.
"Tests pass, so what's the problem?"
Not wrong. But it misses the core issue.
In AI-generated code, the real question is: who decides 'this is correct'?
Tests? Reviewers? Another LLM?
None of them.
This isn't about improving accuracy. It's about where the authority to declare truth lives.
What We're Actually Doing
1. Trusting Tests
"Tests pass, so we're fine."
In practice, this translates to:
"Wechecked a few cases and nothing broke."
This pattern repeats. Especially after adding LLMs.
Most teams have been here.
Tests sample. They miss edge cases. They rarely cover invariants.
So this happens:
All tests pass. Production breaks.
Not an exception. A structural result.
Test passage is observation. Correctness is a property.
2. Trusting Human Review
"A person reviewed it, so it's fine."
Even less stable.
Humans don't scale. LLM code output explodes. Reviews drift toward "looks about right."
More fundamentally:
Code review is not correctness verification. It's a consensus process.
Consensus can be wrong.
3. Trusting LLM-as-Critic
A popular idea lately.
"Run another model to double-check?"
Sounds reasonable. Many teams try this.
But the structure is:
Probabilistic system + probabilistic system
Result stays the same:
Still probabilistic.
You can create consensus. But:
Consensus is not proof.
Two Common Reactions
Raise this topic, and responses split two ways.
First:
"The problem isn't correctness—it's shipping useful things fast."
Second:
"Invariants aren't perfect either, so keep verifying with LLMs."
Both sound reasonable. Both collapse at the same point.
"Shipping vs Correctness" Is a False Choice
Fast at first.
A few tests. One review. Ship it.
But over time:
Every fix breaks something else. Unpredictable. Debugging costs explode.
Eventually:
Lack of correctness kills shipping speed.
With LLMs, this happens much faster.
Why "LLM Verification" Collapses
Second idea:
"LLM can verify invariants too, right?"
This is usually where intuition misaligns.
Structure breaks here.
1. No termination
Need a verifier → who verifies the verifier? → another model? another?
No end.
2. Results aren't fixed
Same input → different output
At this point:
You can't consistently state "what's correct."
3. Authority vanishes
Who's the final judge?
The model? Training data? Prompt?
No clear answer.
This isn't a system. It's:
"A structure that drifts toward whatever seems plausible."
The Question to Ask Again
What matters isn't the method.
"In this system, what decides 'correct'?"
No answer means the system is already unstable.
Trust Boundary Categories
Not all boundaries are equal.
Empirical boundary: Tests, benchmarks, runtime monitors. Observation-based. Cannot prove absence.
Social boundary: Code review, approval workflows. Authority-based. Doesn't scale.
Formal boundary: Invariants, type systems, proofs. Mathematical necessity. Deterministic.
Most systems use a mix of all three.
The problem is not recognizing which one you're relying on.
Where Should the Trust Boundary Be?
Simple choice.
Somewhere:
A point must decide "this is correct."
And that point:
Cannot be probabilistic.
Redefining the Structure
BEFORE (what most systems do)
LLM → Code → Tests → Ship
↓
(uncertain)
No clear decision point anywhere.
AFTER
LLM → Proposes (may be wrong)
System → Encodes to spec
Proof → Decides validity ← (this is the boundary)
Execution → Runs only what passes
The boundary exists explicitly.
The difference is not the tools.
It's where the boundary sits.
One thing matters:
The verification step must be non-negotiable.
Core Definition
A trust boundary is where correctness becomes non-negotiable.
Before: exploration, generation, probability
After: execution, responsibility, determinism
Don't draw this clearly, and:
The entire system stays in "ambiguous territory."
This Isn't a New Idea
Formal methods. Proof systems. Invariant-based design.
These already exist.
What's different now:
We haven't placed them at the center of code generation pipelines.
So What Does This Look Like in Practice?
Naturally, this structure emerges:
Define invariants first
Prove code satisfies them
Execute only what passes
This isn't about writing better tests.
It's about redrawing the trust boundary.
Axiom
One attempt to implement this: Axiom.
Define state in terms of invariants
Decide correctness through proof
Block regression structurally
If the boundary has been implicit until now,
Axiom tries to make it explicit and enforce it.
The key isn't features—it's position.
Clarifying "what holds final authority."
Conclusion
AI systems come down to one of two:
1. Probabilistic systems
Mostly right. Sometimes wrong. Don't know where.
2. Deterministic systems
Only proven code passes. Upfront cost. Doesn't fail.
No middle ground.
Final Question
In your system, who can say "this is correct"?
Can't answer clearly?
Then the trust boundary doesn't exist.
Top comments (0)