For decades we've worshipped the Single Source of Truth. A Markdown spec sits in the repo, the team feels total control, everyone closes the ticket and moves on.
Then autonomous coding agents show up, and that comfort evaporates.
A human colleague with ten years of shared context reads "flexibly accept transactions" and fills the gap with tacit engineering judgment. An LLM reads the same sentence and sees a hole in the context — then fills it by sampling from its own probability distribution, delivering the result with a professor's confidence. Feed it a vague spec and it doesn't ask for clarification. It picks map[string]any, generates a pile of type assertions, dumps everything into a JSONB column, and three iterations later your context window is full of spaghetti and apologies.
The sacred spec quietly became a source of the unknown.
The fog isn't one problem
The industry treats "the spec is unclear" as a single, vague complaint. It's actually three distinct failure modes, and each needs a different fix:
| Failure mode | What it is | Example |
|---|---|---|
| Ambiguity | Several plausible readings of the same text | "flexibly accept transactions" |
| Incompleteness | Missing constraints, schemas, acceptance criteria | no Transaction schema |
| Untraceability | A requirement floats free, never decomposed into functions | a slogan with no link to architecture |
Without this split, "spec precision" stays an abstract incantation. You can't fix what you can't name.
Four ways to respond — and why three of them lose
- Write a "perfect" spec in English. It's stale by evening, and the model still finds slack to invent architectural debt from.
- Give up and let the AI vibe. Play prompt roulette, hope the next autonomous refactor doesn't quietly destroy your logic.
- Escape into formal notation — TLA+, Alloy, executable specs. This genuinely helps, but it doesn't remove the unknown; it relocates it: into abstraction choices, model completeness, traceability cost. And it charges years of notation-learning as the entry fee.
- Set a precision budget. Accept that no spec is ever binary-precise. Decide in advance how much slack you'll tolerate — the way a machinist sets a tolerance on a part — then measure whether the document meets it before you let an agent near your codebase.
Only the last option scales, because it's the only one that admits precision is a spectrum you can instrument.
Three metrics, one for each failure mode
Relational Drift ($K_{drift}$) — the cheapest one. Decompose the spec into a requirements graph (RFLP: Requirements → Functional → Logical → Physical) and count how many requirement nodes have no outgoing edge to a lower layer. No LLM in the loop — it's a deterministic graph traversal you can run as a git hook today. If a fifth of your "source of truth" is disconnected slogans, K_drift catches it before an agent has to invent an implementation from nothing.
Constraint Density ($D_{const}$) — the ratio of strictly typed relations to free-floating prose in that same graph. It doesn't punish prose in general, only prose that ties to nothing. A good spec with real rationale can score lower than a bare schema dump, and that's fine — the point is finding nodes that yield zero typed edges.
Generation Spread ($D_{pair}$) — the most interesting one, and the one that took actual experimentation to get right. Generate N interface implementations (Go structs, migration schemas) from the same spec at a fixed temperature, diff them structurally via AST, and measure how far apart they land. A tight spec produces near-identical outputs; a foggy one scatters. This is self-consistency sampling, borrowed from LLM-uncertainty research, pointed backward at the input instead of the output.
Critical caveat: this measures conditional entropy — spread given the spec and the measurement instrument (model, prompt, temperature, parser). Even a perfectly precise spec produces nonzero spread from instrument noise alone. So the number that matters isn't the raw entropy; it's the delta against a calibration baseline run through the identical protocol.
We didn't just propose this — we ran it
Five specs, two independent instruments (qwen3-coder:30b local and kimi-k2.7-code via Ollama cloud), same protocol, N=10 generations each:
- The reference spec (fully typed) hit a noise floor of zero on both instruments — all ten generations identical down to the AST.
- The "sharp" spec (explicit schemas, traceable requirements) still didn't reach zero — the model kept mapping
Decimal(10,2)to Go three different ways. The metric pointed straight at the exact spot the spec left ambiguous. - Two real
docs/requirements.mdfiles from live projects (session-indexer, ragivka) both saturated the entropy ceiling — every single generation landed in its own cluster. - The counterintuitive part: the more formally tagged spec (ragivka, with NFR markup) was actually more scattered than the prose-heavy one (session-indexer) — because NFR tags pin down quality attributes, not type structure. A control experiment confirmed it: adding architecture docs (not more quality slogans) raised generation similarity by roughly the same amount (+0.12) on both independent instruments. That reproducibility is what makes it a finding, not a fluke.
Variance isn't always the enemy
In an implementation zone — a payment pipeline, a DB schema — variance is risk, and you gate it. But in an exploration zone — a new module's design, an unexplored architecture — the same variance is a resource: generation spread becomes a mutation operator for evolutionary search. Stochastic engineering isn't a war on entropy. It's knowing which zone you're in and applying the opposite policy.
Where this stops being useful
No silver bullets here, on purpose:
- The metrics are relative to the measurement instrument — comparing raw numbers across different models without recalibrating is meaningless.
- Low entropy isn't correctness. A spec can deterministically steer every agent toward the same wrong decision.
- Metrics can be gamed — sprinkling meaningless
@schematags raises $D_{const}$ without improving anything real. - The thresholds in this piece (0.2, 0.35, 0.75) are starting hypotheses for your own corpus, not universal constants.
Full write-up — with the actual formulas, the complete quality-gate YAML, and all the honest caveats about instrument noise — is on my blog: Source of the Unknown. Experimental data (six specs, 120 generated files, the Go AST analyzer) ships alongside it.
Valentyn Solomko — Ukrainian software engineer
Top comments (0)