The verdict that survived its own correction

This project started with a conclusion already sitting in its lap. In June, an earlier stretch of
work on the same machine had tested whether a local model could handle iterative coding: writing
files, running tests, fixing what broke, without a human shuttling text between a chat window and
an editor. The verdict was no. Not viable. That was the finding the whole project was built on.

On 4 July, someone re-tested it. It fell over in about two minutes.

The interesting part isn't that the verdict was wrong. It's that the verdict was right. It had
been stated precisely, with its own explanation attached, and none of that survived the trip into
the new project.

The re-test

The task was a word search game: three files, a 10x10 grid, eight words hidden horizontally,
vertically or diagonally, click-and-drag to select, correct words struck through in the list, a
New Game button. Eight scored criteria. The eighth is the one that matters: ask for one follow-up
fix (make found words green) and check it lands without breaking the drag highlighting that
already worked. That regression check is what had killed every previous run.

Run through little-coder, a small harness that gives the model real write and read tools instead
of asking it to print code into a chat window, against qwen3-coder-30b served by Ollama, the
result was 8 out of 8 in 2.1 minutes. The follow-up fix took another 30 seconds. All of it verified
in a real browser with simulated drag events, not taken from the model's own account of what it had
done.

The same benchmark against the smaller qwen3.5-9b scored 7.5 out of 8 in 44.1 minutes. It lost
half a point on reverse-placed and diagonal words: no reversed-string check, an off-by-one in the
highlight loop. It still passed the regression test, the first local setup here to manage that.

Model	Score	Time
qwen3-coder-30b	8/8	2.1 min
qwen3.5-9b	7.5/8	44.1 min

The quality gap was only half a point; the time gap was a full 42 minutes.

Two honest marks sit against the 8 out of 8. One transient failure died with "Stream ended without
finish_reason" and an identical retry worked. On criterion 8, one green pixel was never confirmed,
because the preview tab had collapsed to a zero-width viewport and inline styles had stopped
applying: impossible in a real browser, so it was scored as a harness fault rather than a code
fault. That was a judgment call, made and written down as one at the time.

What this project wrote down that evening

The June verdict was a harness problem, not a model problem.

It's a clean line. It reads as an insight. It went into the project's working memory as a headline
finding, was quoted straight into the active constraints file, and shaped everything scoped
afterward: if the harness is the variable that matters, you invest in harnesses and stop shopping
for models.

The investment was right. The claim wasn't, and nobody checked it for two weeks, because a
conclusion that's useful and turning out well doesn't feel like it needs auditing.

What the original verdict actually said

The June work had been handed over as a document. Reading it properly, two weeks late, the verdict
is one row in a table of six use cases:

Use case	Locally viable
Claude Code CLI backend	✕ broke — models misinterpret Claude Code's own system prompt format, not a config problem
Multi-file document extraction (LM Studio + Gemma 4 E4B, 8192 ctx)	✓ met — works well, the best multi-file result of anything tried
Single-file extraction, agent + filesystem (Goose + Ollama, qwen3.5-9b)	✓ met — works, but slow: 62 seconds per file
Iterative coding (any local chat UI)	✕ broke — not viable, and structural: five failure modes, not a fixable bug
Single-turn code generation (Goose or LM Studio, max context)	✓ met — viable as a baseline handoff into a human editor
Quota fallback for real sessions (fcc-server)	✕ broke — routed to Gemini instead, local wasn't trusted as the fallback

June's headline verdicts, all six, from the same document that scoped iterative coding to "any
local chat UI." The row that got re-tested is one of six, not the whole page.

The scope is in the verdict line. Not "local models cannot do iterative coding." Local models
cannot do iterative coding in a chat UI, which is a claim about an interface.

Below it, five structural failure modes are listed. Stateless regression, where a fix in turn N
gets silently undone in turn N+1: observed, in that same word search task, as a drag highlighter
fixed and then broken. Overconfident reporting. Silent context truncation. Effective context not
matching advertised. No cross-session persistence.

And then, closing that section:

Why Claude Code avoids 1–2: tool-mediated verification per step, persistent conversation
context — architecture, not raw model capability.

June had already worked out that the constraint was architectural. It's written down, in the
document that was handed over, in bold, five weeks before the July re-test that "discovered" it.

July didn't overturn June. July supplied the harness June had described, and measured what
happened.

Where June was too optimistic

Of the five failure modes, June expected a harness to handle the first two: stateless regression
and overconfident reporting, both covered by tool-mediated verification and persistent context.

The first half held. The second didn't.

Overconfident reporting is a model claiming completion when a feature doesn't actually work, and a
harness only catches that if its tests are adversarial. This project has watched a weak test
launder a false claim into a green, twice. A redaction check passed with
assert "84210.55" not in out while a partially redacted amount, 842[AMOUNT], leaked straight
through it. A loader passed nineteen of nineteen with a leaked loop variable, masked by a fixture
that only ever exercised one file.

Real file tools, persistent context, tests that ran green: both greens were wrong.

So June wasn't simply more careful than its own summary. On this one point it was optimistic, and
the correction only turned up once this project built the harness June had described and ran it
long enough to get burned.

What actually went wrong

Four things actually changed between June and July, worth stating because it means the re-test was
never a controlled experiment. The model changed. The harness changed. The runtime flags changed
(July ran with flash attention and a quantised key-value cache, June predated both). And the prompt
changed, because June's prompt text was never preserved and had to be rebuilt from scratch.

That last one isn't carelessness. The June raw evaluation file survives, and it's 809 bytes: a
findings summary whose entire task description is one line. There was no prompt left to preserve by
the time anyone went looking. Reconstruction was the only option on the table.

But the four variables are a side issue. The real failure was compression.

What travelled from June into this project was the phrase "iterative coding not viable." The scope,
any local chat UI, didn't make the trip. Neither did the mechanism: architecture rather than raw
model capability.

Then someone re-tested the compressed form, found it wanting, and corrected it into a claim broader
and less accurate than the original. A precise, well-reasoned, correctly scoped verdict went in one
end and came out the other as a slogan, and the slogan is what the project ended up arguing with.

The verdict, June: Iterative coding — not viable (structural, see the five failure modes) — any
local chat UI. Scoped to an interface, with its own mechanism named: architecture, not raw model
capability.

What travelled: "Iterative coding not viable." The scope and the mechanism both stayed behind.

What got re-tested: "Harness, not model." Broader and less accurate than the verdict it
replaced, and the claim this project ended up arguing with.

The handover document's own header says it plainly:

Nothing here needs re-deriving — it was all tested hands-on.

This project re-derived it anyway, and ended up with a worse version.

The part that only appeared while writing this

Writing this piece meant going back to the June sources, and the first attempt to find them failed.
The active constraints file listed the handover document under a path in the other project, the one
that had written it. It wasn't there. Searching that project turned up nothing.

The conclusion drawn from that was that both June primaries had been deleted. That conclusion went
into a correction to this project's own files, into an early draft of this piece, and into a
request asking another project to recover them from git history.

The document was sitting in this project's own repository root. It had been there since the first
commit. It had been written elsewhere and moved across at handover, exactly as its own header says.
The raw evaluation file had moved into a shared knowledge vault during a migration weeks earlier,
and was just as findable. The other project located both files in minutes, and mentioned, almost as
an aside, that it has no git history at all, so the recovery being asked for could never have worked
anyway.

One failed lookup at one stale path became "gone from disk," and a fair amount was built on top of
that before anyone checked the obvious place.

That's the same mistake this piece is about, made while writing about it.

An absence, asserted from a single negative result. A conclusion, drawn from a summary (the path in
the constraints file) rather than from the thing itself.

What changed

The finding was rewritten to what the evidence actually carries. The harness thesis is June's, not
this project's, and it was right. What this project added is confirmation and a measurement. The
broad version, "harness, not model," is now marked as over-stated, because it took this project's
own later evidence, not June's, to find the one place a harness alone wasn't enough.

The practical guidance didn't change at all, which is the uncomfortable part. Invest in the
harness. Verify with real tool output. Keep the local tier for bounded work. All still correct, all
still what this project does. A claim can be over-stated and still point the right way, and that
combination is exactly what stops anyone from checking it again.

Two rules came out of this, both narrow and mechanical. Before re-testing an inherited verdict,
read the primary document, not your own summary of it: an active-constraints file strips scope and
exceptions by design, because stripping detail is what it's for. And an absence needs more than one
failed lookup, especially when the path you searched came from the same compressed source you're
already re-deriving from.

What travels between sessions, projects and people is the conclusion. The conditions that produced
it stay behind. A conclusion without its conditions is a rumour with a number attached.

---

Originally published at thekilted.dev/the-verdict-that-survived-its-own-correction.