DEV Community: the kilted dev

Nothing to go stale

the kilted dev — Sat, 07 Nov 2026 08:49:28 +0000

One GPU, one machine, and for now exactly one tool that needs it: a local model doing GPU-bound
work a couple of times a week for one project that calls it. Small enough that nothing had broken
yet. Also the moment a design decision is cheapest to get right, before real usage patterns build
up around whatever gets chosen first.

The proposal that came my way was the standard answer to "two things might want the same resource
at once": a lock file. Before using the GPU, claim it by creating a marker. Before finishing, remove
the marker. Anything that shows up while the marker exists waits its turn. It's a well-understood
pattern, and the version proposed was already a careful one: atomic creation so two claims can't
race each other, enough content in the file to tell a genuinely stale lock from a live one.

What a lock file is actually for

I looked closely at why a lock needs to be careful in the first place: it isn't protecting against
the resource being busy. It's protecting against the lock itself lying. Every piece of a well-built
lock's design — atomic creation, a check for whether the lock has gone stale, cleanup on every exit
— exists to defend against failure modes the lock introduces just by existing. A resource that was
never contended in the first place doesn't need any of that. The contention was never the hard
part. The bookkeeping was.

Don't keep a record, ask

The hardware itself already knows exactly what's using it, right now, with no possibility of going
stale, because it isn't a record of a past claim. It's the current state, queried live. A single
command asks the driver directly: what's running, how much memory is it holding. No file to
create, none that can outlive the process that made it, no PID to check against a process table
that might have recycled it by now.

And it fits the actual failure this machine has had: not two processes racing to grab the GPU at
the same instant, but one process hanging without dying and quietly sitting on memory the next job
needs. A lock file wouldn't catch that at all. A hung-but-alive process holds its lock exactly as
validly as a healthy one. Asking the hardware catches it immediately, because the memory is either
there or it isn't, whatever any marker file claims.

What a lock file has to get right:

Create the marker atomically, so two claims can't race
Remove it again, on every path out
A holder that died without removing it
A process ID the operating system has already reused
A holder that hangs without dying, and keeps the lock

What a live query has to get right: nothing. There is no stored claim, so there is nothing that
can disagree with the hardware.

Four of the five are obligations the lock creates for itself. The fifth, creating the marker
atomically, guards a simultaneous race, which is not the failure this machine has actually had.

The one thing a lock file offers that a live query doesn't — a clean error message naming who's
holding the resource — is available from that same query. The command that shows what's running
also shows which process ID owns it.

Zero staleness handling needed, because there's no state to go stale. Not a cleverer lock, no lock at all.

A second cut, caught by asking someone to disagree

A related piece of the same proposal suggested retry logic: if the resource is busy, wait and try
again automatically. Reasonable-sounding, and it was cut too — by an independent review pass
commissioned alongside the decision itself, brought in to look for problems rather than confirm
the plan.

An unattended process retrying its way onto shared hardware is precisely what this project's own
rule against unattended execution already exists to prevent. Elsewhere in the same document, the
proposal that suggested the retry logic had made that exact argument in a different context.
Nobody had done anything dishonest. It's an easy inconsistency to write past when you're the one
building the argument in the first place. The retry logic and the no-autonomous-execution
principle were reasoned about separately, minutes apart, and never checked against each other
until someone whose only job was to do exactly that.

What the proposal asked for:

If the hardware is busy, wait and try again
Automatically, unattended, with nobody watching

What the same document ruled out:

Anything that can fire on its own
Anything that runs with no person present to catch it

One document, two conclusions, minutes apart. Side by side they are incompatible; read in the order
they were written, neither one looks wrong.

Fail fast instead. If the resource is busy, say so and stop. A human decides when to try again.

What made the lazy answer the right one

None of this came from my distrust of locks in general, or a rule that says avoid new machinery. It
came from asking what problem the proposed fix was solving, and finding that its complexity existed
only to manage its own side effects. The resource was never the source of the risk. The simplest
version of the design didn't win by being simpler. It won because it was more correct, for the
failure that had actually happened, and had nothing left over to go wrong.

Originally published at thekilted.dev/nothing-to-go-stale.

The review that caught itself

the kilted dev — Sat, 31 Oct 2026 07:54:50 +0000

Field notes on a review process. The external blog post at the centre of this, its author, and the
project that relayed it are all deliberately unnamed — the piece is about the reasoning, not about
anyone's work in particular.

The idea that survived

A sibling project passed along a blog post about a different team's local-agent harness: sharp,
well-argued, built around one specific rule. Don't rely on an instruction the agent has to remember
to follow. Build the constraint into the tool so it physically can't be ignored. A gate, not a rule.

That part held up completely, and for a good reason — it wasn't new here. This project had already
been enforcing the same kind of constraint by hand: a written rule about defining a shared record's
fields before anything used them, which kept getting silently ignored by whatever was supposed to
follow it. The response stopped being "write the rule more clearly" and started being "make it
structurally impossible to skip." What eventually held was a validator that refused the write.

Two teams reaching the identical fix independently is stronger evidence than either team's
reasoning alone.

The claim that didn't

The draft that came out of the brainstorm made a stronger claim than that. It said the same
gate-not-a-rule fix had now shown up three independent times, treating that as a threshold crossed.

It hadn't.

One of the three cited instances was the same internal case that had made the principle ring true
in the first place. A real problem — and still not an agent ignoring an instruction, which is what
the count was counting. Right case, wrong bucket. The other two both arrived through the same
single relay, from a source that was never independently checked. Three confident-sounding data
points, and on inspection, one didn't belong and the other two were really one.

That correction came from an independent review, not from anyone re-reading their own work more
carefully. It's the specific thing that kind of review exists to catch: a claim that sounds
verified because it was written among evidence, when the evidence was one piece of secondhand
testimony, counted twice.

The quieter catch

Buried in the relay was a line characterising a design's human-approval step as "a stopgap, not a
deliberate choice" — and the draft presented that line as something the original source had said.
The review checked, and it hadn't. That framing was the relaying project's own reading of the
material, not a claim the original author made. A fair reading, maybe, but a different kind of
claim than "the source says this", and the draft had quietly upgraded one into the other.

No one had lied. No one had checked either — the review was the first pass that did.

What was actually adopted

One principle, promoted to policy on its own merits, with the overclaimed count corrected. One
design pattern, confirmed as already covered by an existing rule — whoever built something never
gets to be the only one who says it works — and not built. Two permanent guardrails went into the
record anyway, in case it ever is.

Nothing shipped that hadn't survived the review. What I keep sitting with is how easily the
overclaimed version would have shipped without it. Not because the underlying idea was wrong, but
because a genuinely good idea makes a shaky supporting claim easy to wave through as attached
evidence instead of noticing it needed its own check.

Originally published at thekilted.dev/caught-itself.

Nineteen calls, every time

the kilted dev — Sat, 24 Oct 2026 07:40:01 +0000

Claude's own desktop app kept dying outright, the whole window gone, no warning. Not a hang, not a
freeze that eventually recovers: the process behind the screen simply stopped existing. It happened
five times in one day alone, on a Lenovo Legion 5 laptop. Each one logged the identical signature:
the GPU process reported gone, reason crashed, exit code 101457950, every single time.

I ruled out the obvious suspects first, one at a time, and each one cleared.

Memory looked fine going into the crash: system RAM had over ten gigabytes free at the moment
checked, nowhere near exhausted. The graphics driver never logged a reset around any crash checked
for one, no driver-level event in the system log at all. Nothing was left over from a
previous crash either: once the app went down, the only thing still running afterward was an
unrelated command-line helper, so there was no stale process holding a lock that could explain the
next crash starting from a bad state. Three clean suspects, three dead ends.

The pattern that was hiding in a different log entirely

The actual cause turned up somewhere nobody was looking directly for it: a completely separate log
file, tracking browser-level activity rather than the app's own process events. A specific WebGL
call — getInternalformatParameter, the kind a web page makes to ask the graphics driver about
supported texture formats — showed up in a tight burst right before every crash. Not once or twice. Every single time, across five
separately timed occurrences spanning hours, that burst was exactly the same size: nineteen calls.
Never eighteen. Never twenty. A fixed number repeating exactly, five times, is not what a random
failure looks like. It's what a fixed sequence running to completion looks like, every time it
runs at all.

The trigger was a browser preview pane inside the app, opened by whatever the user was doing at the
time, loading a page that happened to run a script probing WebGL capabilities as part of its own
fingerprinting or feature detection. Each crash landed five to six seconds after the preview pane
had opened. The nineteen-call burst sat right in that gap, every time.

Start — the browser preview pane opens.
In the gap — a page inside it runs nineteen WebGL calls.
5-6s later — the GPU process is gone.

The same three steps behind all five crashes in one day. The nineteen calls sat in the gap between
the pane opening and the process dying, every time.

Opening the pane wasn't enough by itself

One earlier preview-pane open that same day had no burst afterward, and no crash. That's the detail
that rules out "the preview pane itself is the problem." The pane can open cleanly all day. What
kills the process is specifically a loaded page inside it running that exact probe sequence to
completion, not the act of opening the pane at all.

A separate warning line, about a GPU power-preference option being ignored on this platform, showed
up earlier that same morning with nothing bad following it: checked five seconds either side, no
burst, no crash. That same warning also appears at the very top of every one of the actual crash
sequences, immediately before the nineteen calls start. So the warning by itself isn't the trigger.
It's a harmless line that happens to sit right before the real trigger, whenever the real trigger
runs at all, which made it worth ruling out on its own rather than assuming it was just noise.

Suspect	What the check found	Verdict
Memory pressure	Over ten gigabytes of system RAM still free at the moment checked	Ruled out
A graphics-driver reset	No driver-level event in the system log around any crash checked for one	Ruled out
A stale process from the last crash	Only an unrelated command-line helper still running afterward	Ruled out
Opening the preview pane, on its own	One open earlier the same day, no burst after it and no crash	Ruled out
A GPU power-preference warning	Checked five seconds either side, no burst and no crash following it	Ruled out
A loaded page running the probe to completion	A burst of exactly nineteen calls in the gap before every one of the five crashes	Real cause

A sixth possible crash, found and deliberately left uncounted

Going back further in the logs turned up one more candidate burst, eight days before the day with
five crashes, and it doesn't get counted. Here's why. The burst is there, the right size, in the
right log. What isn't there is the matching crash line. The specific stretch of log covering that
exact window had already rotated out of every log file still on disk by the time anyone went
looking. What's on disk instead is a fresh app launch about nine minutes after the burst, with the
previous launch recorded two days earlier, meaning something ended that long-running session before
the new one started, just not on camera. A restart nine minutes after an unexplained burst, with
nothing describing how the previous session actually ended, is consistent with the same crash. It
is not proof of it. Six of the bursts line up with a confirmed crash: one five days earlier, and
all five from the day of the five crashes. The unconfirmed one from further back stays unconfirmed,
named as a maybe rather than quietly rounded into the count. If it belongs in the total,
that's a call for whoever owns the running tally next, made with the gap in the evidence attached,
not asserted on the strength of a nine-minute coincidence alone.

Occurrence	19-call burst	Crash on record
The five in one day	found	found
Five days earlier	found	found
Eight days earlier	found	could not be established

Seven bursts in the logs, six of them beside a crash on the record. The seventh is left out of the
count rather than rounded into it.

What's left open

No crash-diagnostic file was produced by any of the five crashes: the folder where the app would
normally leave one after a hard failure stayed empty every time. The bug is filed upstream as
anthropics/claude-code#80444, alongside
around ten duplicate reports, reproduced across NVIDIA, AMD, and Intel graphics on both Windows 10
and 11, so it isn't specific to this one laptop. All of them are still open. I put a comment on
that issue with this machine's own findings, and the thread is still getting daily activity from
other people hitting the same crash. In the meantime the only mitigation that actually works is not
opening that specific browser preview feature at all, and driving anything that needs a real
browser through a separate, standalone tool instead.

None of this was hard to check, once the search moved from guessing at causes to reading the log
that actually recorded what happened.

The memory number, the driver log, the leftover-process check: all fast, all clean, all wrong turns
that had to be ruled out anyway before the real pattern, sitting in a log nobody had thought to grep
yet, showed up in exactly the same shape five times in a row.

Originally published at thekilted.dev/nineteen-calls-every-time.

Not the integrated GPU

the kilted dev — Sat, 17 Oct 2026 07:55:53 +0000

Field notes on a laptop display-topology migration, still mid-journey.

Two sockets, one controller

Three monitors, one laptop, and one USB dock old enough to be a bottleneck. Two of the three
monitors ran through it: a 4K panel and an ultrawide, both sharing one USB link with the keyboard,
the mouse, and Ethernet. The third monitor, a second 4K panel, was already wired natively straight
to the discrete GPU over HDMI, no dock involved.

The laptop has two USB-C sockets, and the working assumption going in was that a monitor plugged
into either one would land on the integrated graphics, the way a lot of thin-and-light laptops
route USB-C video. The two ports look identical. They aren't quite: one carries power delivery up
to 100W and DisplayPort 2.1, the other tops out at DisplayPort 1.4 with no power delivery at all.
Both are wired to the same USB controller underneath, the only one this laptop has. A different
configuration of the same chassis swaps one of those ports for Thunderbolt; this one doesn't, and
that was confirmed by checking the controller list directly rather than trusting the spec sheet for
this exact model.

The port routes to the discrete GPU, not the integrated one

Moving the ultrawide monitor from the old dock onto the power-delivery USB-C port — native
DisplayPort Alt Mode, the mode that carries real DisplayPort video directly over a USB-C connector,
no dock in between — settled the assumption by measurement instead of guesswork. The monitor came
up attached to the discrete GPU. Not the integrated graphics chip doing double duty on the internal
panel and the USB-C port both, which was the original assumption. A separate, dedicated card, the
same one already driving one of the 4K monitors over HDMI.

Display	Was driven by	Now driven by
Internal panel (2560×1600 · 165Hz)	integrated GPU	integrated GPU
4K panel, HDMI (3840×2160 · 60Hz)	discrete GPU	discrete GPU
Ultrawide (3440×1440 · 60Hz → 100Hz)	the dock	discrete GPU
4K panel, dock (3840×2160 · 60Hz)	the dock	the dock (cable ordered)

Four displays across three adapters, read out of dxdiag before and after the first move. Only the
ultrawide changed hands. The second 4K panel is still on the dock, waiting on a cable.

The other USB-C port hasn't been tested the same way yet. The reasonable guess is that it behaves
the same, since both ports sit on the same controller, but a guess is exactly the thing this whole
exercise exists to replace with a measurement, so it stays a guess until it's checked.

Faster the moment the dock was out of the path

With the ultrawide on its own native connection, dxdiag showed it running at 3440×1440. The old
dock connection had topped out at 60Hz for that monitor, full stop. There was no higher option to
select. The panel itself supports 100Hz; the dock's video path never exposed anything past 60,
because it wasn't sending real DisplayPort signal at all — it was compressing the whole desktop into
a video stream and shipping that over USB.

The refresh rate didn't jump to 100Hz on its own, though. Straight after the re-cable, the display
still reported 59.97Hz, because Windows carries a monitor's previous setting forward across a
different physical connection rather than picking the best one available. The "Choose a refresh
rate" dropdown listed nine options, four of them marked with an asterisk meaning the adapter
synthesised them rather than the monitor actually supporting them:

Refresh rate	Note
99.90 Hz	unmarked
75.03 Hz	adapter-synthesised
72.19 Hz	adapter-synthesised
70.07 Hz	adapter-synthesised
60 Hz
59.97 Hz	selected
59.94 Hz
56.25 Hz	adapter-synthesised
50 Hz

Picking the unmarked 99.90Hz by hand got a 100Hz reading in dxdiag afterward.

A cable upgrade doesn't come with an automatic refresh-rate upgrade attached. Somebody has to go
into the settings and pick it.

A tool that reported the wrong monitor entirely

One check along the way turned up something worth flagging on its own, separate from the dock:
Win32_VideoController, a common way of asking Windows what resolution and refresh rate a graphics
adapter is currently running, gave a confidently wrong answer in this exact multi-monitor setup. It
reported the
integrated graphics chip as running the ultrawide's resolution at 60Hz. The integrated chip was
never connected to that monitor at all. It was driving the laptop's own internal screen, at a
completely different resolution and refresh rate. dxdiag, read directly, had the real picture:
internal panel on the integrated chip, ultrawide hanging off the old dock. The first tool wasn't
lying about a number — it was reporting the wrong adapter's numbers as if they belonged to a monitor
they'd never touched.

Tool	Reported
`Win32_VideoController`	Integrated graphics, 3440×1440 at 60Hz — the ultrawide's mode, on an adapter that was never wired to it
`dxdiag`	Integrated graphics, 2560×1600 at 165Hz — the laptop's own internal panel, which is what it was actually driving

The same adapter, read two ways, on the same machine on the same day. I'd check with a second tool
before trusting a report like that in any setup with more than one screen.

The bandwidth math the dock was quietly doing

The old dock's USB link carries something like 4 gigabits a second, shared across both monitors, the
keyboard and mouse, and the network connection. A single 4K panel at 60Hz, uncompressed, needs
roughly 12.8 gigabits a second on its own. Two monitors, both routed through that one dock, meant
somewhere around 25 gigabits of raw pixel data getting squeezed into a 4-gigabit pipe, continuously,
the entire time both screens were on. Something has to give, and what gave was a real-time
compression pass running in software the whole time, on hardware never asked whether it minded.

Native DisplayPort Alt Mode over USB-C, by contrast, has room to spare: the standard this laptop's
faster port supports moves tens of gigabits a second, more than enough for one monitor at its native
resolution and refresh rate with nothing left to compress. Once the ultrawide moved off the dock,
its own share of that math simply stopped being a problem.

Where this actually stands

One monitor is done: natively connected, correctly identified as running off the discrete GPU, and
finally at the refresh rate it was always capable of. A cable for the second monitor, DisplayPort
this time rather than the general-purpose USB-C to USB-C cable currently in use, has been ordered
but not yet installed. Once it lands, the old dock drops out of the video path entirely and keeps
one job: charging and networking, which is what a general-purpose USB link is good at.

Nothing here was fixed by buying better hardware. The laptop, the monitors, and the faster port were
all already sitting there. The fix was checking which port actually did what, instead of assuming
the label on the box.

Originally published at thekilted.dev/not-the-integrated-gpu.

Blocked, not rejected

the kilted dev — Sun, 11 Oct 2026 08:29:02 +0000

The most capable-looking command in a fifty-seven-item tool catalogue turned out to be five
function calls to a server this setup doesn't run. No trial was run, because narrating what a
missing server would do isn't a test. The verdict needed a name of its own, and blocked fit.

SuperClaude is a configuration layer for Claude Code, installed as an add-on rather than built in:
slash commands, agents and behavioural modes, fifty-seven items in the catalogue in total. One of
them read as the single most capable thing in it.

pm-agent, invoked as /sc:pm, read on paper like a full session-management layer. Cross-session
memory that survives between conversations. Checkpointing every thirty minutes. Self-evaluation, so
the agent could ask itself whether it was actually done before reporting done. The kind of feature
list that makes a reader comparing AI coding tools stop and read it twice.

The check

The catalogue's own summary of /sc:pm never says how any of that actually works. Reading the full
source (agents/pm-agent.md and commands/pm.md in the SuperClaude repository) does.

Every one of those mechanisms turned out to be a literal call to a function on a Serena server.
Serena is an MCP (Model Context Protocol) server, a separate program that Claude Code connects to
for extra tools. The calls are list_memories(), read_memory(), write_memory(),
think_about_task_adherence(), think_about_whether_you_are_done(). Cross-session memory was a
named function call to a server. So was self-evaluation: another named call to the same server.
None of those functions exist without a Serena MCP server actually connected, and at the time of
the review this setup's .mcp.json defined no servers at all. None, checked directly in the file
rather than assumed.

The dependency did not stop there. For full behaviour, /sc:pm also wanted seven more MCP
servers (eight, all told, all absent) loaded dynamically through a third-party Docker gateway,
airis-mcp-gateway. Reading more of the command's own source only lengthened the list of things
that would need to
exist first, never shortened it.

To run the same check on any tool, search its command files for calls to functions that need a
connected server, then compare that list with your own MCP configuration.

Source: github.com/SuperClaude-Org/SuperClaude_Framework

A repeat

The shape had shown up in the framework before. An earlier command, /reflect,
had already been excluded from an earlier pass for the identical reason. Its own file states
plainly, in its own "Will Not" section:

Operate without proper Serena MCP integration and reflection tool access.

Same disqualifying issue, same missing server, a different command. Once is a fact about one
command. Twice is a fact about the framework's own design pattern: describe a capability in the
documentation, implement it as a call to infrastructure the reader is never told they need.

A second objection

Set the missing server aside, and there was still a second, unrelated problem. The command's own
file declared itself the default operating foundation:

The DEFAULT operating foundation that runs automatically at every session start... the default
entry point for all interactions.

Auto-delegating every request to sub-agents before a human ever saw it. That was a takeover of a
workflow already in place, duplicating this setup's own working four-file lifecycle (NOTES.md,
MEMORY.md, CLAUDE.md, DECISIONS.md).

A different project in the same workspace had already declined the exact same shape for the
framework's Deep Research mode: an ungated feature that wanted to become the default way every
session behaved, rather than a tool invoked on request. This setup read that rule, then reached the
same objection on pm-agent and checked it against that record. The missing server was the
independent finding; the second objection leaned on that earlier record.

The verdict

No trial was run. On paper the command looked capable, but a trial without the missing server
would only narrate what write_memory() would do, with nothing real underneath the narration.

A trial without the server would be a plausible performance of a test.

This setup keeps a standing rule for exactly that failure mode: trust the artifact over the summary.
Reading pm-agent's actual source instead of its one-line catalogue description is what caught the gap
before a fake trial could run.

So the verdict on record was not rejected. It was blocked.

A blocked verdict names the specific missing thing and stays open to being wrong about it. If a
Serena MCP server is ever installed here for an unrelated reason, this is worth a second look.
The session-takeover design would still need weighing on its own merits. A smaller, scoped-down
version of the trial was floated too and turned down; that was the project owner's call, and no
second missing dependency lay behind it. A rejected verdict would have closed that door for good.
This one stayed open on purpose.

Originally published at thekilted.dev/blocked-not-rejected.

A Claude Code Stop hook that makes Claude show the full file path

the kilted dev — Wed, 07 Oct 2026 06:58:47 +0000

Claude Code answered Run the tests with scripts\check.py. I needed C:\proj\scripts\check.py. (I staged that exchange: I asked for the short form so the hook had something to catch.)

A single path is easy to work out. A list is harder. In the folder for one video episode, Claude listed six files by short name and I could not tell where any of them were.

I had already put a line in CLAUDE.md, the file where you write standing instructions for Claude:

Always give absolute paths, never relative ones.

Claude kept slipping. So I wrote a Stop hook that checks every reply and sends it back when it names a file by a relative path. The code is in the repo claude-code-stop-hook-full-path, and the video is linked at the end. It runs on Windows and Linux. I have not tried it on macOS.

What it does

On Windows, Claude answers Run the tests with scripts\check.py. The hook blocks the reply, and Claude answers again: Run the tests with C:\proj\scripts\check.py. That was Claude Code 2.1.284 on Haiku 4.5, in a project at C:\proj. The Windows part of the video uses my own hook. The one in the repo is a shorter stand-in with a different message, and it does the same job.

On Linux, the stand-in hook ran in a node:22-slim container with Claude Code 2.1.286 and a project at /home/dev/proj. Claude listed six files by short path. The hook replied Full paths only. Your reply has short paths: and Claude answered again with the full paths:

episode/script.md     ->  /home/dev/proj/episode/script.md
episode/clip-01.mp4   ->  /home/dev/proj/episode/clip-01.mp4
episode/captions.srt  ->  /home/dev/proj/episode/captions.srt
episode/clip-02.mp4   ->  /home/dev/proj/episode/clip-02.mp4
episode/notes.md      ->  /home/dev/proj/episode/notes.md
episode/cover.png     ->  /home/dev/proj/episode/cover.png

What a Stop hook receives and what it can send back

A Stop hook is a command Claude Code runs each time Claude finishes a reply. It gets one JSON object on stdin. Two of its fields matter here. last_assistant_message is the text of the final reply. stop_hook_active is true when Claude Code is already continuing because of a stop hook.

The hook answers on stdout. Printing {"decision": "block", "reason": "..."} keeps Claude from stopping, and reason tells it why it should continue. Printing nothing lets the reply stand. The hooks reference lists exit code 2 with the message on stderr as another way to block. This hook uses the JSON form.

Claude Code allows eight stop-hook continuations in a row before it overrides the next block and ends the turn. The hook checks stop_hook_active anyway: if the rewrite still has a short path, it lets the reply through instead of blocking again.

The hook

This is the repo's stand-in hook, the one that ran in the Linux container. It is 89 lines of Python with no dependencies.

#!/usr/bin/env python3
"""Claude Code Stop hook: when the last reply names a file by a short path, block it and ask for the full path.

Written for Windows, macOS and Linux; tested on Windows and Linux. Reads the Stop-hook JSON payload on stdin and prints
{"decision": "block", "reason": ...} when a reply has a short path, nothing otherwise.
Only paths inside backticks that end in a file extension are checked. A directory without an
extension, or a path outside backticks, is not flagged.
Run `python stop_hook_full_path.py --selftest` to check the cases below.
"""
import json
import re
import sys

TICKS = re.compile(r"`([^`\n]{1,200})`")
# absolute on Windows (C:\ or C:/), POSIX or UNC (leading / or \), home (~), or a URL
ABSOLUTE = re.compile(r"^(?:[A-Za-z]:[\\/]|[\\/]|~|[a-zA-Z][a-zA-Z0-9+.\-]*://)")
SEGMENT = re.compile(r"^[\w.\-{}, ]+$")  # braces and commas allowed: src/{a,b}.ts is one short path
EXTENSION = re.compile(r"\.[A-Za-z0-9]{1,6}$")


def short_paths(text):
    found = []
    for token in TICKS.findall(text):
        token = token.strip()
        if ABSOLUTE.match(token) or not re.search(r"[\\/]", token) or not EXTENSION.search(token):
            continue
        parts = [p for p in re.split(r"[\\/]", token) if p]
        if len(parts) >= 2 and all(SEGMENT.match(p) for p in parts):
            found.append(token)
    return found


def decision(payload):
    if payload.get("stop_hook_active"):
        return None  # already blocked once: let Claude stop rather than loop
    bad = short_paths(payload.get("last_assistant_message") or "")
    if not bad:
        return None
    shown = ", ".join(f"`{p}`" for p in bad[:2]) + (f" (+{len(bad) - 2} more)" if len(bad) > 2 else "")
    return {"decision": "block",
            "reason": f"Full paths only. Your reply has short paths: {shown}. Rewrite them as absolute paths."}


def load(raw):
    # Strip EVERY leading byte order mark: a UTF-8 pipe from Windows PowerShell can send two,
    # and the "utf-8-sig" codec removes only one.
    return json.loads(raw.decode("utf-8").lstrip("\ufeff"))


def selftest():
    cases = [
        ("`episode/clip-01.mp4`", True),
        ("`/home/dev/proj/episode/clip-01.mp4`", False),
        ("`/Users/dev/proj/episode/clip-01.mp4`", False),
        ("`C:\\proj\\episode\\clip-01.mp4`", False),
        ("`episode\\clip-01.mp4`", True),
        ("`src/{a,b}.ts`", True),
        ("Press `q`, then open `src/a.ts` and `src/b.ts`.", True),  # a one-character code span must not break the pairing
        ("`./src/app.ts`", True),
        ("`~/proj/app.ts`", False),
        ("`clip-01.mp4`", False),
        ("`anthropics/claude-code`", False),
        ("no backticks: episode/clip-01.mp4", False),
    ]
    for text, want in cases:
        got = bool(short_paths(text))
        assert got == want, (text, got, want)
    body = b'{"last_assistant_message": "`episode/a.md`"}'
    for prefix in (b"", b"\xef\xbb\xbf", b"\xef\xbb\xbf\xef\xbb\xbf"):  # none, one BOM, two BOMs
        assert decision(load(prefix + body)), prefix
    assert decision({"stop_hook_active": True, "last_assistant_message": "`a/b.ts`"}) is None
    six = " ".join(f"`episode/f{i}.md`" for i in range(6))
    assert "(+4 more)" in decision({"last_assistant_message": six})["reason"]
    print("all self-tests passed")


def main():
    if "--selftest" in sys.argv:
        return selftest()
    try:
        out = decision(load(sys.stdin.buffer.read()))
    except json.JSONDecodeError:
        return
    if out:
        print(json.dumps(out))


if __name__ == "__main__":
    main()

It flags a path only when it sits inside backticks, contains a / or a \, and ends in a file extension. Anything already absolute is skipped: C:\ or C:/, a leading / or \, ~, or a URL. These are the self-test's own cases:

Reply contains	Result
`episode/clip-01.mp4`	flagged
`episode\clip-01.mp4`	flagged
`./src/app.ts`	flagged
`src/{a,b}.ts`	flagged
`/home/dev/proj/episode/clip-01.mp4`	left alone: absolute
`C:\proj\episode\clip-01.mp4`	left alone: absolute
`~/proj/app.ts`	left alone: absolute (home)
`clip-01.mp4`	left alone: a bare file name
`anthropics/claude-code`	left alone: missing extension
episode/clip-01.mp4 without backticks	left alone: outside backticks

The message is built in decision(). It names the first two short paths and counts the rest as (+N more).

python stop_hook_full_path.py --selftest runs the built-in cases and prints all self-tests passed.

Adding it to Claude Code

Hooks go in a Claude Code settings file. Which one depends on how far the hook should reach:

File	Applies to
`~/.claude/settings.json`	every project on your machine
`.claude/settings.local.json`	one project, and is meant to stay out of git
`.claude/settings.json`	everyone who uses the repository, so use it only if the script is in the repository too

If the file already has a hooks object, merge the Stop entry into it. On Windows:

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python \"C:/Users/you/hooks/stop_hook_full_path.py\"",
            "timeout": 15
          }
        ]
      }
    ]
  }
}

On Linux:

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python3 /home/you/hooks/stop_hook_full_path.py",
            "timeout": 15
          }
        ]
      }
    ]
  }
}

Both paths are examples. Use the absolute path to wherever you saved the file, and make sure python (or python3) is on the PATH of the shell Claude Code runs it in. On Windows that is Git Bash, or PowerShell if Git Bash is not installed. timeout is in seconds. Type /hooks in Claude Code to see the hook listed. The hooks reference says edits to settings files are normally picked up automatically.

Mistake 1: a byte order mark made the hook exit without an error

The check at that point was a short script. It already used decision and reason, because the field mistake below came first:

import json, re, sys

raw = sys.stdin.read()
try:
    data = json.loads(raw)
except Exception:
    sys.exit(0)

msg = data["last_assistant_message"]
bad = re.findall(r"`((?![A-Za-z]:)[\w.-]+\\[^`]+)`", msg)
if bad:
    print(json.dumps({"decision": "block",
                      "reason": f"Use absolute paths, not: {bad[0]}"}))

Fed the same payload from two shells, it behaved two ways in the video. In Bash, python check_rules_v1.py < payload.json printed the block decision. In Windows PowerShell 5.1, Get-Content payload.json | python check_rules_v1.py printed nothing and reported no error. That depends on the console: on a default console with input code page 850, I got the block decision from the same command. The two settings that decide it are below. check_rules_v1.py, peek.py (used below) and payload.json, the sample hook input, are in the repo's demo-bom folder.

The cause is in the first bytes Python reads. A byte order mark is the three bytes EF BB BF that some tools put at the very start of UTF-8 text, and json.loads raises on one. When PowerShell pipes payload.json to Python it can add one. My except Exception: sys.exit(0) swallowed the error, so the hook exited quietly and blocked nothing. That looks the same as a hook with nothing to flag.

The fix reads bytes, decodes them as UTF-8, and strips every leading mark before parsing. It strips every mark, not one, because the pipe can add two (the list below shows when):

raw = sys.stdin.buffer.read().decode("utf-8").lstrip("\ufeff")

Two settings decide how many marks Python receives. $OutputEncoding is the encoding PowerShell uses when it pipes data into a program. The console's input code page is the numbered character set a Windows console uses for the text it reads and writes. A second script, peek.py, prints the first bytes Python receives as hex, and 7B 22 is {", the start of the file. With both settings on UTF-8, Python received EF BB BF EF BB BF 7B 22 68: two marks, and the check printed nothing.

First nine bytes Python receives, by console input code page and $OutputEncoding:

850   / default   7B 22 68 6F 6F 6B 5F 65 76   v1 blocks
850   / UTF-8     EF BB BF 7B 22 68 6F 6F 6B   v1 silent (one byte order mark)
65001 / default   EF BB BF 7B 22 68 6F 6F 6B   v1 silent (one byte order mark)
65001 / UTF-8     EF BB BF EF BB BF 7B 22 68   v1 silent (two byte order marks)

Measured with Get-Content payload.json | python peek.py. EF BB BF is a UTF-8 byte order mark. 7B 22 is {".

I measured this in non-interactive powershell.exe -NoProfile -File runs on Windows PowerShell 5.1, setting [Console]::InputEncoding inside the script. I did not test PowerShell 7. Run peek.py against your own pipe before you trust a hook there.

I tested this by feeding the hook from PowerShell myself. I have not measured what Claude Code itself writes to a hook's stdin, so the hook strips the mark either way.

Reading text with sys.stdin.read() is not enough on Windows. PYTHONIOENCODING is an environment variable that overrides the encoding Python uses for stdin and stdout. When it is not set, Python 3.11 decodes a pipe with the system code page (cp1252 on my machine), and there the mark arrives as three other characters, so lstrip finds nothing. Setting PYTHONUTF8=1, Python's UTF-8 mode, changes that as well.

I had that wrong at first. The demo's first fix used sys.stdin.read(). It passed in a session where PYTHONIOENCODING was set to UTF-8 and failed in a clean one. The repo hook reads bytes the same way, but its first copy used Python's utf-8-sig codec, which removes one mark, so the two-mark case still failed the same silent way. The self-test now sends none, one and two.

Mistake 2: the field name

Before that, my very first version printed systemMessage:

{"systemMessage": "Use absolute paths, not: scripts\\check.py"}

The hooks reference describes systemMessage as a warning message shown to the user. Claude did not react to it, so nothing changed. The change that worked was two lines:

-    print(json.dumps({"systemMessage":
-                      f"Use absolute paths, not: {bad[0]}"}))
+    print(json.dumps({"decision": "block",
+                      "reason": f"Use absolute paths, not: {bad[0]}"}))

I did not keep that first version. The diff is a reconstruction of it, made for the video.

Mistake 3: a check that rejected braces

One reply used brace shorthand: social\clip-0{1,2}-9x16.mp4 means social\clip-01-9x16.mp4 and social\clip-02-9x16.mp4 in one name. My old pattern for a path segment covered letters, digits, dots and dashes. I did not keep it, so this one-line check does the same job, and a brace matches none of it:

python -c "import re; print(re.match(r'^[\w.\-]+$', 'clip-0{1,2}-9x16.mp4'))"
None

The name failed the pattern, so the check ignored the reply. The pattern in the repo's hook allows braces and commas, ^[\w.\-{}, ]+$, and the self-test includes src/{a,b}.ts. In my own hook, looks_like_relative_path returned True for the same name after the change.

Where it stops

It reads the last reply only. It sees paths inside backticks and nothing else, so a path Claude writes without backticks gets through. A path with a line number, src/app.ts:12, is not flagged. A code span such as python/3.11 is flagged although it is not a path. It has run on one Windows machine and in one Linux container, and not on macOS. The byte order mark behaviour above is what I measured on that one machine.

The code and the video

The hook, its self-test, both byte order mark scripts and the Linux demo are in claude-code-stop-hook-full-path, under the MIT licence. The video walks through the same hook: https://youtu.be/LD_rZm5MwWs.

This is an independent project, not affiliated with or endorsed by Anthropic.

The verification tax

the kilted dev — Sat, 03 Oct 2026 12:28:00 +0000

A session narrating its own work inline, no delegation, spent 78% of its context window across
two long stretches with almost no back-and-forth from whoever had asked for it. A separate
session, same shape, burned over a million tokens in one pass. Neither had done anything wrong.
Both had already made the right call on which model to use. They just ran the work themselves
instead of handing it off, and talked through every step on the way.

The fix looked obvious once it had a name: a task can be the right tier and still be the wrong
thing to run inline. Long, low-interaction work burns the same resource, the controlling
session's own context window, no matter which model is doing it or what that model costs.
Background it. Default to a brief checkpoint instead of a running commentary. Cheap rule, easy to
state, and it should have been the end of it.

Testing it on something real

The next real chance to watch the rule work was an actual five-task build: a search feature
threaded through several existing functions, tested live in a browser as each piece landed. Every
task went to a subagent with a full brief and a strict report contract: done, done with concerns,
blocked, pick one, or put the detail in a file rather than the reply. Exactly what the rule asked
for. Implementation never ran inline.

The controlling session's own context climbed anyway. Not dramatically, four points over about
three hours, but steadily, the whole time, driven by the same thing the rule was written to stop:
long stretches of low-interaction activity, just moved to a different part of the job.

What moved into the gap

Dispatching the implementation only covers what happens inside the dispatch. It does nothing for
two costs that stay with the controller regardless of how well the dispatch is written.

One is the brief itself. A good dispatch spec spells out its constraints, the known gotchas, the
exact test sequences, on purpose, so the thing doing the work doesn't have to guess. That text
gets paid for twice: once to write, once for the controller to carry in its own history after
sending it.

The other is the check afterward. The controlling session read every report against the actual
diff, and independently re-tested several claims rather than taking them on trust. None of that
was delegated. It moved from doing the work to checking it, a category of cost the original rule
never named.

Phase	What it costs
The brief	Constraints, known gotchas, exact test sequences. The controller writes it, then carries it for the rest of the run.
The work	One subagent per task, each with a window of its own. The only phase the rule ever reached.
The check	Every report read against the diff, several claims re-run. All of it back on the controller.

Three phases in the build. Only the middle one ever left the controlling session.

The check wasn't wasted

If the fix had been "verify less," the numbers would look better and the build would be worse. The
verification pass caught three real problems in that one run. Twice, a report said something
hadn't worked when the code was fine and the test had failed for an unrelated reason. Once, a
report went quiet about a step it had skipped rather than saying so. None of the three was an
actual defect in the code. Someone caught all three only by checking instead of trusting the
report, and every one of them came out of work done by a cheaper, less careful process than the
one doing the checking.

That's the tax nobody had priced yet: route the doing to something cheaper, and the checking gets more necessary, not less.

And the checking is exactly the kind of long, low-interaction work the rule already knew how to
handle. Nobody had told it to.

A quieter comparison, running at the same time

Two other sessions were doing unrelated work in the same window, and both had something the build
above didn't have: a real stopping point built into the task itself, an approval gate in one case,
a direct question from whoever had started the session in the other. Neither ran anywhere near as
long unsupervised.

Both still climbed. Less than the fully autonomous build, meaningfully less, but nowhere near the
order of magnitude a "no human in the loop" story would predict. A handful of points of the same
resource, spent the same way, just less of it. Whatever drives the growth isn't only whether a
person was watching. It looks closer to how much a session writes and reads on its own, per turn,
regardless of who's nominally there.

Session	Share of context held by message history
Autonomous build	39.2% (climbed from 35.3%)
Gated, direct question	30.1%
Gated, approval	26.4%

What changed, and what didn't

The rule that said background long inline work now covers long inline checking too: re-reading a
report, or re-running a live test to confirm a claim independently. Same resource, same fix, just
applied to a phase the rule hadn't reached yet.

It doesn't make the checking optional. The three catches in the test run are the argument against
that shortcut, not for it. The actual fix isn't spending less attention on verification. It's
spending that attention somewhere that doesn't cost the one session that has to stay coherent for
the whole build.

Originally published at thekilted.dev/the-verification-tax.

Still carrying the diagnosis

the kilted dev — Tue, 29 Sep 2026 06:39:24 +0000

Field notes from a local-model pipeline. A coding harness that had already proved itself went
silent for an hour, and its replacement shipped the same session. The write-up explaining why
needed a correction of its own.

The plan was to have a local model write six Python modules of a real application, one at a time,
each one gated on a test suite it had to make pass. The harness for that job was already chosen,
already benchmarked, and already recorded as working.

It produced nothing. Not bad code, no code. Zero files, for about an hour.

What happened next is worth reading less for the fix, which took a session, than for what happened
to the explanation afterward. The root cause was written down and spread into five documents, then
disproved four days later by a controlled retest run from a different project. And the correction
that replaced it, checked now against a benchmark run two weeks earlier, does not hold up against
that evidence either.

The benchmark behind it

The tool was little-coder, a small command-line coding agent tuned for small models, a thin
wrapper over the pi agent framework, installed from npm, pointed at a local model server.

It hadn't been picked casually. On 4 July it ran a build-a-word-search-game benchmark on this
machine and scored 8 out of 8, with a build time of 2.1 minutes, including a follow-up fix that
didn't regress anything already working. Three files written to disk by the model's own write
tool. Criterion one of that benchmark is literally "3 files saved to disk," and it passed.

That run used little-coder v1.9.11 against a 30-billion-parameter mixture-of-experts coding
model, locally aliased qwen3-coder-30b. That pairing matters later.

The session built a launcher script around the winning recipe so no future session would have to
rediscover the setup, including a prompt rule found the hard way: the word "tool" can't appear
anywhere in the prompt, because even "command-line tool" trips a formatting failure in the model's
output. That's the level of tuning already banked before any of this started.

Then nothing, silently

The next day, given the six modules to delegate, the harness failed completely.

The shape of the failure is the part worth sitting with. The command exited with status 0. It
printed the model's reply. It wrote no file. The reply was raw <function=write> XML, the model
describing a write in text form instead of the harness receiving a structured call it could
actually execute. Nothing parsed it, and nothing complained.

Exit 0 with no output written is the worst failure mode a delegated task can have, because every
cheap way of checking says it worked.

A caller reading the exit code sees success. A caller reading stdout sees a confident, well-formed
response. Only a caller that goes and looks at the filesystem sees the truth.

The first attempt to get modules out of it was a cheap orchestrator agent, given room to fix the
harness itself. It ran for roughly an hour and about fifty tool calls, delivered zero modules, and
along the way took an action nobody had sanctioned: a global npm install -g that upgraded
little-coder from 1.9.11 to 1.9.13, mutating machine-wide tooling mid-run. That upgrade is the
one variable that changed between the benchmark that worked and the delegation that didn't.

The failure was then reproduced directly, by hand, rather than trusted from the agent's own
report. That's the only reason any of what follows is checkable at all.

Depending on less

Rather than downgrade or re-pin a command-line tool that had already proved it could change
underneath a running project, the session wrote a replacement in the same sitting:
delegate-via-rest.py.

Its design is the whole lesson. The model still writes all the code. What the driver removes is
the model's need to call a tool at all. It sends the specification and the test file to the local
server's chat endpoint, takes the reply, pulls out the largest fenced code block, writes that block
to disk itself, runs pytest, and feeds any failure text back for another round. It talks only to
127.0.0.1.

That takes the entire tool-calling scaffold, the layer that had just failed, out of the critical
path. Writing a file is something the harness can do reliably. Asking a model to emit a structured
call that a third-party parser has to recognise depends on two things nobody in this pipeline
controls. The driver keeps the model doing the one part only it can do, and does the mechanical
part itself.

Rounds ran 8 to 30 seconds on this hardware. Across the six modules, three passed their tests on
the first attempt. The other three converged within a few rounds each, usually by sharpening the
specification rather than coaxing the model. The finished application ran to 34 commits and 52
passing tests.

Two features were added later, and each one encodes a failure that had already happened. A
--selfcheck preflight runs a trivial task end to end through the real endpoint before any real
dispatch, because a "verified working" harness had already broken silently between sessions. A
--probe flag runs a held-out adversarial test file after the acceptance tests go green, and its
output is deliberately never shown to the model, because feeding it back would let the model
overfit the probe the same way weak acceptance tests had already overfit twice. The driver also
carries four distinct exit codes instead of a plain pass or fail, including one specifically for
tests green but the held-out probe failed. After a silent exit-0 incident, none of that reads as
over-engineering.

The cause, as recorded

The explanation written down was that version 1.9.13 no longer recognises the model's id, so it
never attaches the tool-calling scaffold, so the model falls back to emitting raw XML.

There was real evidence behind it. When the run failed, the harness printed a warning that the
model wasn't found for the provider and it was falling back to a custom model id. A warning about
the model id, sitting right next to a failure about the model not being wired up correctly, in a
run whose only recent change was a version bump. It read as confirmation.

That explanation made it into the trial write-up, the launcher's own header, the project's
constraints file, its decisions log, and its tool registry.

The retest

Four days later, a different project on the same machine retested the launcher as part of an
unrelated tool inventory, and reproduced the failure in 46 seconds. Rather than stop at
reproduction, it ran controls.

Path	Result
Local server `/v1` directly, with tools, non-streaming	tool calls parsed correctly
Local server `/v1` directly, with tools, streaming	tool calls parsed correctly
Local server `/v1` directly, no tools	prose, no XML leak
`little-coder` + `qwen3-coder-30b`	raw XML, no file written
`little-coder` + `qwen3.5`	writes the file
`little-coder` + `qwen3.5:latest`, an unregistered id	writes the file, warning fires

That last row is the control, and it's the one that actually settles the question. It holds the
model constant and only varies whether the id is registered, and the file gets written anyway even
though the warning still fires.

So the warning was benign, and id resolution wasn't the cause of anything. The original explanation
had inferred causation from adjacency: someone saw a warning next to a failure and wrote down a
mechanism to connect them.

The retest killed two of its own hypotheses too, which is what separates a control from a
demonstration. It expected the harness might be omitting the tools array from the request.
Refuted, since a different model tool-calls fine through that same adapter. It expected the
unregistered id to disable native tool calling outright. Refuted by the control row itself.

The correction went back into the record as a marked block, not a silent edit. The wrong claim is
still sitting there, readable, right next to the right one.

The correction was a claim too

The retest's own write-up had been careful. It reported exactly what it varied, and said plainly
that it hadn't resolved why the harness's request shape defeats that particular model's parser, and
hadn't chased it any further.

What travelled outward from it was shorter: not the id, the model.

That compression is checkable, and it doesn't hold. If the fault were the model itself, the same
model through the same harness would not have scored 8 out of 8 two weeks earlier, writing three
files to disk cleanly. Same box, same model server, same alias, same command-line tool. The one
thing that changed between the run that worked and the run that no-ops is the version bump the
orchestrator performed without asking: 1.9.11 against 1.9.13.

The retest ran entirely on 1.9.13. It never varied the version, because the version wasn't the axis
it set out to test. It was testing the id-resolution claim, and on that axis its own work holds up
fine. What the evidence actually supports is narrower and less quotable than either slogan: on
1.9.13, this specific model and this specific harness don't work together, while the same harness
works fine with a different model and the same model works fine without that harness. It's an
interaction. The mechanism behind it is still unknown.

"The harness broke" was too broad a claim. "The model is at fault" replaced it with a different
claim of the same shape, reached the same way: by reading a summary instead of the run underneath
it. The 8-out-of-8 record was sitting the whole time in a benchmarks file the correction never had
a reason to open.

Two places the correction didn't reach

The correction was folded into four files. The message carrying it listed five places the wrong
claim lived, and one of those was a session log left alone on purpose, because append-only history
shouldn't get rewritten after the fact.

Nine days later, a check of the record for this piece found the refuted mechanism still
stated as plain fact in two places nobody had listed.

The trial write-up still said the harness "no longer recognises the model id... fails to attach
the tool-calling scaffold." And the driver's own docstring, the first thing anyone reads before
using it, still explained its own existence as "after a version bump it stops recognising the
model id."

The tool built because of the diagnosis was still carrying the diagnosis, after the diagnosis
itself had been withdrawn.

Not because anyone ignored the correction. Because propagating a correction requires knowing every
place the original claim was written, and that list had been made from memory instead of from a
search. Both have since been fixed, the docstring in place, the trial write-up with a dated
correction block that leaves the original wording readable, because a results file is a record of
what was believed at the time, not something to quietly rewrite.

The real question isn't how two copies were missed. It's why anyone thought four was the complete
set. A claim written into five documents has usually been written into more than five, and the
cheap check, searching for the sentence instead of trying to recall where it went, was sitting
there the whole time, unused.

What actually carries over

The smallest surface wins twice here, not once. The driver works because it asks the model for
text and writes the file itself, instead of trusting a structured call to survive a parser, a
wrapper, and a version bump all at once. The same instinct applies to how the record of what
happened gets kept. Exit 0 with no artefact is the failure mode worth designing against, which
means checking for the file rather than the status code. Pinning the versions of any tool a
pipeline depends on matters for the same reason, and no agent should be installing anything
globally on its own initiative. A correction is a new claim, not a return to neutral, and it earns
the same scrutiny the original claim never got: a real search for every copy of what it's
replacing, not a list made from memory.

Originally published at thekilted.dev/carried-the-diagnosis.

Never installable

the kilted dev — Sat, 19 Sep 2026 07:51:56 +0000

For three days of real-device testing, the app worked, or seemed to. Every test passed. Every
review came back clean. On a real phone, it had never once been installable, and nothing had ever
said so.

This is a proof-of-concept, not an NHS product. It was built by one technical person working
alongside an NHS dietetic team. No patient data exists anywhere in it. Nothing leaves the
device unless a lookup runs, and even then it sends nothing but the barcode number itself. Clients
had been sending in photos and rough text descriptions of what they'd eaten, and a dietitian was
manually reading that, finding the matching product, and keying it into Nutritics, the accredited
nutrition-analysis software the team already used. Nutritics has its own patient-facing app, Libro,
that would let clients log the food themselves and skip all of that. Libro just isn't enabled for
this region, and turning it on is weeks away at best, if it happens at all. So the trial runs a
smaller experiment instead: making the manual keying-in as fast and unambiguous as it can be, and
doubling as evidence for whether Libro is worth switching on.

That's the frame. The build itself comes down to two decisions that held, and a run of confidently
wrong guesses before the real one.

Why it never asserts a calorie figure

The first thing tried was the obvious move: point a consumer nutrition app at a barcode and let it
fill in the numbers. Cronometer's first four barcode and photo lookups all came back wrong.
Correcting them by hand was painful enough that the question settled itself. MyFitnessPal has kept
barcode scanning behind a Premium paywall since October 2022, so the one app most clients already
had was already a dead end. Only Open Food Facts, a free, open, no-account database of product
names, gave anything back reliably, and even that only names a product. It doesn't know its
nutrition.

Reading digits off a barcode is deterministic and checkable, and that was never where Cronometer
failed. It failed at mapping those digits to the right nutrition record through a crowdsourced
database, and that's the exact spot it got wrong four times running. So the tool draws a hard line
at that exact seam. It reads barcodes. It never writes a nutrition figure. Nutritics, with roughly
1.6 million EUROFIR-accredited foods, stays the only source of truth for what anything actually
contains.

If the tool never claims a calorie number, it can never be the reason someone has to unpick a
wrong one later.

The same discipline runs through barcode validation. A GS1 check digit rejected every single-digit
misread thrown at it in testing, 351 out of 351, which covers the failure mode that actually
happens when a camera misreads a smudged label. But roughly one in ten arbitrary thirteen-digit
numbers will satisfy that check by pure chance, tested against a hundred thousand random numbers,
so "the check digit agrees" only ever means the check digit agrees, not that the barcode is
definitely the right product. An earlier draft of the docs called barcodes "self-verifying." The
project corrected that before shipping, because it was a stronger claim than the tool could
actually back.

Thirty-five foods, not a hundred and twenty

The one insight that shaped everything downstream came from watching how Nutritics itself behaves,
not from the client side at all. Its diet-log workspace works like a spreadsheet, and once a
dietitian has searched for a food and dropped it into one day's log, that same food can be added to
any other day or meal without searching again.

A week's diary might run to something like a hundred and twenty logged eating occasions. Searched
food by food, that's a hundred and twenty searches. But a person doesn't eat a hundred and twenty
different things in a week. The same handful of items repeat across breakfasts and snacks. Counted
by distinct food instead of by occasion, the same week might only need thirty-five searches. That's
the number the design actually optimises, and it's why the data model splits in two: a Food
Library, where every distinct food gets listed once against a short code, and a Daily Log, where
each eating occasion just references a code and a quantity. A dietitian keying a week's diary isn't
doing a hundred and twenty lookups anymore. They're doing thirty-five, and then a lot of quick
copy-paste.

It's a small reframe with a real effect on the person actually doing the keying, and it came from
reading how the existing software worked rather than guessing at what a food diary "should" look
like.

The bugs only a real phone could find

None of this survived contact with a phone unscathed. The front end shipped as a vanilla-JS
progressive web app, deliberately with no build toolchain (this was meant to be a short-lived trial
workaround, not a product, and a bundler is more to maintain than that deserves). Every change
afterward went through a fresh implementer and an independent reviewer before merging, and that
review step earned its keep almost every round.

The export path took the worst of it. The first version silently did nothing when someone tapped
"Download" on an Android phone in DuckDuckGo, because a CDN script, xlsx.full.min.js, that the
service worker hadn't cached could fail to load without throwing anything visible. The fix that followed introduced its own bug:
the downloaded file came back as .bin instead of .xlsx, because the fallback download mechanism
wasn't reliably setting the file's MIME type on that browser. Fixed by building the download from
an explicit Blob instead. Then a third bug, sneakier than either: the app would report "Downloaded"
with nothing actually appearing, because the click handler waited on three IndexedDB round-trips
before triggering the download, and that wait was long enough for the phone to quietly expire the
tap's permission to trigger one at all. Nothing threw an error, no dialog appeared, and the app
still reported "Downloaded" as if it had. The fix was to pre-build the export the moment the screen
opened, so the tap had almost nothing left to wait on.

A fourth export bug turned out not to be this app's to fix. Sharing the file to WhatsApp on Chrome
Android threw a permission error even though the browser's own API had said sharing was possible.
An A/B test settled why: a plain text file shared fine, but the identical share, swapped to the
real xlsx MIME type, failed every time. That's Android Chrome's own share implementation rejecting
the file type at the native layer, and the fix is a fallback, not a workaround: catch that
specific failure and drop back to the working download path.

Of the four export bugs, all found on a real phone, two gave nothing to go on (silent no-op,
nothing downloaded) and were this app's own to fix. The other two threw a visible error, and only
one of those — Android Chrome refusing the file type — turned out not to be this app's bug at all.

Two files, sixty-eight bytes each

The bug that cost the most time wasn't in any of that. It was that the app was never actually
installable, on any browser, on any host, the entire time.

The manifest declared full-size icons, but the two actual icon files behind it were 68-byte
placeholder stubs, one pixel by one pixel. Chrome checks that a PWA's declared icons actually exist
at the sizes it claims. When they don't, nothing in the console warns about it, no error ever
fires, and the "Install app" option just never appears. "Add to Home Screen" still worked, but it
could only ever produce a bookmark shortcut, not a real installed app, so offline never worked
either. The service worker, the manifest, and the caching logic underneath all of it had been
correct from the start.

Before anyone found the icons, two other theories took the blame instead. First the browser: maybe
DuckDuckGo's PWA support was incomplete. Then the host: maybe something about how Namecheap served
the files was the difference from the version that had worked fine on Netlify. Both theories were
plausible. Both were wrong. An earlier comparison between the two hosts had already been written up
as "decisive, host-specific." The actual difference was that Netlify's service-worker cache
had been warm for months and Namecheap's was cold, and that warm cache had quietly been hiding two
unrelated bugs the whole time. What actually surfaced all three bugs at once was a fresh cache forcing every code
path to run for real, on a real phone, for the first time.

A wee sixty-eight-byte file doesn't announce itself as the reason anything is broken. It just
sits there, technically present, while every larger theory takes its turn and falls apart first.

Light, not orientation

One more misdiagnosis is worth keeping, mostly for how the test caught it. A habit had formed of
always photographing barcodes horizontally, and it looked like it mattered: a vertically held
barcode seemed to fail more often. The obvious explanation was that the scanner's decoder reads
horizontal scan lines and could miss a rotated barcode outright.

A direct test said otherwise. A barcode held deliberately vertical, in good light, scanned exactly
as well as a horizontal one. Orientation wasn't the variable. Light was, and the horizontal-photo
habit had just happened to correlate with better lighting, not cause anything on its own. The same
test also cleared the gallery-scan refactor, a different piece of code that had carried an
"unverified on real hardware" flag since it was written. Whatever flakiness it had been suspected
of causing turned out to have nothing to do with it.

What's still open

None of this is a finished product, and it isn't meant to be one. The whole tool is a workaround
for not having Libro, and the plan from day one was to retire it without regret the moment Libro
gets switched on for this region, rather than let it grow into something that has to be maintained.
If the manual keying turns out to be genuinely painful even with a well-built sheet in front of the
dietitian, that's the evidence the trial was actually built to produce.

Either way, what's left isn't a product. It's a barcode reader that only ever claims what it's sure
of, and a debugging log full of things that looked complicated right up until someone found the
actual cause.

Originally published at thekilted.dev/never-installable.

The benchmark that disproved its own result

the kilted dev — Thu, 10 Sep 2026 16:00:40 +0000

An article came past claiming a two-year-old coding-trained model beats a newer general model at
editing structured text that isn't code. Its thesis, verbatim:

A general chat model treats rules as more of a suggestion because it's optimized to sound
helpful. A coding model treats rules as the whole point.

The article contained no measurements. Not weak measurements. None.

That's normally enough to close a review. This project turned it into a test instead, because the
claim has a property most of the genre lacks: it's narrow, it names a mechanism, and it's
falsifiable on hardware already sitting here. "Local models are better than you think" isn't
testable. "A coding-trained model follows structural rules more reliably than a general one" is a
Tuesday afternoon's work.

The test took a few hours. The result was clean, confirmed the article, and was wrong.

The setup

Two models already pulled, no new downloads needed.

qwen3-coder-30b — the coding-trained side
qwen3:8b — the general side

The general side was chosen after checking its chat template directly through the runtime's own
API: a 1,723-character real ChatML template. That check mattered because another model on this box
ships a template that's just a passthrough, which silently degrades everything sent through it, and
picking that one by accident would have produced a large, fake, extremely publishable result.

Five fixtures, one per task type: JSON repair, YAML frontmatter, CSV cleanup, Docker log
extraction, markdown tables.

These models are stochastic, and one run is an anecdote.

So every fixture ran three times, thirty runs in total.

Every fixture gets an objective gate that is full-equality or parse-and-compare, never substring
matching. That distinction isn't pedantry. A substring check asking "does the output
contain the right value" passes an output that contains the right value and three wrong ones too.
This project has had weak tests pass broken local-model work before, twice, so the checkers are
written to be hostile.

Two deliberate choices worth naming. Thinking mode off, because on this box's qwen models it
either hides the answer inside a thinking block or gets rejected outright by the server. Default
temperature rather than the low temperature used for code generation, because the claim is about
how these models behave in ordinary use, and tuning the sampling toward precision would have
quietly answered a different question.

The grader had a bug, and it found it before any model ran

The checkers are the ground truth in an evaluation like this. If the grader's wrong, every number
downstream is decoration. So structured-text-eval.py has a --selftest mode: known-correct answers fed to
each checker, with no model involved, before a single request goes out.

It failed on the YAML fixture, for a reason that would never have been noticed later.

The fixture asks a model to add one key to a YAML frontmatter block without disturbing anything
else, and the checker verified the untouched keys still matched, including a date field,
2026-07-01. YAML's parser doesn't return that as a string. It returns a datetime.date object,
because unquoted ISO dates are a date type in YAML. Comparing that object to the string
"2026-07-01" is always false.

The checker would have failed a perfectly correct answer, on every trial, for both models. The fix
is the str() in the line that guards those untouched keys, still sitting there today:

if fm.get("title") != "Weekly Report" or str(fm.get("date")) != "2026-07-01" \
        or fm.get("tags") != ["ops", "weekly"]:

What matters is when it was caught: before the run, by a check written specifically to distrust the
grading logic, rather than after, when a mysterious YAML failure would have needed explaining and
might have been written up as a finding instead.

The grading logic needs the same adversarial discipline as the thing it grades.

It's code, and nobody reviews it, because it lives in the part of the experiment everyone treats as
infrastructure.

We have written about checks like that before. That check could only
ever come back green, whatever it was pointed at. This one had the same defect running the other
way, and would only ever have come back red.

The first result confirmed the article

Thirty runs, and the coding model came out ahead.

Fixture	qwen3-coder-30b	qwen3:8b
JSON repair	pass	pass
YAML frontmatter	pass	pass
CSV cleanup	pass	pass
Docker log extraction	pass	fail
markdown tables	pass	pass

Every fixture, first run, three trials each (a mark stands for all three — every fixture's trials agreed). Four fixtures agree completely. One doesn't.

That fixture was extracting every ERROR line from a Docker log into structured JSON. The coding
model scored 3 out of 3 there, the general model 0 out of 3.

Look at the shape of that. It isn't noisy. It isn't one unlucky trial. One model succeeded every
time and the other failed every time, on the fixture most about following a structural rule, which
is exactly the mechanism the article named. A clean, consistent, mechanism-matching split, in the
predicted direction.

That's what a real finding looks like. It's also what this one looked like, and it was a bug in
the fixture itself.

What the raw output actually said

The harness records every output verbatim rather than just pass or fail, so the failures were
readable, not just countable.

The general model hadn't mangled the JSON. It hadn't missed lines, invented fields, or drifted
from the schema. Across all three trials it had produced valid, well-formed, correctly structured
output, and put the log-level word inside the message field:

"message": "ERROR Failed to connect to cache: timeout after 5s"

The coding model had stripped it. The expected answer stripped it. So the checker failed the
general model, correctly, against a specification the prompt never actually stated.

The prompt said to extract each ERROR line into {timestamp, service, message}. It never said
whether the word "ERROR" belongs in the message. One model read it one way, the other read it the
other way, and both readings are defensible: the line begins with the level, and whether that
level is part of the message or metadata about it is a genuine ambiguity, not a comprehension
failure.

Reword the prompt, adding one clause: "the message field must contain ONLY the text after the
ERROR level word, not the word ERROR itself." Rerun all thirty.

Fixture	qwen3-coder-30b	qwen3:8b
JSON repair	pass	pass
YAML frontmatter	pass	pass
CSV cleanup	pass	pass
Docker log extraction	pass	pass
markdown tables	pass	pass

The same grid, once the prompt said which part of the line the message excludes. The one cell that failed is the only one that changed.

The gap didn't shrink. It vanished.

The uncomfortable part

The result that ended up under scrutiny was the one that confirmed the hypothesis.

That's backwards from how it usually goes. A disconfirming result gets picked apart, because it's
annoying and someone wants it to be a bug. A confirming result gets written up, because it agrees
with what you already believed and there's nothing left to explain.

Here, the confirming result was the buggy one. The rigour that caught it wasn't scepticism about
the article. Its claim was taken seriously enough to test properly. It was a checklist habit: read
the raw output behind a pass rate before reporting it, regardless of which way the result points.
Without that habit, the write-up would have been "tested it, and the article is right," with a
clean table underneath and no reason for anyone to look again.

The null result is also weaker than it sounds, worth stating plainly. It doesn't show the
article is wrong. The article's actual comparison was a 3B coding model against a much larger
general one, a full generation apart and roughly three times the size difference. Both models
tested here are modern and well above the size where basic structure-following stops being in
question. The claim may be entirely real at the article's scale and simply invisible at this one.
That wasn't tested, because the relevant general model doesn't fit this box's memory budget. So
the finding is narrow: on these two models, on these five task types, with unambiguous
instructions, there's no difference. Nothing about standing practice changed.

One more thing that did not survive

Preparing this write-up meant going back to the raw output file. The committed file holds thirty
results, all passing. The run that produced the twelve-out-of-fifteen split, the one this entire
piece is about, no longer exists: the corrected fixtures were rerun into the same output path and
overwrote it.

The specific output quoted above survives because it was copied into the results log the same
day, by the session that had it on screen. That's a good contemporaneous record. It isn't the
artefact.

Worth noticing which run it was. What didn't survive was the disconfirming evidence: the thing
that was wrong, the run that failed, the state before the correction. Nobody deleted
anything on purpose. The rerun simply wrote to the same filename, because the interesting output
was assumed to be the final one.

The final output is the one you'll never need. It's the one that agrees with the conclusion you
already wrote down. The run worth keeping is the one that made you look twice.

Originally published at thekilted.dev/the-benchmark-that-disproved-itself.

The file nobody reread

the kilted dev — Sat, 05 Sep 2026 18:00:47 +0000

Field notes from a hosting migration for an NHS food-diary proof of concept. A certificate checker
was quietly lying about whether a fix had worked, and two sessions rejected the same hosting idea
twice, four days apart, without either reading the other's reasoning. The move itself was still on
hold when this was written.

This is a proof-of-concept, not an NHS product, built by one technical person working alongside an
NHS dietetic team. No patient data exists anywhere in it. The app stores everything on the
client device and nothing else. The trigger for all of this was mundane: Netlify's free tier caps
a project at 15 credits per production deploy, 300 a month, which works out to roughly twenty
deploys before the meter runs dry. The project was already rationing pushes against that ceiling,
and shared hosting on Namecheap, already being set up for another project the same person runs,
looked like a way out.

Two rejections, four days apart

GitHub Pages came up early, and the project turned it down fast. On the free plan it only serves
public repositories. A private repo needs a paid GitHub plan, and even then the site still
publishes to a public URL with no login wall short of Enterprise. Netlify's free tier deploys from
a private repo to an unlisted one instead, which fit the stated preference to keep the code itself
private. No patient data was ever going to sit in that repo either way. The preference was about
the source, not about anything a patient had typed.

Four days later, GitHub Pages came up again. Not from a different person with a different view. A
coding session working on the project proposed the exact idea an earlier session had already
rejected, because nobody had reread the decisions file before speaking. The occasion was almost
incidental: a different, newer domain had surfaced, already live on GitHub Pages with a working
certificate, and for a moment it looked like an obvious destination. That session looked into it,
then declined it again, for the same reason as the first time. That reason had not moved. It had
simply never been reread. This round did turn up one genuinely new blocker: the food-diary repo
and the newer domain lived under two different GitHub accounts, and GitHub blocks a custom domain
from being claimed by a second account once it's verified on the first, specifically to stop that
kind of takeover. A real finding, arrived at by accident, in the course of re-litigating a question
that had already been settled.

A decisions log nobody rereads is decoration. It was there, it had the right answer sitting in it,
and a session still asked the question again anyway.

A certificate checker that had been lying

The SSL side had its own real bug and its own fake one. The real bug: the domain's nameservers
were pointed at Namecheap's basic DNS, which only handles URL forwarding, not actual hosting.
Switched to the proper hosting DNS and that half resolved. An earlier assumption, picked up from a
web search during planning, claimed the hosting tier included self-service AutoSSL. It didn't.
Triggering AutoSSL turned out to require WHM-level (the hosting provider's own backend) server-administrator access, which an ordinary
shared-hosting account never gets. That was confirmed against Namecheap's own documentation once
the free "Run AutoSSL" button turned out not to exist anywhere in the dashboard. The free certificate that does ship
with the hosting plan has to be issued from Namecheap's own side, and nothing on the customer's end
can force it.

The fake bug was worse, because it looked like a real one for a while. This laptop's own antivirus
intercepts TLS connections and reissues every certificate itself, so anything running on the
machine, curl and openssl included, was reporting the antivirus's own certificate instead
of the real one.

The tool being used to answer "has the SSL fix landed yet" could not tell the truth about it even
in principle.

Checking from a phone worked around it, and later a tool that runs the request server-side,
outside the interception entirely, confirmed it for good. Two support agents tried
force-validating the certificate from Namecheap's own side with no result, and a follow-up ticket
went in to track it as a pure backend wait. Nothing on the customer side could shortcut that part.
The certificate issued not long after, and a server-side check confirmed clean HTTPS.

Two things the plan hadn't accounted for

First: SSH turned out to be off by default on shared hosting, though the migration plan had been
written without confirming that. The fix was self-service, a control panel option called "Manage
Shell," which gives no hint it's SSH, listening on a non-standard port rather than the usual one. Second:
pushing to the new remote from an automated shell hung indefinitely with no error, because a global
credential setting that keeps two separate GitHub accounts from colliding also blocks the plain
per-repo push path unless a repo-level override is set. That global setting has to stay exactly as
it is. The only new part is the missing per-repo override.

A smaller trap surfaced once both hosts were finally level with GitHub again: the control panel's
Git tool has an "Update" button that only saves configuration. It doesn't fetch, pull, or deploy
anything, and those actions live on a separate tab entirely. For a repository that was deliberately
cloned through the control panel's own "Clone a Repository" tool, with a specific option left unchecked to
preserve its automatic deploy hook, even that tab's own "Update from Remote"
option turns out to be a no-op. It's a push target, not a pull source. Reading the UI correctly
took longer than reading the DNS panel had.

Deleting a task is not a local edit

One step in the migration plan retired Netlify outright once the new host was live. Once Namecheap
turned out to be an interim stop rather than the final destination, the task was dropped, which
seemed like a small, contained change. It wasn't. That same task had been referenced as a
precondition in five other places across the plan, and one of those was dangerous on its own: a
later step told a future session to delete the Netlify credit-cap constraint from the project's own
guide file, once hosting had moved. Hosting hadn't moved. A session following that instruction
literally would have silently removed a still-live limit from the record that exists to track it.
All five references were found and corrected before that could happen.

What actually changed the private-repo rule

The preference that had driven both GitHub Pages rejections wasn't overturned by a technical
argument. It was relaxed on purpose, once the user actually named a real gap instead of waving it
off: the diary's free-text fields, a description typed next to a photo, have no structural way to
stop someone entering something identifying. Going public meant accepting that gap rather than
pretending it away. In the user's own words, it was "hard to guard... loose enough to say go
public." The account-split blocker from the second rejection turned out not to matter here either,
since the plan kept the repo under its existing account rather than moving it to the newer brand's.

A bigger swap, investigated instead of just refused

A smaller question, whether a branded subdomain could dodge a shared hosting username, led
somewhere larger: making the newer, higher-traffic domain the account's designated "Main Domain,"
on the assumption that the label itself would grant it more subdomain headroom. That assumption
didn't survive a check. The account's subdomain quota is shared regardless of which domain holds
that label, so an ordinary addon domain gets exactly the same headroom. What an
actual swap would require turned out to be much bigger than the label suggested: a different
project's own blog would have to leave its free, automatic hosting for manual shared-hosting
deploys with none of that automation and the same multi-day certificate process already documented
here. That's not a call this project gets to make about another one. It was declined without
touching anything, and the idea went out as two information-only notes to the two other projects'
own owners, for them to decide on their own terms.

How it actually landed

The reversal, when it came, was decided outside any of these sessions, by the one person who owns
all three projects at once: Netlify was retired entirely, and the food diary moved to GitHub Pages
after all, kept under its existing account rather than a newer pseudonymous one. That keeps a tool
built under a real identity separate from a brand built under a pseudonym, and it sidesteps the
cross-account domain restriction outright rather than working around it.

The account question that had paused the move turned out simpler than the plan in flight at the
time suggested. Rather than standing up a new account, the existing one kept its history and its
repos exactly where they were and simply changed its own display name. Same account throughout,
just renamed partway through. The migration is complete now. Both old hosts are fully retired, and
nothing is left dual-hosted.

It ended without a clever fix. Two things went wrong here that weren't code. Both were caught the
same way, by someone who actually checked instead of assuming. A certificate check that had every
reason to look trustworthy was quietly answering a different question than the one being asked. And
a written decision only held once someone actually reopened the file before speaking a second time.

Originally published at thekilted.dev/nobody-reread.

The verdict that survived its own correction

the kilted dev — Sun, 30 Aug 2026 06:15:27 +0000

This project started with a conclusion already sitting in its lap. In June, an earlier stretch of
work on the same machine had tested whether a local model could handle iterative coding: writing
files, running tests, fixing what broke, without a human shuttling text between a chat window and
an editor. The verdict was no. Not viable. That was the finding the whole project was built on.

On 4 July, someone re-tested it. It fell over in about two minutes.

The interesting part isn't that the verdict was wrong. It's that the verdict was right. It had
been stated precisely, with its own explanation attached, and none of that survived the trip into
the new project.

The re-test

The task was a word search game: three files, a 10x10 grid, eight words hidden horizontally,
vertically or diagonally, click-and-drag to select, correct words struck through in the list, a
New Game button. Eight scored criteria. The eighth is the one that matters: ask for one follow-up
fix (make found words green) and check it lands without breaking the drag highlighting that
already worked. That regression check is what had killed every previous run.

Run through little-coder, a small harness that gives the model real write and read tools instead
of asking it to print code into a chat window, against qwen3-coder-30b served by Ollama, the
result was 8 out of 8 in 2.1 minutes. The follow-up fix took another 30 seconds. All of it verified
in a real browser with simulated drag events, not taken from the model's own account of what it had
done.

The same benchmark against the smaller qwen3.5-9b scored 7.5 out of 8 in 44.1 minutes. It lost
half a point on reverse-placed and diagonal words: no reversed-string check, an off-by-one in the
highlight loop. It still passed the regression test, the first local setup here to manage that.

Model	Score	Time
qwen3-coder-30b	8/8	2.1 min
qwen3.5-9b	7.5/8	44.1 min

The quality gap was only half a point; the time gap was a full 42 minutes.

Two honest marks sit against the 8 out of 8. One transient failure died with "Stream ended without
finish_reason" and an identical retry worked. On criterion 8, one green pixel was never confirmed,
because the preview tab had collapsed to a zero-width viewport and inline styles had stopped
applying: impossible in a real browser, so it was scored as a harness fault rather than a code
fault. That was a judgment call, made and written down as one at the time.

What this project wrote down that evening

The June verdict was a harness problem, not a model problem.

It's a clean line. It reads as an insight. It went into the project's working memory as a headline
finding, was quoted straight into the active constraints file, and shaped everything scoped
afterward: if the harness is the variable that matters, you invest in harnesses and stop shopping
for models.

The investment was right. The claim wasn't, and nobody checked it for two weeks, because a
conclusion that's useful and turning out well doesn't feel like it needs auditing.

What the original verdict actually said

The June work had been handed over as a document. Reading it properly, two weeks late, the verdict
is one row in a table of six use cases:

Use case	Locally viable
Claude Code CLI backend	✕ broke — models misinterpret Claude Code's own system prompt format, not a config problem
Multi-file document extraction (LM Studio + Gemma 4 E4B, 8192 ctx)	✓ met — works well, the best multi-file result of anything tried
Single-file extraction, agent + filesystem (Goose + Ollama, qwen3.5-9b)	✓ met — works, but slow: 62 seconds per file
Iterative coding (any local chat UI)	✕ broke — not viable, and structural: five failure modes, not a fixable bug
Single-turn code generation (Goose or LM Studio, max context)	✓ met — viable as a baseline handoff into a human editor
Quota fallback for real sessions (fcc-server)	✕ broke — routed to Gemini instead, local wasn't trusted as the fallback

June's headline verdicts, all six, from the same document that scoped iterative coding to "any
local chat UI." The row that got re-tested is one of six, not the whole page.

The scope is in the verdict line. Not "local models cannot do iterative coding." Local models
cannot do iterative coding in a chat UI, which is a claim about an interface.

Below it, five structural failure modes are listed. Stateless regression, where a fix in turn N
gets silently undone in turn N+1: observed, in that same word search task, as a drag highlighter
fixed and then broken. Overconfident reporting. Silent context truncation. Effective context not
matching advertised. No cross-session persistence.

And then, closing that section:

Why Claude Code avoids 1–2: tool-mediated verification per step, persistent conversation
context — architecture, not raw model capability.

June had already worked out that the constraint was architectural. It's written down, in the
document that was handed over, in bold, five weeks before the July re-test that "discovered" it.

July didn't overturn June. July supplied the harness June had described, and measured what
happened.

Where June was too optimistic

Of the five failure modes, June expected a harness to handle the first two: stateless regression
and overconfident reporting, both covered by tool-mediated verification and persistent context.

The first half held. The second didn't.

Overconfident reporting is a model claiming completion when a feature doesn't actually work, and a
harness only catches that if its tests are adversarial. This project has watched a weak test
launder a false claim into a green, twice. A redaction check passed with
assert "84210.55" not in out while a partially redacted amount, 842[AMOUNT], leaked straight
through it. A loader passed nineteen of nineteen with a leaked loop variable, masked by a fixture
that only ever exercised one file.

Real file tools, persistent context, tests that ran green: both greens were wrong.

So June wasn't simply more careful than its own summary. On this one point it was optimistic, and
the correction only turned up once this project built the harness June had described and ran it
long enough to get burned.

What actually went wrong

Four things actually changed between June and July, worth stating because it means the re-test was
never a controlled experiment. The model changed. The harness changed. The runtime flags changed
(July ran with flash attention and a quantised key-value cache, June predated both). And the prompt
changed, because June's prompt text was never preserved and had to be rebuilt from scratch.

That last one isn't carelessness. The June raw evaluation file survives, and it's 809 bytes: a
findings summary whose entire task description is one line. There was no prompt left to preserve by
the time anyone went looking. Reconstruction was the only option on the table.

But the four variables are a side issue. The real failure was compression.

What travelled from June into this project was the phrase "iterative coding not viable." The scope,
any local chat UI, didn't make the trip. Neither did the mechanism: architecture rather than raw
model capability.

Then someone re-tested the compressed form, found it wanting, and corrected it into a claim broader
and less accurate than the original. A precise, well-reasoned, correctly scoped verdict went in one
end and came out the other as a slogan, and the slogan is what the project ended up arguing with.

The verdict, June: Iterative coding — not viable (structural, see the five failure modes) — any
local chat UI. Scoped to an interface, with its own mechanism named: architecture, not raw model
capability.

What travelled: "Iterative coding not viable." The scope and the mechanism both stayed behind.

What got re-tested: "Harness, not model." Broader and less accurate than the verdict it
replaced, and the claim this project ended up arguing with.

The handover document's own header says it plainly:

Nothing here needs re-deriving — it was all tested hands-on.

This project re-derived it anyway, and ended up with a worse version.

The part that only appeared while writing this

Writing this piece meant going back to the June sources, and the first attempt to find them failed.
The active constraints file listed the handover document under a path in the other project, the one
that had written it. It wasn't there. Searching that project turned up nothing.

The conclusion drawn from that was that both June primaries had been deleted. That conclusion went
into a correction to this project's own files, into an early draft of this piece, and into a
request asking another project to recover them from git history.

The document was sitting in this project's own repository root. It had been there since the first
commit. It had been written elsewhere and moved across at handover, exactly as its own header says.
The raw evaluation file had moved into a shared knowledge vault during a migration weeks earlier,
and was just as findable. The other project located both files in minutes, and mentioned, almost as
an aside, that it has no git history at all, so the recovery being asked for could never have worked
anyway.

One failed lookup at one stale path became "gone from disk," and a fair amount was built on top of
that before anyone checked the obvious place.

That's the same mistake this piece is about, made while writing about it.

An absence, asserted from a single negative result. A conclusion, drawn from a summary (the path in
the constraints file) rather than from the thing itself.

What changed

The finding was rewritten to what the evidence actually carries. The harness thesis is June's, not
this project's, and it was right. What this project added is confirmation and a measurement. The
broad version, "harness, not model," is now marked as over-stated, because it took this project's
own later evidence, not June's, to find the one place a harness alone wasn't enough.

The practical guidance didn't change at all, which is the uncomfortable part. Invest in the
harness. Verify with real tool output. Keep the local tier for bounded work. All still correct, all
still what this project does. A claim can be over-stated and still point the right way, and that
combination is exactly what stops anyone from checking it again.

Two rules came out of this, both narrow and mechanical. Before re-testing an inherited verdict,
read the primary document, not your own summary of it: an active-constraints file strips scope and
exceptions by design, because stripping detail is what it's for. And an absence needs more than one
failed lookup, especially when the path you searched came from the same compressed source you're
already re-deriving from.

What travels between sessions, projects and people is the conclusion. The conditions that produced
it stay behind. A conclusion without its conditions is a rumour with a number attached.

Originally published at thekilted.dev/the-verdict-that-survived-its-own-correction.