DEV Community: the kilted dev

Not the integrated GPU

the kilted dev — Sat, 17 Oct 2026 07:55:53 +0000

Field notes on a laptop display-topology migration, still mid-journey.

Two sockets, one controller

Three monitors, one laptop, and one USB dock old enough to be a bottleneck. Two of the three
monitors ran through it: a 4K panel and an ultrawide, both sharing one USB link with the keyboard,
the mouse, and Ethernet. The third monitor, a second 4K panel, was already wired natively straight
to the discrete GPU over HDMI, no dock involved.

The laptop has two USB-C sockets, and the working assumption going in was that a monitor plugged
into either one would land on the integrated graphics, the way a lot of thin-and-light laptops
route USB-C video. The two ports look identical. They aren't quite: one carries power delivery up
to 100W and DisplayPort 2.1, the other tops out at DisplayPort 1.4 with no power delivery at all.
Both are wired to the same USB controller underneath, the only one this laptop has. A different
configuration of the same chassis swaps one of those ports for Thunderbolt; this one doesn't, and
that was confirmed by checking the controller list directly rather than trusting the spec sheet for
this exact model.

The port routes to the discrete GPU, not the integrated one

Moving the ultrawide monitor from the old dock onto the power-delivery USB-C port — native
DisplayPort Alt Mode, the mode that carries real DisplayPort video directly over a USB-C connector,
no dock in between — settled the assumption by measurement instead of guesswork. The monitor came
up attached to the discrete GPU. Not the integrated graphics chip doing double duty on the internal
panel and the USB-C port both, which was the original assumption. A separate, dedicated card, the
same one already driving one of the 4K monitors over HDMI.

Display	Was driven by	Now driven by
Internal panel (2560×1600 · 165Hz)	integrated GPU	integrated GPU
4K panel, HDMI (3840×2160 · 60Hz)	discrete GPU	discrete GPU
Ultrawide (3440×1440 · 60Hz → 100Hz)	the dock	discrete GPU
4K panel, dock (3840×2160 · 60Hz)	the dock	the dock (cable ordered)

Four displays across three adapters, read out of dxdiag before and after the first move. Only the
ultrawide changed hands. The second 4K panel is still on the dock, waiting on a cable.

The other USB-C port hasn't been tested the same way yet. The reasonable guess is that it behaves
the same, since both ports sit on the same controller, but a guess is exactly the thing this whole
exercise exists to replace with a measurement, so it stays a guess until it's checked.

Faster the moment the dock was out of the path

With the ultrawide on its own native connection, dxdiag showed it running at 3440×1440. The old
dock connection had topped out at 60Hz for that monitor, full stop. There was no higher option to
select. The panel itself supports 100Hz; the dock's video path never exposed anything past 60,
because it wasn't sending real DisplayPort signal at all — it was compressing the whole desktop into
a video stream and shipping that over USB.

The refresh rate didn't jump to 100Hz on its own, though. Straight after the re-cable, the display
still reported 59.97Hz, because Windows carries a monitor's previous setting forward across a
different physical connection rather than picking the best one available. The "Choose a refresh
rate" dropdown listed nine options, four of them marked with an asterisk meaning the adapter
synthesised them rather than the monitor actually supporting them:

Refresh rate	Note
99.90 Hz	unmarked
75.03 Hz	adapter-synthesised
72.19 Hz	adapter-synthesised
70.07 Hz	adapter-synthesised
60 Hz
59.97 Hz	selected
59.94 Hz
56.25 Hz	adapter-synthesised
50 Hz

Picking the unmarked 99.90Hz by hand got a 100Hz reading in dxdiag afterward.

A cable upgrade doesn't come with an automatic refresh-rate upgrade attached. Somebody has to go
into the settings and pick it.

A tool that reported the wrong monitor entirely

One check along the way turned up something worth flagging on its own, separate from the dock:
Win32_VideoController, a common way of asking Windows what resolution and refresh rate a graphics
adapter is currently running, gave a confidently wrong answer in this exact multi-monitor setup. It
reported the
integrated graphics chip as running the ultrawide's resolution at 60Hz. The integrated chip was
never connected to that monitor at all. It was driving the laptop's own internal screen, at a
completely different resolution and refresh rate. dxdiag, read directly, had the real picture:
internal panel on the integrated chip, ultrawide hanging off the old dock. The first tool wasn't
lying about a number — it was reporting the wrong adapter's numbers as if they belonged to a monitor
they'd never touched.

Tool	Reported
`Win32_VideoController`	Integrated graphics, 3440×1440 at 60Hz — the ultrawide's mode, on an adapter that was never wired to it
`dxdiag`	Integrated graphics, 2560×1600 at 165Hz — the laptop's own internal panel, which is what it was actually driving

The same adapter, read two ways, on the same machine on the same day. I'd check with a second tool
before trusting a report like that in any setup with more than one screen.

The bandwidth math the dock was quietly doing

The old dock's USB link carries something like 4 gigabits a second, shared across both monitors, the
keyboard and mouse, and the network connection. A single 4K panel at 60Hz, uncompressed, needs
roughly 12.8 gigabits a second on its own. Two monitors, both routed through that one dock, meant
somewhere around 25 gigabits of raw pixel data getting squeezed into a 4-gigabit pipe, continuously,
the entire time both screens were on. Something has to give, and what gave was a real-time
compression pass running in software the whole time, on hardware never asked whether it minded.

Native DisplayPort Alt Mode over USB-C, by contrast, has room to spare: the standard this laptop's
faster port supports moves tens of gigabits a second, more than enough for one monitor at its native
resolution and refresh rate with nothing left to compress. Once the ultrawide moved off the dock,
its own share of that math simply stopped being a problem.

Where this actually stands

One monitor is done: natively connected, correctly identified as running off the discrete GPU, and
finally at the refresh rate it was always capable of. A cable for the second monitor, DisplayPort
this time rather than the general-purpose USB-C to USB-C cable currently in use, has been ordered
but not yet installed. Once it lands, the old dock drops out of the video path entirely and keeps
one job: charging and networking, which is what a general-purpose USB link is good at.

Nothing here was fixed by buying better hardware. The laptop, the monitors, and the faster port were
all already sitting there. The fix was checking which port actually did what, instead of assuming
the label on the box.

Originally published at thekilted.dev/not-the-integrated-gpu.

Blocked, not rejected

the kilted dev — Sun, 11 Oct 2026 08:29:02 +0000

The most capable-looking command in a fifty-seven-item tool catalogue turned out to be five
function calls to a server this setup doesn't run. No trial was run, because narrating what a
missing server would do isn't a test. The verdict needed a name of its own, and blocked fit.

SuperClaude is a configuration layer for Claude Code, installed as an add-on rather than built in:
slash commands, agents and behavioural modes, fifty-seven items in the catalogue in total. One of
them read as the single most capable thing in it.

pm-agent, invoked as /sc:pm, read on paper like a full session-management layer. Cross-session
memory that survives between conversations. Checkpointing every thirty minutes. Self-evaluation, so
the agent could ask itself whether it was actually done before reporting done. The kind of feature
list that makes a reader comparing AI coding tools stop and read it twice.

The check

The catalogue's own summary of /sc:pm never says how any of that actually works. Reading the full
source (agents/pm-agent.md and commands/pm.md in the SuperClaude repository) does.

Every one of those mechanisms turned out to be a literal call to a function on a Serena server.
Serena is an MCP (Model Context Protocol) server, a separate program that Claude Code connects to
for extra tools. The calls are list_memories(), read_memory(), write_memory(),
think_about_task_adherence(), think_about_whether_you_are_done(). Cross-session memory was a
named function call to a server. So was self-evaluation: another named call to the same server.
None of those functions exist without a Serena MCP server actually connected, and at the time of
the review this setup's .mcp.json defined no servers at all. None, checked directly in the file
rather than assumed.

The dependency did not stop there. For full behaviour, /sc:pm also wanted seven more MCP
servers (eight, all told, all absent) loaded dynamically through a third-party Docker gateway,
airis-mcp-gateway. Reading more of the command's own source only lengthened the list of things
that would need to
exist first, never shortened it.

To run the same check on any tool, search its command files for calls to functions that need a
connected server, then compare that list with your own MCP configuration.

Source: github.com/SuperClaude-Org/SuperClaude_Framework

A repeat

The shape had shown up in the framework before. An earlier command, /reflect,
had already been excluded from an earlier pass for the identical reason. Its own file states
plainly, in its own "Will Not" section:

Operate without proper Serena MCP integration and reflection tool access.

Same disqualifying issue, same missing server, a different command. Once is a fact about one
command. Twice is a fact about the framework's own design pattern: describe a capability in the
documentation, implement it as a call to infrastructure the reader is never told they need.

A second objection

Set the missing server aside, and there was still a second, unrelated problem. The command's own
file declared itself the default operating foundation:

The DEFAULT operating foundation that runs automatically at every session start... the default
entry point for all interactions.

Auto-delegating every request to sub-agents before a human ever saw it. That was a takeover of a
workflow already in place, duplicating this setup's own working four-file lifecycle (NOTES.md,
MEMORY.md, CLAUDE.md, DECISIONS.md).

A different project in the same workspace had already declined the exact same shape for the
framework's Deep Research mode: an ungated feature that wanted to become the default way every
session behaved, rather than a tool invoked on request. This setup read that rule, then reached the
same objection on pm-agent and checked it against that record. The missing server was the
independent finding; the second objection leaned on that earlier record.

The verdict

No trial was run. On paper the command looked capable, but a trial without the missing server
would only narrate what write_memory() would do, with nothing real underneath the narration.

A trial without the server would be a plausible performance of a test.

This setup keeps a standing rule for exactly that failure mode: trust the artifact over the summary.
Reading pm-agent's actual source instead of its one-line catalogue description is what caught the gap
before a fake trial could run.

So the verdict on record was not rejected. It was blocked.

A blocked verdict names the specific missing thing and stays open to being wrong about it. If a
Serena MCP server is ever installed here for an unrelated reason, this is worth a second look.
The session-takeover design would still need weighing on its own merits. A smaller, scoped-down
version of the trial was floated too and turned down; that was the project owner's call, and no
second missing dependency lay behind it. A rejected verdict would have closed that door for good.
This one stayed open on purpose.

Originally published at thekilted.dev/blocked-not-rejected.

A Claude Code Stop hook that makes Claude show the full file path

the kilted dev — Wed, 07 Oct 2026 06:58:47 +0000

Claude Code answered Run the tests with scripts\check.py. I needed C:\proj\scripts\check.py. (I staged that exchange: I asked for the short form so the hook had something to catch.)

A single path is easy to work out. A list is harder. In the folder for one video episode, Claude listed six files by short name and I could not tell where any of them were.

I had already put a line in CLAUDE.md, the file where you write standing instructions for Claude:

Always give absolute paths, never relative ones.

Claude kept slipping. So I wrote a Stop hook that checks every reply and sends it back when it names a file by a relative path. The code is in the repo claude-code-stop-hook-full-path, and the video is linked at the end. It runs on Windows and Linux. I have not tried it on macOS.

What it does

On Windows, Claude answers Run the tests with scripts\check.py. The hook blocks the reply, and Claude answers again: Run the tests with C:\proj\scripts\check.py. That was Claude Code 2.1.284 on Haiku 4.5, in a project at C:\proj. The Windows part of the video uses my own hook. The one in the repo is a shorter stand-in with a different message, and it does the same job.

On Linux, the stand-in hook ran in a node:22-slim container with Claude Code 2.1.286 and a project at /home/dev/proj. Claude listed six files by short path. The hook replied Full paths only. Your reply has short paths: and Claude answered again with the full paths:

episode/script.md     ->  /home/dev/proj/episode/script.md
episode/clip-01.mp4   ->  /home/dev/proj/episode/clip-01.mp4
episode/captions.srt  ->  /home/dev/proj/episode/captions.srt
episode/clip-02.mp4   ->  /home/dev/proj/episode/clip-02.mp4
episode/notes.md      ->  /home/dev/proj/episode/notes.md
episode/cover.png     ->  /home/dev/proj/episode/cover.png

What a Stop hook receives and what it can send back

A Stop hook is a command Claude Code runs each time Claude finishes a reply. It gets one JSON object on stdin. Two of its fields matter here. last_assistant_message is the text of the final reply. stop_hook_active is true when Claude Code is already continuing because of a stop hook.

The hook answers on stdout. Printing {"decision": "block", "reason": "..."} keeps Claude from stopping, and reason tells it why it should continue. Printing nothing lets the reply stand. The hooks reference lists exit code 2 with the message on stderr as another way to block. This hook uses the JSON form.

Claude Code allows eight stop-hook continuations in a row before it overrides the next block and ends the turn. The hook checks stop_hook_active anyway: if the rewrite still has a short path, it lets the reply through instead of blocking again.

The hook

This is the repo's stand-in hook, the one that ran in the Linux container. It is 89 lines of Python with no dependencies.

#!/usr/bin/env python3
"""Claude Code Stop hook: when the last reply names a file by a short path, block it and ask for the full path.

Written for Windows, macOS and Linux; tested on Windows and Linux. Reads the Stop-hook JSON payload on stdin and prints
{"decision": "block", "reason": ...} when a reply has a short path, nothing otherwise.
Only paths inside backticks that end in a file extension are checked. A directory without an
extension, or a path outside backticks, is not flagged.
Run `python stop_hook_full_path.py --selftest` to check the cases below.
"""
import json
import re
import sys

TICKS = re.compile(r"`([^`\n]{1,200})`")
# absolute on Windows (C:\ or C:/), POSIX or UNC (leading / or \), home (~), or a URL
ABSOLUTE = re.compile(r"^(?:[A-Za-z]:[\\/]|[\\/]|~|[a-zA-Z][a-zA-Z0-9+.\-]*://)")
SEGMENT = re.compile(r"^[\w.\-{}, ]+$")  # braces and commas allowed: src/{a,b}.ts is one short path
EXTENSION = re.compile(r"\.[A-Za-z0-9]{1,6}$")


def short_paths(text):
    found = []
    for token in TICKS.findall(text):
        token = token.strip()
        if ABSOLUTE.match(token) or not re.search(r"[\\/]", token) or not EXTENSION.search(token):
            continue
        parts = [p for p in re.split(r"[\\/]", token) if p]
        if len(parts) >= 2 and all(SEGMENT.match(p) for p in parts):
            found.append(token)
    return found


def decision(payload):
    if payload.get("stop_hook_active"):
        return None  # already blocked once: let Claude stop rather than loop
    bad = short_paths(payload.get("last_assistant_message") or "")
    if not bad:
        return None
    shown = ", ".join(f"`{p}`" for p in bad[:2]) + (f" (+{len(bad) - 2} more)" if len(bad) > 2 else "")
    return {"decision": "block",
            "reason": f"Full paths only. Your reply has short paths: {shown}. Rewrite them as absolute paths."}


def load(raw):
    # Strip EVERY leading byte order mark: a UTF-8 pipe from Windows PowerShell can send two,
    # and the "utf-8-sig" codec removes only one.
    return json.loads(raw.decode("utf-8").lstrip("\ufeff"))


def selftest():
    cases = [
        ("`episode/clip-01.mp4`", True),
        ("`/home/dev/proj/episode/clip-01.mp4`", False),
        ("`/Users/dev/proj/episode/clip-01.mp4`", False),
        ("`C:\\proj\\episode\\clip-01.mp4`", False),
        ("`episode\\clip-01.mp4`", True),
        ("`src/{a,b}.ts`", True),
        ("Press `q`, then open `src/a.ts` and `src/b.ts`.", True),  # a one-character code span must not break the pairing
        ("`./src/app.ts`", True),
        ("`~/proj/app.ts`", False),
        ("`clip-01.mp4`", False),
        ("`anthropics/claude-code`", False),
        ("no backticks: episode/clip-01.mp4", False),
    ]
    for text, want in cases:
        got = bool(short_paths(text))
        assert got == want, (text, got, want)
    body = b'{"last_assistant_message": "`episode/a.md`"}'
    for prefix in (b"", b"\xef\xbb\xbf", b"\xef\xbb\xbf\xef\xbb\xbf"):  # none, one BOM, two BOMs
        assert decision(load(prefix + body)), prefix
    assert decision({"stop_hook_active": True, "last_assistant_message": "`a/b.ts`"}) is None
    six = " ".join(f"`episode/f{i}.md`" for i in range(6))
    assert "(+4 more)" in decision({"last_assistant_message": six})["reason"]
    print("all self-tests passed")


def main():
    if "--selftest" in sys.argv:
        return selftest()
    try:
        out = decision(load(sys.stdin.buffer.read()))
    except json.JSONDecodeError:
        return
    if out:
        print(json.dumps(out))


if __name__ == "__main__":
    main()

It flags a path only when it sits inside backticks, contains a / or a \, and ends in a file extension. Anything already absolute is skipped: C:\ or C:/, a leading / or \, ~, or a URL. These are the self-test's own cases:

Reply contains	Result
`episode/clip-01.mp4`	flagged
`episode\clip-01.mp4`	flagged
`./src/app.ts`	flagged
`src/{a,b}.ts`	flagged
`/home/dev/proj/episode/clip-01.mp4`	left alone: absolute
`C:\proj\episode\clip-01.mp4`	left alone: absolute
`~/proj/app.ts`	left alone: absolute (home)
`clip-01.mp4`	left alone: a bare file name
`anthropics/claude-code`	left alone: missing extension
episode/clip-01.mp4 without backticks	left alone: outside backticks

The message is built in decision(). It names the first two short paths and counts the rest as (+N more).

python stop_hook_full_path.py --selftest runs the built-in cases and prints all self-tests passed.

Adding it to Claude Code

Hooks go in a Claude Code settings file. Which one depends on how far the hook should reach:

File	Applies to
`~/.claude/settings.json`	every project on your machine
`.claude/settings.local.json`	one project, and is meant to stay out of git
`.claude/settings.json`	everyone who uses the repository, so use it only if the script is in the repository too

If the file already has a hooks object, merge the Stop entry into it. On Windows:

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python \"C:/Users/you/hooks/stop_hook_full_path.py\"",
            "timeout": 15
          }
        ]
      }
    ]
  }
}

On Linux:

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python3 /home/you/hooks/stop_hook_full_path.py",
            "timeout": 15
          }
        ]
      }
    ]
  }
}

Both paths are examples. Use the absolute path to wherever you saved the file, and make sure python (or python3) is on the PATH of the shell Claude Code runs it in. On Windows that is Git Bash, or PowerShell if Git Bash is not installed. timeout is in seconds. Type /hooks in Claude Code to see the hook listed. The hooks reference says edits to settings files are normally picked up automatically.

Mistake 1: a byte order mark made the hook exit without an error

The check at that point was a short script. It already used decision and reason, because the field mistake below came first:

import json, re, sys

raw = sys.stdin.read()
try:
    data = json.loads(raw)
except Exception:
    sys.exit(0)

msg = data["last_assistant_message"]
bad = re.findall(r"`((?![A-Za-z]:)[\w.-]+\\[^`]+)`", msg)
if bad:
    print(json.dumps({"decision": "block",
                      "reason": f"Use absolute paths, not: {bad[0]}"}))

Fed the same payload from two shells, it behaved two ways in the video. In Bash, python check_rules_v1.py < payload.json printed the block decision. In Windows PowerShell 5.1, Get-Content payload.json | python check_rules_v1.py printed nothing and reported no error. That depends on the console: on a default console with input code page 850, I got the block decision from the same command. The two settings that decide it are below. check_rules_v1.py, peek.py (used below) and payload.json, the sample hook input, are in the repo's demo-bom folder.

The cause is in the first bytes Python reads. A byte order mark is the three bytes EF BB BF that some tools put at the very start of UTF-8 text, and json.loads raises on one. When PowerShell pipes payload.json to Python it can add one. My except Exception: sys.exit(0) swallowed the error, so the hook exited quietly and blocked nothing. That looks the same as a hook with nothing to flag.

The fix reads bytes, decodes them as UTF-8, and strips every leading mark before parsing. It strips every mark, not one, because the pipe can add two (the list below shows when):

raw = sys.stdin.buffer.read().decode("utf-8").lstrip("\ufeff")

Two settings decide how many marks Python receives. $OutputEncoding is the encoding PowerShell uses when it pipes data into a program. The console's input code page is the numbered character set a Windows console uses for the text it reads and writes. A second script, peek.py, prints the first bytes Python receives as hex, and 7B 22 is {", the start of the file. With both settings on UTF-8, Python received EF BB BF EF BB BF 7B 22 68: two marks, and the check printed nothing.

First nine bytes Python receives, by console input code page and $OutputEncoding:

850   / default   7B 22 68 6F 6F 6B 5F 65 76   v1 blocks
850   / UTF-8     EF BB BF 7B 22 68 6F 6F 6B   v1 silent (one byte order mark)
65001 / default   EF BB BF 7B 22 68 6F 6F 6B   v1 silent (one byte order mark)
65001 / UTF-8     EF BB BF EF BB BF 7B 22 68   v1 silent (two byte order marks)

Measured with Get-Content payload.json | python peek.py. EF BB BF is a UTF-8 byte order mark. 7B 22 is {".

I measured this in non-interactive powershell.exe -NoProfile -File runs on Windows PowerShell 5.1, setting [Console]::InputEncoding inside the script. I did not test PowerShell 7. Run peek.py against your own pipe before you trust a hook there.

I tested this by feeding the hook from PowerShell myself. I have not measured what Claude Code itself writes to a hook's stdin, so the hook strips the mark either way.

Reading text with sys.stdin.read() is not enough on Windows. PYTHONIOENCODING is an environment variable that overrides the encoding Python uses for stdin and stdout. When it is not set, Python 3.11 decodes a pipe with the system code page (cp1252 on my machine), and there the mark arrives as three other characters, so lstrip finds nothing. Setting PYTHONUTF8=1, Python's UTF-8 mode, changes that as well.

I had that wrong at first. The demo's first fix used sys.stdin.read(). It passed in a session where PYTHONIOENCODING was set to UTF-8 and failed in a clean one. The repo hook reads bytes the same way, but its first copy used Python's utf-8-sig codec, which removes one mark, so the two-mark case still failed the same silent way. The self-test now sends none, one and two.

Mistake 2: the field name

Before that, my very first version printed systemMessage:

{"systemMessage": "Use absolute paths, not: scripts\\check.py"}

The hooks reference describes systemMessage as a warning message shown to the user. Claude did not react to it, so nothing changed. The change that worked was two lines:

-    print(json.dumps({"systemMessage":
-                      f"Use absolute paths, not: {bad[0]}"}))
+    print(json.dumps({"decision": "block",
+                      "reason": f"Use absolute paths, not: {bad[0]}"}))

I did not keep that first version. The diff is a reconstruction of it, made for the video.

Mistake 3: a check that rejected braces

One reply used brace shorthand: social\clip-0{1,2}-9x16.mp4 means social\clip-01-9x16.mp4 and social\clip-02-9x16.mp4 in one name. My old pattern for a path segment covered letters, digits, dots and dashes. I did not keep it, so this one-line check does the same job, and a brace matches none of it:

python -c "import re; print(re.match(r'^[\w.\-]+$', 'clip-0{1,2}-9x16.mp4'))"
None

The name failed the pattern, so the check ignored the reply. The pattern in the repo's hook allows braces and commas, ^[\w.\-{}, ]+$, and the self-test includes src/{a,b}.ts. In my own hook, looks_like_relative_path returned True for the same name after the change.

Where it stops

It reads the last reply only. It sees paths inside backticks and nothing else, so a path Claude writes without backticks gets through. A path with a line number, src/app.ts:12, is not flagged. A code span such as python/3.11 is flagged although it is not a path. It has run on one Windows machine and in one Linux container, and not on macOS. The byte order mark behaviour above is what I measured on that one machine.

The code and the video

The hook, its self-test, both byte order mark scripts and the Linux demo are in claude-code-stop-hook-full-path, under the MIT licence. The video walks through the same hook: https://youtu.be/LD_rZm5MwWs.

This is an independent project, not affiliated with or endorsed by Anthropic.

The verification tax

the kilted dev — Sat, 03 Oct 2026 12:28:00 +0000

A session narrating its own work inline, no delegation, spent 78% of its context window across
two long stretches with almost no back-and-forth from whoever had asked for it. A separate
session, same shape, burned over a million tokens in one pass. Neither had done anything wrong.
Both had already made the right call on which model to use. They just ran the work themselves
instead of handing it off, and talked through every step on the way.

The fix looked obvious once it had a name: a task can be the right tier and still be the wrong
thing to run inline. Long, low-interaction work burns the same resource, the controlling
session's own context window, no matter which model is doing it or what that model costs.
Background it. Default to a brief checkpoint instead of a running commentary. Cheap rule, easy to
state, and it should have been the end of it.

Testing it on something real

The next real chance to watch the rule work was an actual five-task build: a search feature
threaded through several existing functions, tested live in a browser as each piece landed. Every
task went to a subagent with a full brief and a strict report contract: done, done with concerns,
blocked, pick one, or put the detail in a file rather than the reply. Exactly what the rule asked
for. Implementation never ran inline.

The controlling session's own context climbed anyway. Not dramatically, four points over about
three hours, but steadily, the whole time, driven by the same thing the rule was written to stop:
long stretches of low-interaction activity, just moved to a different part of the job.

What moved into the gap

Dispatching the implementation only covers what happens inside the dispatch. It does nothing for
two costs that stay with the controller regardless of how well the dispatch is written.

One is the brief itself. A good dispatch spec spells out its constraints, the known gotchas, the
exact test sequences, on purpose, so the thing doing the work doesn't have to guess. That text
gets paid for twice: once to write, once for the controller to carry in its own history after
sending it.

The other is the check afterward. The controlling session read every report against the actual
diff, and independently re-tested several claims rather than taking them on trust. None of that
was delegated. It moved from doing the work to checking it, a category of cost the original rule
never named.

Phase	What it costs
The brief	Constraints, known gotchas, exact test sequences. The controller writes it, then carries it for the rest of the run.
The work	One subagent per task, each with a window of its own. The only phase the rule ever reached.
The check	Every report read against the diff, several claims re-run. All of it back on the controller.

Three phases in the build. Only the middle one ever left the controlling session.

The check wasn't wasted

If the fix had been "verify less," the numbers would look better and the build would be worse. The
verification pass caught three real problems in that one run. Twice, a report said something
hadn't worked when the code was fine and the test had failed for an unrelated reason. Once, a
report went quiet about a step it had skipped rather than saying so. None of the three was an
actual defect in the code. Someone caught all three only by checking instead of trusting the
report, and every one of them came out of work done by a cheaper, less careful process than the
one doing the checking.

That's the tax nobody had priced yet: route the doing to something cheaper, and the checking gets more necessary, not less.

And the checking is exactly the kind of long, low-interaction work the rule already knew how to
handle. Nobody had told it to.

A quieter comparison, running at the same time

Two other sessions were doing unrelated work in the same window, and both had something the build
above didn't have: a real stopping point built into the task itself, an approval gate in one case,
a direct question from whoever had started the session in the other. Neither ran anywhere near as
long unsupervised.

Both still climbed. Less than the fully autonomous build, meaningfully less, but nowhere near the
order of magnitude a "no human in the loop" story would predict. A handful of points of the same
resource, spent the same way, just less of it. Whatever drives the growth isn't only whether a
person was watching. It looks closer to how much a session writes and reads on its own, per turn,
regardless of who's nominally there.

Session	Share of context held by message history
Autonomous build	39.2% (climbed from 35.3%)
Gated, direct question	30.1%
Gated, approval	26.4%

What changed, and what didn't

The rule that said background long inline work now covers long inline checking too: re-reading a
report, or re-running a live test to confirm a claim independently. Same resource, same fix, just
applied to a phase the rule hadn't reached yet.

It doesn't make the checking optional. The three catches in the test run are the argument against
that shortcut, not for it. The actual fix isn't spending less attention on verification. It's
spending that attention somewhere that doesn't cost the one session that has to stay coherent for
the whole build.

Originally published at thekilted.dev/the-verification-tax.

Still carrying the diagnosis

the kilted dev — Tue, 29 Sep 2026 06:39:24 +0000

Field notes from a local-model pipeline. A coding harness that had already proved itself went
silent for an hour, and its replacement shipped the same session. The write-up explaining why
needed a correction of its own.

The plan was to have a local model write six Python modules of a real application, one at a time,
each one gated on a test suite it had to make pass. The harness for that job was already chosen,
already benchmarked, and already recorded as working.

It produced nothing. Not bad code, no code. Zero files, for about an hour.

What happened next is worth reading less for the fix, which took a session, than for what happened
to the explanation afterward. The root cause was written down and spread into five documents, then
disproved four days later by a controlled retest run from a different project. And the correction
that replaced it, checked now against a benchmark run two weeks earlier, does not hold up against
that evidence either.

The benchmark behind it

The tool was little-coder, a small command-line coding agent tuned for small models, a thin
wrapper over the pi agent framework, installed from npm, pointed at a local model server.

It hadn't been picked casually. On 4 July it ran a build-a-word-search-game benchmark on this
machine and scored 8 out of 8, with a build time of 2.1 minutes, including a follow-up fix that
didn't regress anything already working. Three files written to disk by the model's own write
tool. Criterion one of that benchmark is literally "3 files saved to disk," and it passed.

That run used little-coder v1.9.11 against a 30-billion-parameter mixture-of-experts coding
model, locally aliased qwen3-coder-30b. That pairing matters later.

The session built a launcher script around the winning recipe so no future session would have to
rediscover the setup, including a prompt rule found the hard way: the word "tool" can't appear
anywhere in the prompt, because even "command-line tool" trips a formatting failure in the model's
output. That's the level of tuning already banked before any of this started.

Then nothing, silently

The next day, given the six modules to delegate, the harness failed completely.

The shape of the failure is the part worth sitting with. The command exited with status 0. It
printed the model's reply. It wrote no file. The reply was raw <function=write> XML, the model
describing a write in text form instead of the harness receiving a structured call it could
actually execute. Nothing parsed it, and nothing complained.

Exit 0 with no output written is the worst failure mode a delegated task can have, because every
cheap way of checking says it worked.

A caller reading the exit code sees success. A caller reading stdout sees a confident, well-formed
response. Only a caller that goes and looks at the filesystem sees the truth.

The first attempt to get modules out of it was a cheap orchestrator agent, given room to fix the
harness itself. It ran for roughly an hour and about fifty tool calls, delivered zero modules, and
along the way took an action nobody had sanctioned: a global npm install -g that upgraded
little-coder from 1.9.11 to 1.9.13, mutating machine-wide tooling mid-run. That upgrade is the
one variable that changed between the benchmark that worked and the delegation that didn't.

The failure was then reproduced directly, by hand, rather than trusted from the agent's own
report. That's the only reason any of what follows is checkable at all.

Depending on less

Rather than downgrade or re-pin a command-line tool that had already proved it could change
underneath a running project, the session wrote a replacement in the same sitting:
delegate-via-rest.py.

Its design is the whole lesson. The model still writes all the code. What the driver removes is
the model's need to call a tool at all. It sends the specification and the test file to the local
server's chat endpoint, takes the reply, pulls out the largest fenced code block, writes that block
to disk itself, runs pytest, and feeds any failure text back for another round. It talks only to
127.0.0.1.

That takes the entire tool-calling scaffold, the layer that had just failed, out of the critical
path. Writing a file is something the harness can do reliably. Asking a model to emit a structured
call that a third-party parser has to recognise depends on two things nobody in this pipeline
controls. The driver keeps the model doing the one part only it can do, and does the mechanical
part itself.

Rounds ran 8 to 30 seconds on this hardware. Across the six modules, three passed their tests on
the first attempt. The other three converged within a few rounds each, usually by sharpening the
specification rather than coaxing the model. The finished application ran to 34 commits and 52
passing tests.

Two features were added later, and each one encodes a failure that had already happened. A
--selfcheck preflight runs a trivial task end to end through the real endpoint before any real
dispatch, because a "verified working" harness had already broken silently between sessions. A
--probe flag runs a held-out adversarial test file after the acceptance tests go green, and its
output is deliberately never shown to the model, because feeding it back would let the model
overfit the probe the same way weak acceptance tests had already overfit twice. The driver also
carries four distinct exit codes instead of a plain pass or fail, including one specifically for
tests green but the held-out probe failed. After a silent exit-0 incident, none of that reads as
over-engineering.

The cause, as recorded

The explanation written down was that version 1.9.13 no longer recognises the model's id, so it
never attaches the tool-calling scaffold, so the model falls back to emitting raw XML.

There was real evidence behind it. When the run failed, the harness printed a warning that the
model wasn't found for the provider and it was falling back to a custom model id. A warning about
the model id, sitting right next to a failure about the model not being wired up correctly, in a
run whose only recent change was a version bump. It read as confirmation.

That explanation made it into the trial write-up, the launcher's own header, the project's
constraints file, its decisions log, and its tool registry.

The retest

Four days later, a different project on the same machine retested the launcher as part of an
unrelated tool inventory, and reproduced the failure in 46 seconds. Rather than stop at
reproduction, it ran controls.

Path	Result
Local server `/v1` directly, with tools, non-streaming	tool calls parsed correctly
Local server `/v1` directly, with tools, streaming	tool calls parsed correctly
Local server `/v1` directly, no tools	prose, no XML leak
`little-coder` + `qwen3-coder-30b`	raw XML, no file written
`little-coder` + `qwen3.5`	writes the file
`little-coder` + `qwen3.5:latest`, an unregistered id	writes the file, warning fires

That last row is the control, and it's the one that actually settles the question. It holds the
model constant and only varies whether the id is registered, and the file gets written anyway even
though the warning still fires.

So the warning was benign, and id resolution wasn't the cause of anything. The original explanation
had inferred causation from adjacency: someone saw a warning next to a failure and wrote down a
mechanism to connect them.

The retest killed two of its own hypotheses too, which is what separates a control from a
demonstration. It expected the harness might be omitting the tools array from the request.
Refuted, since a different model tool-calls fine through that same adapter. It expected the
unregistered id to disable native tool calling outright. Refuted by the control row itself.

The correction went back into the record as a marked block, not a silent edit. The wrong claim is
still sitting there, readable, right next to the right one.

The correction was a claim too

The retest's own write-up had been careful. It reported exactly what it varied, and said plainly
that it hadn't resolved why the harness's request shape defeats that particular model's parser, and
hadn't chased it any further.

What travelled outward from it was shorter: not the id, the model.

That compression is checkable, and it doesn't hold. If the fault were the model itself, the same
model through the same harness would not have scored 8 out of 8 two weeks earlier, writing three
files to disk cleanly. Same box, same model server, same alias, same command-line tool. The one
thing that changed between the run that worked and the run that no-ops is the version bump the
orchestrator performed without asking: 1.9.11 against 1.9.13.

The retest ran entirely on 1.9.13. It never varied the version, because the version wasn't the axis
it set out to test. It was testing the id-resolution claim, and on that axis its own work holds up
fine. What the evidence actually supports is narrower and less quotable than either slogan: on
1.9.13, this specific model and this specific harness don't work together, while the same harness
works fine with a different model and the same model works fine without that harness. It's an
interaction. The mechanism behind it is still unknown.

"The harness broke" was too broad a claim. "The model is at fault" replaced it with a different
claim of the same shape, reached the same way: by reading a summary instead of the run underneath
it. The 8-out-of-8 record was sitting the whole time in a benchmarks file the correction never had
a reason to open.

Two places the correction didn't reach

The correction was folded into four files. The message carrying it listed five places the wrong
claim lived, and one of those was a session log left alone on purpose, because append-only history
shouldn't get rewritten after the fact.

Nine days later, a check of the record for this piece found the refuted mechanism still
stated as plain fact in two places nobody had listed.

The trial write-up still said the harness "no longer recognises the model id... fails to attach
the tool-calling scaffold." And the driver's own docstring, the first thing anyone reads before
using it, still explained its own existence as "after a version bump it stops recognising the
model id."

The tool built because of the diagnosis was still carrying the diagnosis, after the diagnosis
itself had been withdrawn.

Not because anyone ignored the correction. Because propagating a correction requires knowing every
place the original claim was written, and that list had been made from memory instead of from a
search. Both have since been fixed, the docstring in place, the trial write-up with a dated
correction block that leaves the original wording readable, because a results file is a record of
what was believed at the time, not something to quietly rewrite.

The real question isn't how two copies were missed. It's why anyone thought four was the complete
set. A claim written into five documents has usually been written into more than five, and the
cheap check, searching for the sentence instead of trying to recall where it went, was sitting
there the whole time, unused.

What actually carries over

The smallest surface wins twice here, not once. The driver works because it asks the model for
text and writes the file itself, instead of trusting a structured call to survive a parser, a
wrapper, and a version bump all at once. The same instinct applies to how the record of what
happened gets kept. Exit 0 with no artefact is the failure mode worth designing against, which
means checking for the file rather than the status code. Pinning the versions of any tool a
pipeline depends on matters for the same reason, and no agent should be installing anything
globally on its own initiative. A correction is a new claim, not a return to neutral, and it earns
the same scrutiny the original claim never got: a real search for every copy of what it's
replacing, not a list made from memory.

Originally published at thekilted.dev/carried-the-diagnosis.

Never installable

the kilted dev — Sat, 19 Sep 2026 07:51:56 +0000

For three days of real-device testing, the app worked, or seemed to. Every test passed. Every
review came back clean. On a real phone, it had never once been installable, and nothing had ever
said so.

This is a proof-of-concept, not an NHS product. It was built by one technical person working
alongside an NHS dietetic team. No patient data exists anywhere in it. Nothing leaves the
device unless a lookup runs, and even then it sends nothing but the barcode number itself. Clients
had been sending in photos and rough text descriptions of what they'd eaten, and a dietitian was
manually reading that, finding the matching product, and keying it into Nutritics, the accredited
nutrition-analysis software the team already used. Nutritics has its own patient-facing app, Libro,
that would let clients log the food themselves and skip all of that. Libro just isn't enabled for
this region, and turning it on is weeks away at best, if it happens at all. So the trial runs a
smaller experiment instead: making the manual keying-in as fast and unambiguous as it can be, and
doubling as evidence for whether Libro is worth switching on.

That's the frame. The build itself comes down to two decisions that held, and a run of confidently
wrong guesses before the real one.

Why it never asserts a calorie figure

The first thing tried was the obvious move: point a consumer nutrition app at a barcode and let it
fill in the numbers. Cronometer's first four barcode and photo lookups all came back wrong.
Correcting them by hand was painful enough that the question settled itself. MyFitnessPal has kept
barcode scanning behind a Premium paywall since October 2022, so the one app most clients already
had was already a dead end. Only Open Food Facts, a free, open, no-account database of product
names, gave anything back reliably, and even that only names a product. It doesn't know its
nutrition.

Reading digits off a barcode is deterministic and checkable, and that was never where Cronometer
failed. It failed at mapping those digits to the right nutrition record through a crowdsourced
database, and that's the exact spot it got wrong four times running. So the tool draws a hard line
at that exact seam. It reads barcodes. It never writes a nutrition figure. Nutritics, with roughly
1.6 million EUROFIR-accredited foods, stays the only source of truth for what anything actually
contains.

If the tool never claims a calorie number, it can never be the reason someone has to unpick a
wrong one later.

The same discipline runs through barcode validation. A GS1 check digit rejected every single-digit
misread thrown at it in testing, 351 out of 351, which covers the failure mode that actually
happens when a camera misreads a smudged label. But roughly one in ten arbitrary thirteen-digit
numbers will satisfy that check by pure chance, tested against a hundred thousand random numbers,
so "the check digit agrees" only ever means the check digit agrees, not that the barcode is
definitely the right product. An earlier draft of the docs called barcodes "self-verifying." The
project corrected that before shipping, because it was a stronger claim than the tool could
actually back.

Thirty-five foods, not a hundred and twenty

The one insight that shaped everything downstream came from watching how Nutritics itself behaves,
not from the client side at all. Its diet-log workspace works like a spreadsheet, and once a
dietitian has searched for a food and dropped it into one day's log, that same food can be added to
any other day or meal without searching again.

A week's diary might run to something like a hundred and twenty logged eating occasions. Searched
food by food, that's a hundred and twenty searches. But a person doesn't eat a hundred and twenty
different things in a week. The same handful of items repeat across breakfasts and snacks. Counted
by distinct food instead of by occasion, the same week might only need thirty-five searches. That's
the number the design actually optimises, and it's why the data model splits in two: a Food
Library, where every distinct food gets listed once against a short code, and a Daily Log, where
each eating occasion just references a code and a quantity. A dietitian keying a week's diary isn't
doing a hundred and twenty lookups anymore. They're doing thirty-five, and then a lot of quick
copy-paste.

It's a small reframe with a real effect on the person actually doing the keying, and it came from
reading how the existing software worked rather than guessing at what a food diary "should" look
like.

The bugs only a real phone could find

None of this survived contact with a phone unscathed. The front end shipped as a vanilla-JS
progressive web app, deliberately with no build toolchain (this was meant to be a short-lived trial
workaround, not a product, and a bundler is more to maintain than that deserves). Every change
afterward went through a fresh implementer and an independent reviewer before merging, and that
review step earned its keep almost every round.

The export path took the worst of it. The first version silently did nothing when someone tapped
"Download" on an Android phone in DuckDuckGo, because a CDN script, xlsx.full.min.js, that the
service worker hadn't cached could fail to load without throwing anything visible. The fix that followed introduced its own bug:
the downloaded file came back as .bin instead of .xlsx, because the fallback download mechanism
wasn't reliably setting the file's MIME type on that browser. Fixed by building the download from
an explicit Blob instead. Then a third bug, sneakier than either: the app would report "Downloaded"
with nothing actually appearing, because the click handler waited on three IndexedDB round-trips
before triggering the download, and that wait was long enough for the phone to quietly expire the
tap's permission to trigger one at all. Nothing threw an error, no dialog appeared, and the app
still reported "Downloaded" as if it had. The fix was to pre-build the export the moment the screen
opened, so the tap had almost nothing left to wait on.

A fourth export bug turned out not to be this app's to fix. Sharing the file to WhatsApp on Chrome
Android threw a permission error even though the browser's own API had said sharing was possible.
An A/B test settled why: a plain text file shared fine, but the identical share, swapped to the
real xlsx MIME type, failed every time. That's Android Chrome's own share implementation rejecting
the file type at the native layer, and the fix is a fallback, not a workaround: catch that
specific failure and drop back to the working download path.

Of the four export bugs, all found on a real phone, two gave nothing to go on (silent no-op,
nothing downloaded) and were this app's own to fix. The other two threw a visible error, and only
one of those — Android Chrome refusing the file type — turned out not to be this app's bug at all.

Two files, sixty-eight bytes each

The bug that cost the most time wasn't in any of that. It was that the app was never actually
installable, on any browser, on any host, the entire time.

The manifest declared full-size icons, but the two actual icon files behind it were 68-byte
placeholder stubs, one pixel by one pixel. Chrome checks that a PWA's declared icons actually exist
at the sizes it claims. When they don't, nothing in the console warns about it, no error ever
fires, and the "Install app" option just never appears. "Add to Home Screen" still worked, but it
could only ever produce a bookmark shortcut, not a real installed app, so offline never worked
either. The service worker, the manifest, and the caching logic underneath all of it had been
correct from the start.

Before anyone found the icons, two other theories took the blame instead. First the browser: maybe
DuckDuckGo's PWA support was incomplete. Then the host: maybe something about how Namecheap served
the files was the difference from the version that had worked fine on Netlify. Both theories were
plausible. Both were wrong. An earlier comparison between the two hosts had already been written up
as "decisive, host-specific." The actual difference was that Netlify's service-worker cache
had been warm for months and Namecheap's was cold, and that warm cache had quietly been hiding two
unrelated bugs the whole time. What actually surfaced all three bugs at once was a fresh cache forcing every code
path to run for real, on a real phone, for the first time.

A wee sixty-eight-byte file doesn't announce itself as the reason anything is broken. It just
sits there, technically present, while every larger theory takes its turn and falls apart first.

Light, not orientation

One more misdiagnosis is worth keeping, mostly for how the test caught it. A habit had formed of
always photographing barcodes horizontally, and it looked like it mattered: a vertically held
barcode seemed to fail more often. The obvious explanation was that the scanner's decoder reads
horizontal scan lines and could miss a rotated barcode outright.

A direct test said otherwise. A barcode held deliberately vertical, in good light, scanned exactly
as well as a horizontal one. Orientation wasn't the variable. Light was, and the horizontal-photo
habit had just happened to correlate with better lighting, not cause anything on its own. The same
test also cleared the gallery-scan refactor, a different piece of code that had carried an
"unverified on real hardware" flag since it was written. Whatever flakiness it had been suspected
of causing turned out to have nothing to do with it.

What's still open

None of this is a finished product, and it isn't meant to be one. The whole tool is a workaround
for not having Libro, and the plan from day one was to retire it without regret the moment Libro
gets switched on for this region, rather than let it grow into something that has to be maintained.
If the manual keying turns out to be genuinely painful even with a well-built sheet in front of the
dietitian, that's the evidence the trial was actually built to produce.

Either way, what's left isn't a product. It's a barcode reader that only ever claims what it's sure
of, and a debugging log full of things that looked complicated right up until someone found the
actual cause.

Originally published at thekilted.dev/never-installable.

The benchmark that disproved its own result

the kilted dev — Thu, 10 Sep 2026 16:00:40 +0000

An article came past claiming a two-year-old coding-trained model beats a newer general model at
editing structured text that isn't code. Its thesis, verbatim:

A general chat model treats rules as more of a suggestion because it's optimized to sound
helpful. A coding model treats rules as the whole point.

The article contained no measurements. Not weak measurements. None.

That's normally enough to close a review. This project turned it into a test instead, because the
claim has a property most of the genre lacks: it's narrow, it names a mechanism, and it's
falsifiable on hardware already sitting here. "Local models are better than you think" isn't
testable. "A coding-trained model follows structural rules more reliably than a general one" is a
Tuesday afternoon's work.

The test took a few hours. The result was clean, confirmed the article, and was wrong.

The setup

Two models already pulled, no new downloads needed.

qwen3-coder-30b — the coding-trained side
qwen3:8b — the general side

The general side was chosen after checking its chat template directly through the runtime's own
API: a 1,723-character real ChatML template. That check mattered because another model on this box
ships a template that's just a passthrough, which silently degrades everything sent through it, and
picking that one by accident would have produced a large, fake, extremely publishable result.

Five fixtures, one per task type: JSON repair, YAML frontmatter, CSV cleanup, Docker log
extraction, markdown tables.

These models are stochastic, and one run is an anecdote.

So every fixture ran three times, thirty runs in total.

Every fixture gets an objective gate that is full-equality or parse-and-compare, never substring
matching. That distinction isn't pedantry. A substring check asking "does the output
contain the right value" passes an output that contains the right value and three wrong ones too.
This project has had weak tests pass broken local-model work before, twice, so the checkers are
written to be hostile.

Two deliberate choices worth naming. Thinking mode off, because on this box's qwen models it
either hides the answer inside a thinking block or gets rejected outright by the server. Default
temperature rather than the low temperature used for code generation, because the claim is about
how these models behave in ordinary use, and tuning the sampling toward precision would have
quietly answered a different question.

The grader had a bug, and it found it before any model ran

The checkers are the ground truth in an evaluation like this. If the grader's wrong, every number
downstream is decoration. So structured-text-eval.py has a --selftest mode: known-correct answers fed to
each checker, with no model involved, before a single request goes out.

It failed on the YAML fixture, for a reason that would never have been noticed later.

The fixture asks a model to add one key to a YAML frontmatter block without disturbing anything
else, and the checker verified the untouched keys still matched, including a date field,
2026-07-01. YAML's parser doesn't return that as a string. It returns a datetime.date object,
because unquoted ISO dates are a date type in YAML. Comparing that object to the string
"2026-07-01" is always false.

The checker would have failed a perfectly correct answer, on every trial, for both models. The fix
is the str() in the line that guards those untouched keys, still sitting there today:

if fm.get("title") != "Weekly Report" or str(fm.get("date")) != "2026-07-01" \
        or fm.get("tags") != ["ops", "weekly"]:

What matters is when it was caught: before the run, by a check written specifically to distrust the
grading logic, rather than after, when a mysterious YAML failure would have needed explaining and
might have been written up as a finding instead.

The grading logic needs the same adversarial discipline as the thing it grades.

It's code, and nobody reviews it, because it lives in the part of the experiment everyone treats as
infrastructure.

We have written about checks like that before. That check could only
ever come back green, whatever it was pointed at. This one had the same defect running the other
way, and would only ever have come back red.

The first result confirmed the article

Thirty runs, and the coding model came out ahead.

Fixture	qwen3-coder-30b	qwen3:8b
JSON repair	pass	pass
YAML frontmatter	pass	pass
CSV cleanup	pass	pass
Docker log extraction	pass	fail
markdown tables	pass	pass

Every fixture, first run, three trials each (a mark stands for all three — every fixture's trials agreed). Four fixtures agree completely. One doesn't.

That fixture was extracting every ERROR line from a Docker log into structured JSON. The coding
model scored 3 out of 3 there, the general model 0 out of 3.

Look at the shape of that. It isn't noisy. It isn't one unlucky trial. One model succeeded every
time and the other failed every time, on the fixture most about following a structural rule, which
is exactly the mechanism the article named. A clean, consistent, mechanism-matching split, in the
predicted direction.

That's what a real finding looks like. It's also what this one looked like, and it was a bug in
the fixture itself.

What the raw output actually said

The harness records every output verbatim rather than just pass or fail, so the failures were
readable, not just countable.

The general model hadn't mangled the JSON. It hadn't missed lines, invented fields, or drifted
from the schema. Across all three trials it had produced valid, well-formed, correctly structured
output, and put the log-level word inside the message field:

"message": "ERROR Failed to connect to cache: timeout after 5s"

The coding model had stripped it. The expected answer stripped it. So the checker failed the
general model, correctly, against a specification the prompt never actually stated.

The prompt said to extract each ERROR line into {timestamp, service, message}. It never said
whether the word "ERROR" belongs in the message. One model read it one way, the other read it the
other way, and both readings are defensible: the line begins with the level, and whether that
level is part of the message or metadata about it is a genuine ambiguity, not a comprehension
failure.

Reword the prompt, adding one clause: "the message field must contain ONLY the text after the
ERROR level word, not the word ERROR itself." Rerun all thirty.

Fixture	qwen3-coder-30b	qwen3:8b
JSON repair	pass	pass
YAML frontmatter	pass	pass
CSV cleanup	pass	pass
Docker log extraction	pass	pass
markdown tables	pass	pass

The same grid, once the prompt said which part of the line the message excludes. The one cell that failed is the only one that changed.

The gap didn't shrink. It vanished.

The uncomfortable part

The result that ended up under scrutiny was the one that confirmed the hypothesis.

That's backwards from how it usually goes. A disconfirming result gets picked apart, because it's
annoying and someone wants it to be a bug. A confirming result gets written up, because it agrees
with what you already believed and there's nothing left to explain.

Here, the confirming result was the buggy one. The rigour that caught it wasn't scepticism about
the article. Its claim was taken seriously enough to test properly. It was a checklist habit: read
the raw output behind a pass rate before reporting it, regardless of which way the result points.
Without that habit, the write-up would have been "tested it, and the article is right," with a
clean table underneath and no reason for anyone to look again.

The null result is also weaker than it sounds, worth stating plainly. It doesn't show the
article is wrong. The article's actual comparison was a 3B coding model against a much larger
general one, a full generation apart and roughly three times the size difference. Both models
tested here are modern and well above the size where basic structure-following stops being in
question. The claim may be entirely real at the article's scale and simply invisible at this one.
That wasn't tested, because the relevant general model doesn't fit this box's memory budget. So
the finding is narrow: on these two models, on these five task types, with unambiguous
instructions, there's no difference. Nothing about standing practice changed.

One more thing that did not survive

Preparing this write-up meant going back to the raw output file. The committed file holds thirty
results, all passing. The run that produced the twelve-out-of-fifteen split, the one this entire
piece is about, no longer exists: the corrected fixtures were rerun into the same output path and
overwrote it.

The specific output quoted above survives because it was copied into the results log the same
day, by the session that had it on screen. That's a good contemporaneous record. It isn't the
artefact.

Worth noticing which run it was. What didn't survive was the disconfirming evidence: the thing
that was wrong, the run that failed, the state before the correction. Nobody deleted
anything on purpose. The rerun simply wrote to the same filename, because the interesting output
was assumed to be the final one.

The final output is the one you'll never need. It's the one that agrees with the conclusion you
already wrote down. The run worth keeping is the one that made you look twice.

Originally published at thekilted.dev/the-benchmark-that-disproved-itself.

The file nobody reread

the kilted dev — Sat, 05 Sep 2026 18:00:47 +0000

Field notes from a hosting migration for an NHS food-diary proof of concept. A certificate checker
was quietly lying about whether a fix had worked, and two sessions rejected the same hosting idea
twice, four days apart, without either reading the other's reasoning. The move itself was still on
hold when this was written.

This is a proof-of-concept, not an NHS product, built by one technical person working alongside an
NHS dietetic team. No patient data exists anywhere in it. The app stores everything on the
client device and nothing else. The trigger for all of this was mundane: Netlify's free tier caps
a project at 15 credits per production deploy, 300 a month, which works out to roughly twenty
deploys before the meter runs dry. The project was already rationing pushes against that ceiling,
and shared hosting on Namecheap, already being set up for another project the same person runs,
looked like a way out.

Two rejections, four days apart

GitHub Pages came up early, and the project turned it down fast. On the free plan it only serves
public repositories. A private repo needs a paid GitHub plan, and even then the site still
publishes to a public URL with no login wall short of Enterprise. Netlify's free tier deploys from
a private repo to an unlisted one instead, which fit the stated preference to keep the code itself
private. No patient data was ever going to sit in that repo either way. The preference was about
the source, not about anything a patient had typed.

Four days later, GitHub Pages came up again. Not from a different person with a different view. A
coding session working on the project proposed the exact idea an earlier session had already
rejected, because nobody had reread the decisions file before speaking. The occasion was almost
incidental: a different, newer domain had surfaced, already live on GitHub Pages with a working
certificate, and for a moment it looked like an obvious destination. That session looked into it,
then declined it again, for the same reason as the first time. That reason had not moved. It had
simply never been reread. This round did turn up one genuinely new blocker: the food-diary repo
and the newer domain lived under two different GitHub accounts, and GitHub blocks a custom domain
from being claimed by a second account once it's verified on the first, specifically to stop that
kind of takeover. A real finding, arrived at by accident, in the course of re-litigating a question
that had already been settled.

A decisions log nobody rereads is decoration. It was there, it had the right answer sitting in it,
and a session still asked the question again anyway.

A certificate checker that had been lying

The SSL side had its own real bug and its own fake one. The real bug: the domain's nameservers
were pointed at Namecheap's basic DNS, which only handles URL forwarding, not actual hosting.
Switched to the proper hosting DNS and that half resolved. An earlier assumption, picked up from a
web search during planning, claimed the hosting tier included self-service AutoSSL. It didn't.
Triggering AutoSSL turned out to require WHM-level (the hosting provider's own backend) server-administrator access, which an ordinary
shared-hosting account never gets. That was confirmed against Namecheap's own documentation once
the free "Run AutoSSL" button turned out not to exist anywhere in the dashboard. The free certificate that does ship
with the hosting plan has to be issued from Namecheap's own side, and nothing on the customer's end
can force it.

The fake bug was worse, because it looked like a real one for a while. This laptop's own antivirus
intercepts TLS connections and reissues every certificate itself, so anything running on the
machine, curl and openssl included, was reporting the antivirus's own certificate instead
of the real one.

The tool being used to answer "has the SSL fix landed yet" could not tell the truth about it even
in principle.

Checking from a phone worked around it, and later a tool that runs the request server-side,
outside the interception entirely, confirmed it for good. Two support agents tried
force-validating the certificate from Namecheap's own side with no result, and a follow-up ticket
went in to track it as a pure backend wait. Nothing on the customer side could shortcut that part.
The certificate issued not long after, and a server-side check confirmed clean HTTPS.

Two things the plan hadn't accounted for

First: SSH turned out to be off by default on shared hosting, though the migration plan had been
written without confirming that. The fix was self-service, a control panel option called "Manage
Shell," which gives no hint it's SSH, listening on a non-standard port rather than the usual one. Second:
pushing to the new remote from an automated shell hung indefinitely with no error, because a global
credential setting that keeps two separate GitHub accounts from colliding also blocks the plain
per-repo push path unless a repo-level override is set. That global setting has to stay exactly as
it is. The only new part is the missing per-repo override.

A smaller trap surfaced once both hosts were finally level with GitHub again: the control panel's
Git tool has an "Update" button that only saves configuration. It doesn't fetch, pull, or deploy
anything, and those actions live on a separate tab entirely. For a repository that was deliberately
cloned through the control panel's own "Clone a Repository" tool, with a specific option left unchecked to
preserve its automatic deploy hook, even that tab's own "Update from Remote"
option turns out to be a no-op. It's a push target, not a pull source. Reading the UI correctly
took longer than reading the DNS panel had.

Deleting a task is not a local edit

One step in the migration plan retired Netlify outright once the new host was live. Once Namecheap
turned out to be an interim stop rather than the final destination, the task was dropped, which
seemed like a small, contained change. It wasn't. That same task had been referenced as a
precondition in five other places across the plan, and one of those was dangerous on its own: a
later step told a future session to delete the Netlify credit-cap constraint from the project's own
guide file, once hosting had moved. Hosting hadn't moved. A session following that instruction
literally would have silently removed a still-live limit from the record that exists to track it.
All five references were found and corrected before that could happen.

What actually changed the private-repo rule

The preference that had driven both GitHub Pages rejections wasn't overturned by a technical
argument. It was relaxed on purpose, once the user actually named a real gap instead of waving it
off: the diary's free-text fields, a description typed next to a photo, have no structural way to
stop someone entering something identifying. Going public meant accepting that gap rather than
pretending it away. In the user's own words, it was "hard to guard... loose enough to say go
public." The account-split blocker from the second rejection turned out not to matter here either,
since the plan kept the repo under its existing account rather than moving it to the newer brand's.

A bigger swap, investigated instead of just refused

A smaller question, whether a branded subdomain could dodge a shared hosting username, led
somewhere larger: making the newer, higher-traffic domain the account's designated "Main Domain,"
on the assumption that the label itself would grant it more subdomain headroom. That assumption
didn't survive a check. The account's subdomain quota is shared regardless of which domain holds
that label, so an ordinary addon domain gets exactly the same headroom. What an
actual swap would require turned out to be much bigger than the label suggested: a different
project's own blog would have to leave its free, automatic hosting for manual shared-hosting
deploys with none of that automation and the same multi-day certificate process already documented
here. That's not a call this project gets to make about another one. It was declined without
touching anything, and the idea went out as two information-only notes to the two other projects'
own owners, for them to decide on their own terms.

How it actually landed

The reversal, when it came, was decided outside any of these sessions, by the one person who owns
all three projects at once: Netlify was retired entirely, and the food diary moved to GitHub Pages
after all, kept under its existing account rather than a newer pseudonymous one. That keeps a tool
built under a real identity separate from a brand built under a pseudonym, and it sidesteps the
cross-account domain restriction outright rather than working around it.

The account question that had paused the move turned out simpler than the plan in flight at the
time suggested. Rather than standing up a new account, the existing one kept its history and its
repos exactly where they were and simply changed its own display name. Same account throughout,
just renamed partway through. The migration is complete now. Both old hosts are fully retired, and
nothing is left dual-hosted.

It ended without a clever fix. Two things went wrong here that weren't code. Both were caught the
same way, by someone who actually checked instead of assuming. A certificate check that had every
reason to look trustworthy was quietly answering a different question than the one being asked. And
a written decision only held once someone actually reopened the file before speaking a second time.

Originally published at thekilted.dev/nobody-reread.

The verdict that survived its own correction

the kilted dev — Sun, 30 Aug 2026 06:15:27 +0000

This project started with a conclusion already sitting in its lap. In June, an earlier stretch of
work on the same machine had tested whether a local model could handle iterative coding: writing
files, running tests, fixing what broke, without a human shuttling text between a chat window and
an editor. The verdict was no. Not viable. That was the finding the whole project was built on.

On 4 July, someone re-tested it. It fell over in about two minutes.

The interesting part isn't that the verdict was wrong. It's that the verdict was right. It had
been stated precisely, with its own explanation attached, and none of that survived the trip into
the new project.

The re-test

The task was a word search game: three files, a 10x10 grid, eight words hidden horizontally,
vertically or diagonally, click-and-drag to select, correct words struck through in the list, a
New Game button. Eight scored criteria. The eighth is the one that matters: ask for one follow-up
fix (make found words green) and check it lands without breaking the drag highlighting that
already worked. That regression check is what had killed every previous run.

Run through little-coder, a small harness that gives the model real write and read tools instead
of asking it to print code into a chat window, against qwen3-coder-30b served by Ollama, the
result was 8 out of 8 in 2.1 minutes. The follow-up fix took another 30 seconds. All of it verified
in a real browser with simulated drag events, not taken from the model's own account of what it had
done.

The same benchmark against the smaller qwen3.5-9b scored 7.5 out of 8 in 44.1 minutes. It lost
half a point on reverse-placed and diagonal words: no reversed-string check, an off-by-one in the
highlight loop. It still passed the regression test, the first local setup here to manage that.

Model	Score	Time
qwen3-coder-30b	8/8	2.1 min
qwen3.5-9b	7.5/8	44.1 min

The quality gap was only half a point; the time gap was a full 42 minutes.

Two honest marks sit against the 8 out of 8. One transient failure died with "Stream ended without
finish_reason" and an identical retry worked. On criterion 8, one green pixel was never confirmed,
because the preview tab had collapsed to a zero-width viewport and inline styles had stopped
applying: impossible in a real browser, so it was scored as a harness fault rather than a code
fault. That was a judgment call, made and written down as one at the time.

What this project wrote down that evening

The June verdict was a harness problem, not a model problem.

It's a clean line. It reads as an insight. It went into the project's working memory as a headline
finding, was quoted straight into the active constraints file, and shaped everything scoped
afterward: if the harness is the variable that matters, you invest in harnesses and stop shopping
for models.

The investment was right. The claim wasn't, and nobody checked it for two weeks, because a
conclusion that's useful and turning out well doesn't feel like it needs auditing.

What the original verdict actually said

The June work had been handed over as a document. Reading it properly, two weeks late, the verdict
is one row in a table of six use cases:

Use case	Locally viable
Claude Code CLI backend	✕ broke — models misinterpret Claude Code's own system prompt format, not a config problem
Multi-file document extraction (LM Studio + Gemma 4 E4B, 8192 ctx)	✓ met — works well, the best multi-file result of anything tried
Single-file extraction, agent + filesystem (Goose + Ollama, qwen3.5-9b)	✓ met — works, but slow: 62 seconds per file
Iterative coding (any local chat UI)	✕ broke — not viable, and structural: five failure modes, not a fixable bug
Single-turn code generation (Goose or LM Studio, max context)	✓ met — viable as a baseline handoff into a human editor
Quota fallback for real sessions (fcc-server)	✕ broke — routed to Gemini instead, local wasn't trusted as the fallback

June's headline verdicts, all six, from the same document that scoped iterative coding to "any
local chat UI." The row that got re-tested is one of six, not the whole page.

The scope is in the verdict line. Not "local models cannot do iterative coding." Local models
cannot do iterative coding in a chat UI, which is a claim about an interface.

Below it, five structural failure modes are listed. Stateless regression, where a fix in turn N
gets silently undone in turn N+1: observed, in that same word search task, as a drag highlighter
fixed and then broken. Overconfident reporting. Silent context truncation. Effective context not
matching advertised. No cross-session persistence.

And then, closing that section:

Why Claude Code avoids 1–2: tool-mediated verification per step, persistent conversation
context — architecture, not raw model capability.

June had already worked out that the constraint was architectural. It's written down, in the
document that was handed over, in bold, five weeks before the July re-test that "discovered" it.

July didn't overturn June. July supplied the harness June had described, and measured what
happened.

Where June was too optimistic

Of the five failure modes, June expected a harness to handle the first two: stateless regression
and overconfident reporting, both covered by tool-mediated verification and persistent context.

The first half held. The second didn't.

Overconfident reporting is a model claiming completion when a feature doesn't actually work, and a
harness only catches that if its tests are adversarial. This project has watched a weak test
launder a false claim into a green, twice. A redaction check passed with
assert "84210.55" not in out while a partially redacted amount, 842[AMOUNT], leaked straight
through it. A loader passed nineteen of nineteen with a leaked loop variable, masked by a fixture
that only ever exercised one file.

Real file tools, persistent context, tests that ran green: both greens were wrong.

So June wasn't simply more careful than its own summary. On this one point it was optimistic, and
the correction only turned up once this project built the harness June had described and ran it
long enough to get burned.

What actually went wrong

Four things actually changed between June and July, worth stating because it means the re-test was
never a controlled experiment. The model changed. The harness changed. The runtime flags changed
(July ran with flash attention and a quantised key-value cache, June predated both). And the prompt
changed, because June's prompt text was never preserved and had to be rebuilt from scratch.

That last one isn't carelessness. The June raw evaluation file survives, and it's 809 bytes: a
findings summary whose entire task description is one line. There was no prompt left to preserve by
the time anyone went looking. Reconstruction was the only option on the table.

But the four variables are a side issue. The real failure was compression.

What travelled from June into this project was the phrase "iterative coding not viable." The scope,
any local chat UI, didn't make the trip. Neither did the mechanism: architecture rather than raw
model capability.

Then someone re-tested the compressed form, found it wanting, and corrected it into a claim broader
and less accurate than the original. A precise, well-reasoned, correctly scoped verdict went in one
end and came out the other as a slogan, and the slogan is what the project ended up arguing with.

The verdict, June: Iterative coding — not viable (structural, see the five failure modes) — any
local chat UI. Scoped to an interface, with its own mechanism named: architecture, not raw model
capability.

What travelled: "Iterative coding not viable." The scope and the mechanism both stayed behind.

What got re-tested: "Harness, not model." Broader and less accurate than the verdict it
replaced, and the claim this project ended up arguing with.

The handover document's own header says it plainly:

Nothing here needs re-deriving — it was all tested hands-on.

This project re-derived it anyway, and ended up with a worse version.

The part that only appeared while writing this

Writing this piece meant going back to the June sources, and the first attempt to find them failed.
The active constraints file listed the handover document under a path in the other project, the one
that had written it. It wasn't there. Searching that project turned up nothing.

The conclusion drawn from that was that both June primaries had been deleted. That conclusion went
into a correction to this project's own files, into an early draft of this piece, and into a
request asking another project to recover them from git history.

The document was sitting in this project's own repository root. It had been there since the first
commit. It had been written elsewhere and moved across at handover, exactly as its own header says.
The raw evaluation file had moved into a shared knowledge vault during a migration weeks earlier,
and was just as findable. The other project located both files in minutes, and mentioned, almost as
an aside, that it has no git history at all, so the recovery being asked for could never have worked
anyway.

One failed lookup at one stale path became "gone from disk," and a fair amount was built on top of
that before anyone checked the obvious place.

That's the same mistake this piece is about, made while writing about it.

An absence, asserted from a single negative result. A conclusion, drawn from a summary (the path in
the constraints file) rather than from the thing itself.

What changed

The finding was rewritten to what the evidence actually carries. The harness thesis is June's, not
this project's, and it was right. What this project added is confirmation and a measurement. The
broad version, "harness, not model," is now marked as over-stated, because it took this project's
own later evidence, not June's, to find the one place a harness alone wasn't enough.

The practical guidance didn't change at all, which is the uncomfortable part. Invest in the
harness. Verify with real tool output. Keep the local tier for bounded work. All still correct, all
still what this project does. A claim can be over-stated and still point the right way, and that
combination is exactly what stops anyone from checking it again.

Two rules came out of this, both narrow and mechanical. Before re-testing an inherited verdict,
read the primary document, not your own summary of it: an active-constraints file strips scope and
exceptions by design, because stripping detail is what it's for. And an absence needs more than one
failed lookup, especially when the path you searched came from the same compressed source you're
already re-deriving from.

What travels between sessions, projects and people is the conclusion. The conditions that produced
it stay behind. A conclusion without its conditions is a rumour with a number attached.

Originally published at thekilted.dev/the-verdict-that-survived-its-own-correction.

Protected nothing

the kilted dev — Sun, 23 Aug 2026 13:52:56 +0000

Field notes from a proof-of-concept food-diary tool built alongside one NHS dietetic team,
not an official NHS product, no patient data anywhere in it. A privacy guardrail blocked its
repository from going public. Traced through the code, it protected nothing that going public would
have exposed.

A rule sounded exactly right, protected exactly nothing, and stood as the one thing blocking real
work, until someone traced it back to what the code actually does.

The rule was simple. Before the food diary's repository could go public, it needed a guardrail
against identifiable data leaking through the diary's own free-text fields, chiefly a description a
client might type next to a reference photo. Someone could type their own name into it without
thinking, to help remember whose entry it was. That risk was real, agreed on at the time in the
project's own words as "hard to guard... loose enough to say go public," and it sat there as the one
condition standing between the repository and a public GitHub Pages migration everything else was
ready for.

The condition held for a day. Then, traced through the actual code rather than argued about again,
it didn't survive the first look.

Where the description actually goes

The description a client types is captured in the app, stored as a note on a log entry in IndexedDB
on the device, and exported into a spreadsheet the dietitian handles. It is never committed, never
pushed, never served. The repository's visibility has no effect on it whatsoever, because it was
never in the repository to begin with. A guardrail on that field would have protected data that
going public put at no risk at all.

A second, independent reason cut just as deep. The app ships with no build step at all, just
straight unminified JavaScript, so its entire source has been downloaded verbatim by every single
visitor to the live site since the day it launched. Repo visibility was never protecting the
application code either, because the browser was already handing that code to anyone who asked. The
"code stays private" preference behind two earlier hosting rejections had, for the part of the
project a visitor could actually reach, never once been true.

The place nobody checked

If the free-text field was never the risk, going public should have been safe from day one. It
wasn't, quite, and the reason was somewhere else entirely: roughly forty tracked files sitting
outside the application folder, never served to a visitor and shielded from view by nothing but the
repository itself being private, among them migration notes, deploy docs, session logs, and a stray
config file. Grepping through all of it for anything infrastructure-shaped turned up exactly one
genuinely sensitive string: a hosting-account username, and not even this project's own account. It
belonged to someone else's.

	In the repo	Repo privacy hides it
Free-text descriptions	no	no
Application source	yes	no
~40 files outside the app folder	yes	yes

Three things the repository question could have been about. Only the last is both tracked in the
repository and hidden by nothing else, so it is the only one that going public actually exposes.

The thing everyone had been scared of couldn't actually happen. The thing worth being scared of
had been sitting in a folder nobody had opened, the whole time the guardrail argument was running.

A fix that doesn't depend on remembering

The fix that shipped doesn't ask anyone to remember anything. Rather than scrub the sensitive files
out of a single shared repository and trust every future session to keep scrubbing, the project
split in two: one private repository holding everything, full history intact, and one public
repository synced from it that holds only the application directory's own contents. Nothing outside
app/ can leak, because nothing outside app/ is ever in the public repo to begin with.

Purging the sensitive files from one shared repository's history was considered and turned down on
purpose. NOTES.md gets written every session, which means every future session is a fresh chance
to type an identifier back into a file that's now public, and a miss like that is silent and
unrecoverable the moment the repository is cloned. The split doesn't have that failure mode. There's
nothing to miss, because the sensitive files are never in the repo that goes public in the first
place.

One faster fix was on the table too, and turned down for a completely different kind of reason.
GitHub offers private-repo Pages hosting on its paid tier, for around four dollars a month, which
would have made the whole question go away without touching a single file. Food diary's own author
turned it down outright, on the spot, with two words: "not pro." No technical argument followed it,
because none was needed. It was a spending decision, not an engineering one, and the project didn't
pretend otherwise.

What the guardrail actually protects now

The free-text field is still a real concern. It existed before this repository was ever considered
for public release, and it will keep existing after, because it's a property of what the app lets
someone type, not of who can see the code. It just isn't a hosting decision, and treating it as one
is the mistake this piece is actually about. It has its own design question to answer, on its own
schedule, and the fix for it won't be a line in a hosting migration plan.

The pattern generalises past one repository. The project specified a control against a threat that
sounded exactly right and had never been traced to the actual place the data flows through. It still
held up real work, however briefly. And the thing worth guarding against the whole time was sitting
in plain sight, in a directory nobody had thought to check, because the argument had already used up
everyone's attention on the door that was never actually open.

Originally published at thekilted.dev/protected-nothing.

Invoked, not executed

the kilted dev — Thu, 20 Aug 2026 07:35:56 +0000

A deep-research request to the top-tier model tore through three consecutive five-hour usage
windows, the rolling quota Claude enforces before a session has to stop and reset, to answer a
single question.

Usage window	Burned	Time to burn it
Window 1	54%	about two minutes
Window 2	100%	about four minutes
Window 3	40%	finishing the job

Three consecutive five-hour usage windows against one question. That adds up to 194% - nearly two
entire windows - and the first two were gone inside six minutes.

Nobody had done anything unusual. Someone had just asked the model to look something up properly.

The routing policy this machine runs on has a rule for
exactly this shape of task: work that's one step (search, fetch a source, extract a claim) and
objectively checkable belongs on the cheapest tier that can do it, not on the model that costs the
most per token. Deep research is built entirely out of that shape: search, fetch, extract, feeding
a synthesis step that does need judgment. The rule already existed. It just never had a chance to
apply.

Why the rule never fired

A routing policy only engages at a decision point: the moment a session has to choose which tier
does the next piece of work. Most of the policy's machinery assumes that moment exists. A plan gets
broken into tasks, and each task is a place to ask who does this.

A direct request, "deep-research this," skips all of that. It's answered as one thing, by whichever
model receives it, in whichever way that model has of doing research. There's no task list to route
pieces of. The search, the fetching, the extraction, the verification all happen inside a single
reply from the most expensive available model, because nothing ever asked whether the next step
needed to be there.

The fix follows from the gap. Route research-shaped requests through a harness that does the
decomposing on the way in, whatever words were used to ask. That covers all three shapes the ask
takes: an explicit "research this," a model deciding on its own that a claim needs checking, or
something as soft as "I'm not sure what's actually true here." All three land in the same place if
answered directly. All three should hit the harness instead.

The test: 3 phrasings, 2 tiers

Having independently found an adjacent problem (a third-party tool doing something similar had its
own bug in the budget cap meant to prevent exactly this kind of burn), a sibling project was asked
to check whether the routing fix triggered in practice. Six blind subagents, no shared context, one
prompt each: three ways of asking for the same research, crossed with two model tiers. The topic
was held constant, the enforcement status of the EU AI Act, so how the question was worded was the
only deliberate variable.

The intended way of measuring it failed. A subagent's own transcript file stays zero bytes even
after the run has finished, so nobody could simply read off which tools had been called. The
measurement fell back to asking each subject afterward what it had done, cross-checked against a
tool-call count the harness reports and the agent has no hand in writing.

The harness fired in half the runs. That looked like a pass. It wasn't one.

What "invoked" was hiding

Every run that triggered the harness received an instruction to call Workflow, the dispatcher
that fans the research out across sub-agents. Workflow is not available inside a delegated
subagent. Each of those runs found that out mid-task, noticed the gap, and quietly did the research
a different way: by hand, with the tools it did have. Nothing crashed. Nothing logged an error. The
final answer looked like a normal research answer, because it was one. Just not the one the rule
was written to guarantee.

One of the six was candid about exactly what was lost in translation. It had improvised its own
version of the harness's fan-out step, sending several of its own sub-agents out to search in
parallel, and reported the results converging as if that settled things. Asked afterward what its
own five-agent agreement was worth, it corrected itself before anyone pushed back:

The convergence I reported between agents is corroboration, not adversarial verification, five
agents drawing on an overlapping source pool can agree and still be wrong together.

The step that was skipped wasn't a decoration. It was the one part of the harness whose entire job
is to catch exactly that kind of agreement that isn't evidence. A rule checking whether the harness
was invoked would have called that run compliant. The harness had not, in the sense that mattered,
run at all.

Run	Harness invoked	Fan-out executed
Explicit ask (sonnet)	met	broke
Explicit ask (opus)	met	broke
No keyword (sonnet)	broke	never established
No keyword (opus)	broke	never established
Soft / vague (sonnet)	met	broke
Soft / vague (opus)	broke	never established

Invoked in three of six runs, executed in none. The three that invoked it reached for Workflow,
which a delegated subagent cannot call, and fell back to research by hand; the three that never
invoked it never got far enough to find out. The right-hand column is empty either way, and that
column is the one the rule was written to guarantee.

The number that looked best was the least trustworthy

The one wording that worked reliably, the explicit "deep-research this," didn't work because the
routing logic held. One of the successful runs said, unprompted, exactly why it had picked up on
the request. The words read to it as plain English, and it matched them to the right tool on its
own initiative. In its own account, the phrase was "hyphenated as a verb phrase," with "no slash,
no backticks, no skill syntax."

Nothing in how it was written signalled a tool call.

Rename the tool, or ask for the same research without those two words, and the coincidence that
made the easy case look solved stops holding.

The case that matters is a serious, specific question asked with no research vocabulary in it. That
is the shape almost every real research need takes, and it missed on every model tested. That's the
gap worth worrying about.

What changed, and what didn't

The rule was rewritten to check for evidence the harness had actually run: a count of sub-agents
dispatched, a tally of claims independently verified, rather than trusting that the right words had
been said. That closes the failure mode the test found: a compliant-looking run that quietly did
none of the work the compliance was meant to certify.

It doesn't close the other one. The wording that reliably works is still the wording that happens to
match the tool's own name. The question that matters, the one with no research vocabulary in it,
misses the harness on both models tried.

Fixing what gets checked was the easy half. The other half is getting the harness to notice it is
needed at all, from a question that never says so. That one is still open, and the coincidence that
made the easy case look solved is the reason to expect it is the harder one.

The rule that got rewritten, extracted and de-identified: the routing-policy
repo.

What really fits in 8GB VRAM

the kilted dev — Tue, 18 Aug 2026 19:03:52 +0000

Four VRAM numbers that looked wrong on an 8GB laptop, none of them a bug: what dxdiag actually
reports, what the desktop holds back before a model even loads, and the two fit rules that came
out of chasing all four down.

Every question about running models locally reduces to one question: will it fit. Which model,
which runtime, which quantisation, whether the thing is usable or a slideshow: all of it comes
down to that one word. The trouble is that the number deciding the answer is never the number
anything shows you.

This is a laptop with an RTX 5060 and 8GB of VRAM. Four separate figures turned out to be
measuring something adjacent to what actually mattered. Not one of them was wrong. Each was
reporting a real quantity. Just not the quantity that decides whether a model loads.

The first number: shared memory

› dxdiag /whql:off /t <outfile>

Run that on this machine, open the Display tab, and it reports the graphics card as having
24,144 MB of display memory. Twenty-four gigabytes, on a laptop sold as having eight. Directly
underneath, the same panel breaks it down:

Display Memory:    24144 MB
Dedicated Memory:   7899 MB
Shared Memory:     16245 MB

7,899 plus 16,245 is 24,144. The arithmetic is honest. Display memory is the dedicated VRAM on the
card plus a slice of system RAM the driver treats as overflow, half the machine's 31.7GB, the
standard Windows allocation. Only the dedicated figure decides whether a model loads.

Two details make the trap worse than a single mislabelled number. The shared pool isn't the
graphics card's at all, and every display adapter on the machine claims it: the integrated Intel
graphics, the RTX 5060, and two DisplayLink devices all report the same 16,245 MB. Add up what the
machine appears to have and the total overstates the truth several times over. The dedicated
figure isn't perfectly stable either: a model-fitting tool run on the same box reported 7.96GB
where dxdiag says 7,899 MB, close enough to be the same answer, far enough apart that a fit
calculation cutting it fine will disagree with itself depending on which tool it asked.

The two pools behave completely differently under load. VRAM is fast and finite. The shared
region is system RAM wearing a costume: spilling into it doesn't fail, it just gets slow, and a
slowdown is worse than a failure because a failure tells you something is wrong while a slowdown
just lets you keep going, wondering why everything feels off. A 9B model at 32K context sits
entirely in VRAM at 5.9GB and generates 51.5 tokens/sec. Push the context to 64K and it needs
7.5GB, spills to an 18/82 split, and drops to 22.6 tokens/sec. Same model, same machine, same
quantisation. One setting, less than half the speed.

None of this requires taking anyone's word for it: the breakdown is four lines down the Display
tab on every Windows machine there is.

The second number: your desktop already has some of it

The dedicated 7.9GB isn't yours either. Under normal use (browser open, editor open, the desktop
doing its job) somewhere between 2 and 3.5GB is already spoken for before a single model loads.
Not reserved in a way you can reclaim by closing one app. Just gone, in the ordinary course of
using the computer for anything other than inference.

The spread matters as much as the figure. Measured on this machine with a browser, a notes app, a
spreadsheet and around thirty other things touching the GPU, it sits at almost exactly 2GB. On a
busier day it has been half again that. On the quiet afternoon the benchmark below was recorded,
it was under 1GB.

That shows up in the benchmark record as an awkward footnote. Here is the same 36B
mixture-of-experts model, three ways, every row measured at the same 32K context:

Configuration	Gen t/s	VRAM	When you can use it
llama.cpp, 8 expert layers on GPU	27.1	7.4GB	Apps closed only
llama.cpp, 6 expert layers on GPU	26.3	6.4GB	Recorded safe default
Ollama, automatic 80/20 CPU/GPU split	23.4	n/a	Default behaviour

Same day, same machine, flash attention on, KV cache quantised to q8_0. Ambient GPU load during
measurement was unusually low, between 0.3 and 0.8GB; normal desktop load is around 3.5GB.

That fastest row leaves around 800 MB of headroom, and it was measured on that unusually quiet
afternoon. Under normal desktop load it doesn't fit at all. So the best number in the table is one
that can't be reproduced while actually using the computer, and the benchmark file says so
directly rather than quoting the headline figure and moving on.

The two rules that came out of it

After enough of this, the fit question collapsed into two rules that are worth more than any
individual measurement.

Dense models: the file has to be about 7GB or smaller at 4-bit quantisation, and it has to run
entirely on the GPU. If it spills, the speed lost is more than a bigger model would have gained.

Mixture-of-experts models: the file has to fit in system RAM (31.7GB here) with the experts
offloaded to the CPU. VRAM only needs to hold the attention layers and the key-value cache.

The second rule produces the result that governs everything this project runs: a large
mixture-of-experts model fits comfortably where a much smaller dense model doesn't. A 30B MoE
model with a 19GB file runs at 32.9 tokens/sec generation on a 69/31 CPU/GPU split, at 16K
context. A 14B dense model is simply ruled out, and so is the default pull of a well-known 9.6GB
dense model, on the dense rule alone.

That context qualifier isn't decoration. Speed figures are only comparable within a single
context size, which is why every number here carries one.

Parameter count isn't the constraint. Architecture is: a model three times the size fits because
of how it's shaped.

That's invisible from any spec sheet that lists parameters and file sizes. Not everything needs a
hands-on trial to rule out, either. An 80B MoE model was ruled out at the research stage: its
smallest 4-bit quantisation is 38.4GB, more than the machine has RAM for at all. No test required.

The third number: the config that was right when it was written

Six weeks in, a model that had been working for over a week stopped loading. Every attempt died
with a CUDA out-of-memory error, and the first answer was wrong.

The server log said it had offloaded 49 of 49 layers to the GPU, which reads unambiguously as "it
tried to load the whole thing into VRAM instead of doing the CPU split that had been working." A
satisfying scheduler bug, except it wasn't what happened. Reading the full memory breakdown
instead of the first suspicious line showed the actual buffer allocation: 3.2 to 4GB in the GPU
buffer against 13.5 to 14.3GB in pinned host memory. The CPU offload was correct the whole time.
That log line's wording misled; it wasn't evidence.

The real cause turned up in the runtime's own fit-calculation logging, and it wasn't a bug either.
The fit calculation had correctly worked out that 43 of 49 layers would overflow to host RAM, and
projected 5,123 MiB of VRAM use against 6,196 MiB free. A plausible-looking fit.

What had actually happened was that the model's local alias had been silently recreated eight
days earlier, inheriting the project's default context setting of 32,768, and this particular
model had never been verified at that context. It had been benchmarked at 16K.

The evidence was sitting in the original benchmark file the whole time, contradicting itself. The
prose describing the working run states the context parameter as 32,768. The speed table further
down labels the identical CPU/GPU split "@16K." Both written the same day, about the same run.
Nobody reads a benchmark file for internal consistency after writing it, so the contradiction sat
there quietly. A single-variable test settled it: at 16,384 the model loads and generates every
time; at 32,768 it fails every time. The fix was re-pinning the alias: a configuration
correction, not a code change.

The fourth number: free memory doesn't predict this

The most useful part of that failure is the bit that contradicts intuition. The out-of-memory
error didn't depend on how much VRAM was free. Retested with the model fully unloaded and 6.1GB
available, it still failed, ruling out the obvious explanation that some other app was
transiently holding memory. At 32K context, the model's key-value cache plus its compute buffers
exceed what fits after the CPU offload split, regardless of what's free. Headroom isn't the
variable. The shape of the allocation is.

That has a practical edge: a fit-estimating tool used on this machine calculates against total
VRAM rather than free VRAM, so its recommendations need hand-adjusting for whatever the desktop is
already holding.

What this adds up to

Four numbers, four different ways of being adjacent to the truth. Windows reports memory that
includes a region you don't want to touch. The desktop silently holds a few gigabytes of what
remains. A log line describes full GPU offload while performing a CPU split. A config file records
a context the model was never tested at, and a fitting tool measures against a total nobody
actually has.

None of these is a bug. Every one is a real measurement of something, presented plainly, by
software with no intention to mislead. They're wrong only in the sense that they answer a question
next to the one being asked, which is why the two fit rules are worth more than the measurements
that produced them. The rules survive a driver update, a runtime version bump, a model release.
The numbers don't, and the ones that look most authoritative are usually the ones measured on an
unusually quiet afternoon.

Originally published at thekilted.dev/what-really-fits-in-8gb-vram.