DEV Community: hyuga

You are in the cockpit now. Nobody gave you instruments.

hyuga — Sun, 30 Aug 2026 11:36:38 +0000

I noticed recently that I have stopped typing.

The agent writes the code. What I do is look at what comes back and press y. Dozens of times a day.

And — you probably recognize this — somewhere around the tenth one, I stopped reading.

"Same as the last one." It usually is. But if one of those fifty had touched production, would you have caught it? I wouldn't have.

That is not supervision. That is a rubber stamp.

Am I flying this, or am I a passenger?

Here is the thing I keep coming back to.

A pilot does not see the outside world directly. They see instruments: fuel, damage, remaining ordnance, whether the limiter is engaged. Reaction time matters, but knowing what is happening right now comes first.

An aircraft with broken instruments is not the same aircraft with a small problem. It is a different aircraft.

So: everyone running an AI agent is already sitting in a cockpit. You don't move the controls yourself; the machine acts and you decide at the moments that matter. That is a pilot's job description.

You are just sitting there without an instrument panel.

What you can see is one approval dialog at a time. What you cannot see: what this session has done in total, how far into dangerous territory it has gone, what it got wrong last time.

I wanted that panel, so I built it.

npx @hyuga/airframe install

The panel

It says nothing most of the time. Silence is the correct state. You look when you want to look.

$ airframe status

sortie   2026-08-30T07-07-43-412Z-18aa80
form     strike / fire
limiter  0
wingmen  0
fuel     0 (no budget set)
mounted
  + airframe      the vessel
  + redline       limiter — counts the sortie, not the call
  + habit         learns from the corrections you make by hand
  + carbon        keeps the draft about to be written over — cruise only
  + groundtruth   completion gate — called from your code, not from a hook
  + llm-safe-sql  runs the write, measures it, rolls back — from your code

The vocabulary is aviation, partly because I like it and partly because it turned out to be the right shape:

sortie — one run of work. What the last one left unfinished survives into the next
limiter — how much irreversible ground this sortie has covered
wingmen — how many subagents were launched
fuel — how many moves are left

limiter 0 means nothing dangerous has happened yet. When it climbs, the machine is pushing into something.

A limiter is not there to slow you down

This is the part I most want to land.

"Guardrails" sounds like friction. It is the opposite.

The limiter is what lets you open the throttle.

Without instruments there is exactly one safe way to operate: go slow, always. And that is what everyone does — confirm every step, never look away. Sitting in a fast machine, driving at walking pace forever.

With a number on the panel, "safe to let it run" and "hold the reins" become different situations. Zero? Leave it alone. Climbing? Pay attention. It is a part that makes you faster.

My own instrument lied to me

Then the instrument lied to me.

While publishing this repository, the limiter — threshold 3 — reached 20. More than six times over.

I looked at what it had charged for. Every single one was a search.

grep -n "npm publish" .github/workflows/release.yml    # +3
grep -rn "rm -rf" packages/                             # +3

Looking for the string npm publish scored the same as actually publishing. The limiter was matching a regex against the whole command as one blob of text.

The problem is not that it was too strict. The problem is that once you learn 19 of 20 charges are noise, you skip the 20th too. And the one you skip is the one that mattered.

A broken instrument is worse than no instrument, because it lets you feel watched while nothing is watching.

Fixed the same day:

Command	Before	After
`grep "npm publish" README.md`	3	0
`grep "npm publish" README.md && npm publish`	3	3
`npm publish`	3	3

The arguments to grep are text, not actions. But sed, xargs and node -e are still charged on what they contain — their arguments are commands, and nothing here can tell whether that string is about to run or about to be printed.

Charging when the answer is genuinely unknown is design. Charging when the answer was obvious was the bug.

Past the limit, it does not stop you

When the score goes past the threshold, the machine does not stop. It says:

This sortie has spent 20 against a limit of 3. You are flying it, so it is your call — but say out loud that you are past the edge before going on.

It does not stop because there is no guarantee the machine is more right than the person in the seat. The call belongs to the pilot.

One exception: set AIRFRAME_AUTONOMY to the reason you are running unattended — a loop, a timer — and then it will halt. With nobody in the seat there is nobody to advise.

Advise when someone is aboard. Halt when nobody is. The condition for releasing the limiter is decided by the seat, not by the machine.

Converging and diverging are opposite jobs

One more thing you switch by hand.

airframe mode strike   # implement it, fix it, ship it
airframe mode cruise   # draft, design, decide what to build

While you are implementing, you want the gate: did those rows actually land? Stop me if not.

While you are drafting, the same gate is a wrist being grabbed mid-sentence. An idea that has not taken shape yet gets reported as "unmet." The better the idea, the earlier it closes.

Opposite jobs. Tune for one and you break the other.

And the machine never guesses which one you are in. Guessing wrong is the same bug as not having the two. You type it. It takes a second.

In cruise, the draft keeper runs: when the agent is about to overwrite a file git does not have, the previous version is kept.

$ carbon list
2026-08-30T07-32-32-255Z-0bef6e1d.md    1832 bytes

Files git already tracks are never copied — git show gets those back. Only what git is not keeping. For anyone who has had an agent overwrite a draft and found git diff empty.

Three CI files that had never run once

One more thing that actually happened.

When I merged the parts into one repository, I moved each part's .github/workflows/ along with it, into the package folders. GitHub only reads .github/ at the repository root.

Three CI files. Three release files. After the merge, not one of them had run.

And nothing says so. Nothing is failing. On the GitHub UI, "zero failures" and "zero runs" look exactly the same.

Which is the general form of what the completion gate does: at the moment something claims "done", go re-fetch the actual state.

groundtruth verify --probe "psql -tAc 'select count(*) from t where batch=123'" --count 45

Told 45 rows landed? Go count them. Zero? Stop there. No LLM, no API key — the raw output of the command you gave it, shown as evidence.

Where this actually pays off

Honestly: this thing helps in a narrow set of situations, and it is worth naming them.

Late in a long day. Work with an agent from morning to evening and your judgment degrades. You cannot feel it happen. The approval dialog asks you from scratch every time, so the tenth one looks exactly like the first. The limiter is the opposite: a number that does not go down. Name your production paths and the day's total is right there.

{ "production": ["/var/www/", "/srv/client-sites/"] }

When it runs unattended. Set AIRFRAME_AUTONOMY to the reason — a loop, a timer — and the machine really does halt. With nobody in the seat, "advise" means nothing. I don't know of another brake that works for unattended runs.

When you are writing prose with an agent. Articles, proposals, design notes. Overwriting a file git does not track, in cruise, keeps the previous version. And the completion gate goes quiet, so a draft never gets reported as "unmet."

Just as clearly, where it does not help:

Short one-off tasks — it's over before you'd look at the panel
Local, reversible experiments — nothing to count
Overwrites via sed -i or a formatter — those never pass through a hook, so the draft keeper cannot see them
Judgment calls — "is this design good" has no probe, so the gate has nothing to check

Long sessions, a lot delegated, hard to undo. The more of those three you have, the more this earns its place. None of them? Don't install it.

Where I think this goes

Soon enough:

agents will run while you sleep
you will fly several at once, not one
a single run will last hours, not minutes

When that happens, I don't think the bottleneck is model quality. It is whether a human can stay in the loop at all.

You wake up to eight agents' worth of work. Can you reconstruct what happened? Which of them touched production? Did any of them quietly revert the fix you made by hand yesterday?

That is why status has a wingmen row. How many were launched is a number a pilot should be able to see afterwards, and there is no other way to know it.

Whether "AI does the work" means you are in command or you are being carried is decided, I think, by the cockpit rather than by the model.

What none of this buys

It does not make the agent correct. None of these parts read your code. They watch what happens around the work.

The limiter is a gauge, not a guard. Its number is summed out of a ledger the agent can write to. It is built for the agent that has lost count, not the one trying to get past you — anything that can run commands on your behalf is already inside the boundary.

The draft keeper matches names, not contents. .env is never copied; a draft with a key pasted into the third paragraph is copied like any other draft. A part that opens every draft to judge it is a part that opens every draft. I liked that less.

npx @hyuga/airframe install

Node 18+. No dependencies, no daemon, no network, no LLM. MIT.

github.com/hyuga611/airframe

If you are sitting in the seat, look at the instruments.

cd, type go, and your workspace builds itself — tsumugi 0.3

hyuga — Thu, 27 Aug 2026 13:24:11 +0000

A while back I wrote about building a terminal because I could never remember the keybindings. That terminal is tsumugi, and I just shipped 0.3. This post is about the three things that went in — a workspace command, screenshot paste, and self-update — and why each one is shaped the way it is.

github.com/hyuga611/tsumugi — Rust, MIT OR Apache-2.0.

The premise

Short version, since the first post covered it. tsumugi is built on one claim:

The grid is a document. The process is appending to its end.

Your scrollback is not a log, it is what you just did, so it should be readable, selectable and reusable. Vim motions over scrollback, ac to select a command with its output, :e to turn the pane into an editor — they all fall out of that one line.

And one rule on top: every command has a mouse path, enforced in CI. A test walks the command registry and fails if any command lacks one.

Those two decided the shape of almost everything below.

1. `go` builds a workspace in one message

cd ~/dev/tsumugi
go

The pane you are in becomes the middle one, a directory tree opens on the left and an AI agent on the right, and the tab is renamed after the directory. 20 / 50 / 30. One tab per repository, and the tab names tell you which is which.

One message, not four

Sending "split", "open the tree", "start the agent", "rename the tab" separately costs you twice: intermediate layouts get broadcast and the screen jumps, and you end up needing "the id of the pane that was just created" in the next message — an async hole. So it is a single ClientMsg::Workspace. The server assembles it and broadcasts once.

The tree has no motions of its own

The tree implements Buffer. That is the whole trick: j k gg G /, visual selection and the mouse already work, because they are the same ones you use on the terminal. I did not write a single tree-specific motion.

Only a handful of keys are added inside the tree: Enter opens, l / h fold a branch, a / A create, r renames, R reloads. They are not in the default keymap — a and r meaning something else in a terminal would be wrong. Double-click opens, drag and drop moves. :q closes the tree and the pane goes back to being a shell, with the shell underneath still running.

There is no delete. An operation you cannot undo does not belong on a list your finger slides across. Delete from the shell in the middle pane.

The listing is read server-side

With [domains], the files are on the far machine, so the tree is read there. Only branches you have opened are walked, so a directory with node_modules or target in it stays fast.

It coexists with the Go toolchain

go is a function the shell integration installs; it calls tsg --workspace. Anything with arguments is forwarded to the real go, so go build still works, and a bare go outside tsumugi still runs Go. TSUMUGI_NO_GO=1 gives the name back. From the keyboard it is Space then w; from the palette, :go.

2. Paste a screenshot into what you are telling the agent

Ctrl+Shift+V writes the clipboard image out as a PNG and types its path into the prompt — it does not press Enter, so you can keep writing before you send. Dropping an image file on the window lands in the same place.

No new command was added. Paste is one verb that does the closest thing to what it is holding: text if text, a path if a picture. Adding verbs adds things to remember.

The server writes the file, into a per-session temp area — so when you are connected to a [domains] host, the file appears where the agent reading it lives. The last 32 are kept.

3. `tsg update` starts no external process

tsg update fetches the latest release and swaps it in. It downloads nothing when you are already on the latest.

Why the download-and-run shape had to go

The first version ran powershell -Command "irm <URL> | iex" inside itself, to keep exactly one way of installing. But that shape is indistinguishable from malware that downloads and runs code. Windows Defender on a corporate machine blocked it — the process was denied (os error 5) and the installed binary was removed along with it.

The machines that need it most are the strictest, so that is exactly where it broke. It now pulls the release over HTTPS itself (ureq / rustls) and replaces the file. No child process at all. The first-time install.ps1 / install.sh are unchanged — there is no tsg yet, so that one has to start from outside.

After the swap, no reopening and no `--kill`

tsg update reopens the window on the new build. The multiplexer is a separate process, so replacing only the window reconnects you with your shells and agents still alive. I watched the window process change while the multiplexer's PID stayed the same.

It checks before it commits: it asks the new binary for its protocol version (--protocol) and reopens only if it matches the running server. Only when they differ does it become a conversation about stopping sessions — and it does not stop them for you, because stopping one ends the shells and agents inside it.

Related decision: the protocol version is not bumped for additive changes. Bumping it means a running server stops accepting new windows, which means killing the shells inside. It moves only when the meaning of an existing message changes.

Install

No admin rights.

Windows (PowerShell)

irm https://raw.githubusercontent.com/hyuga611/tsumugi/main/install.ps1 | iex

That one line leaves you with a working setup: the binary in %USERPROFILE%\bin, Start Menu / PATH / folder context menu, the shell integration, and the Claude Code and Codex hooks when they are present.

It used to be three steps, and skipping the last two left you with a terminal where the gutter, [[, ac and go all did nothing — installed but not working. It was one errand for the person typing, so it is one command now. Every change is printed, and tsg --uninstall takes all of it back out.

The CRT is linked statically, so it runs on machines without the VC++ redistributable.

macOS / Linux (experimental)

curl -fsSL https://raw.githubusercontent.com/hyuga611/tsumugi/main/install.sh | sh

Untested. CI builds and tests on three OSes, but the window (winit / wgpu), the IME and --install have parts written against Windows APIs, and nobody has run them yet. Shipping the binaries is to make trying it easy, not a promise that it works. Reports very welcome.

Check it took — a shell reads its config at startup, so the window you typed in does not have it yet. Open a new one:

tsg -V             # prints the version
Get-Command go     # Function means the shell integration took

You need no configuration to use an agent. The waiting mark on the tab and the notifications come from the hooks, so typing claude or codex in a pane just works. Configuration only decides what go starts on the right.

[shell]
program = "pwsh"

[workspace]
agent = "claude"

Updating

tsg update

Windows SmartScreen will warn on first launch — the binary is unsigned. More info, then Run anyway.

Also fixed in 0.3.x

Twelve patch releases, almost all of them from running it on a different machine — a locked-down corporate laptop. A sample:

Windows PowerShell 5.1 and PowerShell 7 read different profiles; the installer wrote to whichever it found first. Now it writes to all of them.
It guessed where $PROFILE was. If you have never created one, neither directory exists and it gave up. Now it asks PowerShell.
The default shell was cmd.exe (from COMSPEC) — and cmd.exe has no way to emit OSC 133, which everything in tsumugi is built on. Everyone who installed it started from a state where nothing worked.
Agent state was landing on the wrong pane: the hook ran tsg --agent-state without naming a pane, so it attached to whichever was selected. Two agents, one badge that never clears. It reads TSUMUGI_PANE now.
Three bugs on top of Claude Code, all one root cause: ownership was decided by alt screen alone, and some tools paint the whole screen without it.

And 0.3.13 is the four I hit while taking the screenshot for this post. Worst first:

[workspace] agent = "claude" built no workspace at all — the headline feature of this post, with the configuration this post recommends. npm puts both claude (an extensionless shell script) and claude.cmd on Windows; CreateProcessW does not consult PATHEXT, so a bare name hits the former and dies with os error 193. A bare name is now resolved against the extensions CreateProcessW can actually start.
That reason never reached the person who typed it. tsg --workspace sent the request, slept 200ms and detached without reading the reply, so the server's error went nowhere. go calls exactly that, so all you saw was nothing happening.
A failed go leaked a pane. The tree spawns before the agent; when the agent failed, the tree stayed alive and attached to no tab. One more every time you retried.
The shell integration could be dead in every pane (PowerShell only). The guard against double-sourcing was an environment variable — and environment variables are inherited by children. Start the mux from a shell that already sourced it (that is: type tsg in your terminal) and every pane's shell decides it has already run. Gutter, [[ ]], ac / io and go all go dark. Launch from the Start Menu and you never see it; type tsg in a terminal and you always do.

Both halves of that list say the same thing. "Installed but not working" never reproduces on the machine of the person who installed it. Every test passed here.

Full list in the CHANGELOG.

https://github.com/hyuga611/tsumugi

I built a terminal because I could never remember the keybindings

hyuga — Tue, 25 Aug 2026 13:07:40 +0000

I could never remember the keybindings.

I know WezTerm and NeoVim are good. I installed both more than once, and backed
out both more than once, always at the same place. It isn't that I refused to
learn them — I stalled before I got far enough to learn anything.

So I built the terminal I wanted instead: one where you can do everything
without knowing a single key, and where knowing the keys only makes you faster.

https://github.com/hyuga611/tsumugi

Rust, MIT / Apache-2.0. Developed on Windows.

One idea: the screen is a document

In a normal terminal, past output is over. You can scroll back and look at it,
but you can't really use it — you end up dragging with the mouse and hoping
you grabbed the right lines.

But scrollback isn't a log. It's what you just did, and it's meant to be read
back and reused. So tsumugi treats it as a document:

walk past output with vim motions (j k w [[ ]])
select a command and its output as one thing (ac / io text objects)
open a file in the same pane (:e)

The screen and a file are both "documents", so the same keys work on both.
There is no separate editor to launch.

Selecting "a command and its output" honestly

Guessing at this always breaks. Decide that "lines starting with $ are
prompts" and the first $ inside some output ruins it.

So tsumugi uses OSC 133 — shell integration. The shell tells the terminal
where the prompt starts, where the command starts, where the output starts, and
what the exit code was.

tsg --install-shell-integration

After that you get ✓ / ✗ marks in the left gutter, [[ ]] to jump between
prompts, and ]e to walk only the failed commands. Nothing is matched with a
regex against your output, so no mark is ever a lie.

Every command has a mouse path — enforced by a test

Since the whole point is "you don't have to know the keys", this isn't left to
good intentions:

#[test]
fn every_command_is_mouse_reachable() { ... }

Add a command without a mouse path and CI fails. There are 78 of them right now.
Right-click gives you "what you can do here". Space t labels every path and URL
on screen with one or two letters, and that letter opens it.

Running several AI agents turned me into a search function

This part came out of using it, not planning it.

I had three Claude Code sessions going on different jobs, and noticed that most
of my actual work had become hunting for which one was waiting on me. Look
at one pane, look at the next, find the stalled one, answer it, start over.

So I made the agents say so themselves:

tsg --install-agent-hooks

That wires one line into Claude Code's and Codex's hooks. Tabs get a ● mark, the
status bar counts how many are waiting, and Space a jumps to the next one.

If the window isn't focused, the taskbar flashes. Anything running inside can
also call for you with OSC 9 when a long job finishes.

Losing the machine doesn't lose the session

Closing the window was already fine — the multiplexer is a separate process, so
shells and agents keep running. Losing power was not.

Now the layout and each pane's directory come back on the next launch. But
the screen contents are never written to disk. SECURITY.md promises that
scrollback never reaches the filesystem, and I didn't want to quietly break it
for a convenience feature.

So what comes back is the shape, the directories, and the argv. And the
conversation? The agent owns its own transcript, so tsumugi only needs to
know how to say "continue". A restored pane gets claude --continue typed at
the prompt but not run — pressing it is your call.

Reviewing what the agent wrote, where you are

Space g opens git diff in colour. Space G stages the hunk under the
cursor, Space R reverts it (twice — it destroys work).

A diff you only look at means making the same decision again somewhere else
later. And if an agent rewrites a file you have open, tsumugi reloads it (unless
you have unsaved edits, in which case it tells you instead). Without that you
read stale text, edit it, and delete the agent's work the moment you save.

Also in the box

language servers: squiggles for errors, gd to definition, Ctrl+Space to complete
:grep searches the whole project and opens straight from the results
:123, s/old/new/, g/ for a regular expression
SSH domains — the far side keeps running when the link drops
Markdown rendered in place, images (Kitty graphics / Sixel)
narrowing the window re-wraps scrollback instead of truncating it

What building it actually taught me

The lesson that stuck: you cannot test your own work with your own habits.

Seven keys listed in the command palette did nothing when pressed — Space g
for the diff, Space a to jump to a waiting agent, and five more. I had been
verifying everything through the CLI (tsg --run <id>) and had never once gone
through the key path. A test that mechanically cross-checks the list against the
implementation found all seven in one run.

Opening a tab-indented file deleted every tab — the worst kind, because
saving then destroys someone else's indentation. That fell out of writing a test
for auto-indent, not from using the editor.

Wiring up LSP, I sent initialized without waiting for the initialize
response (a spec violation) and rust-analyzer simply went quiet: running fine,
zero diagnostics. Then a second one — language servers return file:///c:/...
with a lowercase drive letter, and I was comparing paths as strings, so every
diagnostic was arriving and being thrown away.

All three looked like they were working.

Being straight about the state

Developed on Windows. macOS and Linux build and pass CI, but nobody has run
the window layer (decorations, IME, the installer). The README says untested
rather than supported, and I'd rather keep it that way until someone has.

It isn't finished so much as mine — I use it daily and can fix it myself. I
started out unable to remember keybindings and now drive about half of it from
the keyboard. Not because I finally memorised them, but because I could keep
using it while I didn't.

Your AGENTS.md is lying to your agent. So was my linter.

hyuga — Sun, 23 Aug 2026 01:39:00 +0000

AGENTS.md and CLAUDE.md rot silently. You write "setup is npm run build", "the entry point is src/index.ts", someone renames a script, and the instruction quietly becomes a lie. The agent believes it and breaks things.

So I built reflint: a linter that ignores wording and style entirely and checks one thing — do the references still resolve? Back-quoted paths against the disk, markdown link targets against the repo, npm run <script> against package.json. Zero dependencies, language-agnostic, exits 1 in CI.

That is the launch paragraph. The rest of this post is what happened after I published it.

I ran reflint against 118 public repositories that have an AGENTS.md or CLAUDE.md, in both the before and after versions. What it found was mostly my own breakage.

1. A flag I had advertised in a blog post had never done anything

reflint has --code-blocks. The help text says it makes the tool "also check paths inside fenced code blocks", which implies the default skips them. Months earlier I had written a post saying, in so many words: turn this on and it checks bare paths inside code blocks.

Across 118 repositories, the number where the flag changed the result was zero.

The cause was wiring. Only the fourth of four scanners consulted the in-a-fence state. The npm run scanner, the back-quoted reference scanner and the markdown link scanner never looked at it — they read fenced content by default, so turning the flag on could not add anything. The flag was dead.

I moved all three inside the fence check. On the same 118 repositories, 10 now change their result depending on the flag.

2. `pnpm -r build` was reported as "no script named -r"

AGENTS.md:3   `pnpm -r` — no script "-r" in package.json
AGENTS.md:4   `pnpm --filter` — no script "--filter" in package.json
AGENTS.md:5   `pnpm -w` — no script "-w" in package.json

The character class in the script-name regex included -, so the flag right after pnpm was read as the script name. 16 of the 39 script-kind findings were this. In any repo using pnpm workspaces, correct documentation turned red.

Those forms run per-workspace-package, so the root package.json is not the right thing to check them against. reflint now skips the whole invocation.

3. One misplaced fence and it checked nothing, then said everything was fine

Fence state was tracked with the obvious inFence = !inFence toggle. That breaks on any document that demonstrates a fence by wrapping it in a longer fence: the marker count goes odd and everything below sticks to "inside a fence" forever.

One of the 118 — a 3,461-line AGENTS.md — did exactly this. Headings and prose were being treated as fenced content.

Combined with fix #1, this is the dangerous one. Fenced content is skipped by default now, so from the misplaced marker down, reflint inspects nothing, exits 0, and prints reflint: all references resolve.

For a linter that is the worst possible failure, and I had already been here once. In 0.9.2, the CLI installed via npm i -g or npx exited 0 without running at all, because the entry-point check compared process.argv[1] against import.meta.url and those two never match through a symlink. "Found no problems" and "never ran" are indistinguishable — to the user and to CI.

Closing is now CommonMark's rule: same character, at least as long as the opener, no info string.

If you go quiet, say why you went quiet

Every one of those fixes creates a region that is not checked by default. If the count drops, the user cannot tell whether things got fixed or whether the tool stopped looking. That is the same failure again, one level up.

So reflint now always reports what it held back:

reflint: 2 broken references (3 inside code blocks, not checked — run with --code-blocks)
reflint: all references resolve (1 inside code blocks, not checked — run with --code-blocks)

The JSON output gained a skipped field. HTML comments are not counted — they are disabled text, not content that --code-blocks would bring back.

The numbers

Across the 118 repositories: 208 findings became 185. All 23 that disappeared were noise, and zero new findings appeared.

For honesty's sake, two other fixes in the same release — HTML comments and indented code blocks — had zero occurrences in this corpus. They are reproducible and I closed them, but they moved no numbers. The three above did.

A different shape, same lesson: only the people who wrote it correctly got warned

The release after the audit fixed something else. Agent instruction files are full of prohibitions:

Never run npm run release; releases are performed by a human.
  → `npm run release` — no script "release" in package.json

Never read or execute `scripts/deprecated.sh`; it was removed after the migration.
  → reference `scripts/deprecated.sh` does not exist

Both of those are correctly written documents. If you explicitly tell the agent not to use something that was removed, then of course it does not exist — that is the entire point of the sentence. And yet the person who documented it was the only one getting flagged.

Prohibition lines are no longer scanned as references, and that test runs outside fences only, so a # never edit this comment inside a block does not silently disqualify the real references under it.

I have now hit this exact shape in three of my linters. Text rules see mention, not execution. The more dangerous a thing is, the more likely a careful author is to name it in order to declare, forbid, or scope it — so false positives concentrate on the people writing the best documentation.

The part I actually want to leave here

Through all of it, the test suite was green before any of these fixes.

There were tests for --code-blocks. They asserted that enabling it made the fourth scanner check fenced paths. They did not assert that the other three were skipping them. There were tests for fence tracking. None of them fed in a document with an odd marker count.

The tests I write only cover the ways I imagined it breaking. 118 repositories of other people's AGENTS.md cover the ways I did not. Until I ran it against them, I believed my tool worked.

Try it

npx @hyuga/reflint                 # auto-detects AGENTS.md / llms.txt / CLAUDE.md
npx @hyuga/reflint --code-blocks   # also check inside fenced blocks (this works now)
npx @hyuga/reflint --since main    # only files changed against a git ref
npx @hyuga/reflint --format json

In CI, which is the point:

name: reflint
on: [push, pull_request]
jobs:
  reflint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hyuga611/reflint@v1

Findings show up as inline PR annotations. One correction I had to make to my own README: GitHub does not block a merge on a failing check until you mark that check required. Until then it is a red X that anyone can click past.

Repo: https://github.com/hyuga611/reflint — the changelog is written up per release, including the ones where I was wrong.

I asked one question. All four of my linters failed it.

hyuga — Fri, 21 Aug 2026 01:23:33 +0000

Three separate times, in one of my linters, the same bug came back wearing a different hat: a text rule was looking at mentions and treating them as execution.

That matters more than it sounds, because of who writes mentions. If you never think about scripts/deprecated.sh, you never type it. The person who writes Never run scripts/deprecated.sh — it was removed in the migration is the person being careful. My rules were reading that line as a step and reporting it. The tool punished exactly the people who did it right.

After the third occurrence I stopped fixing instances and asked whether the shape was in my other tools. It was. In all of them.

The question

I did not ask for a code review. Reviews come back with style notes. I asked a single adversarial question and gave it the source:

Construct an input that punishes the person who wrote it carefully.

Four tools, four hits.

Tool	Version	What the careful writer got
skills-lint	0.9.0	A prohibition silenced a real broken reference
tracklint	0.7.0	Comments counted as code — in both directions
tokenlint	0.5.0	React inline styles missed entirely, inflating the headline number
reflint	0.12.0	Commands and paths inside prohibition sentences counted as steps

The worst one was not a false positive

skills-lint checks whether a SKILL.md references files that do not exist. It has a sensible exemption: if the document says the skill produces a file, that file is not expected to be there yet. PRODUCES matched on stems like generat.

So this happened:

Never generate `scripts/missing.py`.
When asked to deploy, execute `scripts/missing.py`.

Line 1 registered scripts/missing.py as an artifact this skill produces. Once a name is registered, references to it are exempt anywhere in the document — including line 2, which is a genuinely broken reference to a file that does not exist.

The tool printed ✓ all clean.

This is the part I keep thinking about. A false positive is loud: someone opens an issue, you fix it. A warning that never fires is invisible. If a user had hit this, they would have concluded their skill was fine. There is no complaint to receive. The only reason I found it is that I went looking for the input that would cause it.

Broken in both directions at once

tracklint checks conversion tracking on web forms, and it was matching regexes against raw text. Comments are raw text.

The direction that punishes the honest:

<!-- This site intentionally does not call gtag() or load Google Analytics. -->

That one line convinced the tool the project has analytics, which switched on every wiring rule. An identical form went from clean to 1 error because someone documented that they don't track.

The direction that hides the real problem:

// gtag must never be called here; conversion tracking is not implemented yet.

Counted as a real call, so an AJAX form with genuinely no success-time tracking stopped reporting ajax-no-conversion.

The fix was a stripComments() that blanks comments while preserving length and newlines, so line numbers don't shift. I deliberately made it conservative — https:// and src="//cdn..." are not comments, and over-stripping would make the tool miss the analytics install and go quiet. Between a fix that over-reports and a fix that under-reports, I took over-reporting, for the same reason as above: noise is visible, silence isn't.

When a miss flatters your own headline number

tokenlint reports a design-token coverage percentage: tokenized / (tokenized + hardcoded).

Its markup scanner only understood quoted HTML attributes, style="...". It had never once matched:

<div style={{ color: "#ff0000" }}>

That is the ordinary way to hardcode a color in React. Files consisting entirely of hardcoded colors reported zero hardcoded values — and because the miss lands in the denominator, every miss pushes the coverage number up. My tool's own advertised metric was wrong in the flattering direction.

Worth separating two things: this was a miss, not a judgment call. A // line comment containing a Tailwind class is still not scanned, and that one stays — // cannot be told apart from https:// without hiding whole URL lines, which trades a false positive for a miss. That trade I'm making on purpose. The React one I just never checked.

One caveat about how this was found

The model I asked could not run node in its sandbox, so it validated by calling internal functions and reasoning about them. That is not evidence. Every claim it made, I re-ran through the real CLI myself before writing a fix. After publishing, I installed each package fresh from npm and re-ran the failing path against the released build.

An adversarial reviewer is very good at generating the input you would not have thought of. It is not a substitute for executing that input.

Two questions worth asking your own tool

"Construct an input that punishes the person who wrote it carefully." Better than "review this." It forces a concrete artifact you can run instead of a list of concerns.
"What does this change stop reporting?" Every one of these bugs was easy to see once framed as a miss, and invisible while framed as a fix. If your tool has a coverage percentage or a clean/dirty verdict, ask which direction a bug in it would move that number.

Sources, if you want the actual diffs. The changelogs are written in Japanese, but each entry contains the literal failing input, and that part reads fine in any language:

All four were green on their own test suites the entire time. That's the other thing worth saying out loud: a test suite you wrote is a picture of what you already thought of.

I audited 2,465 published agent skills. 192 of them cannot be selected the way the spec says

hyuga — Tue, 18 Aug 2026 01:03:53 +0000

192 skills have no frontmatter at all

An agent picks a skill by reading its description and deciding whether the request matches. No description, no selection — whatever else is in the file.

I drew 2,465 skills at random from a public registry and ran a reference-integrity and portability check over them. 192 of them (7.8%) ship without a single line of YAML frontmatter. No name, no description. They are published, downloadable, and cannot be chosen the way the specification says skills get chosen.

I should be careful about what that does and doesn't say. It is a statement about the files, not about any particular runtime — what a given agent does when the string is missing is something I did not measure, and inferring runtime behaviour from files is a mistake I made elsewhere in this same audit (below). What the count says is that in 192 published skills, the input that selection is supposed to run on is not there.

The rest of the breakdown

1,424 of 2,465 skills (57.8%) have at least one error-level finding. Percentages are the share of sampled skills with at least one finding of that kind, not the share of findings:

	skills	of sample
frontmatter `name` differs from the registry slug	720	29.2%
a back-quoted path or link that does not resolve in the package	445	18.1%
ships with no YAML frontmatter at all	192	7.8%
an absolute path that resolves only on the author's machine	94	3.8%
malformed `allowed-tools`	58	2.4%
a provider API key read straight from the environment	52	2.1%
an external CLI the skill never declares	31	1.3%

One caveat I want to put next to the top row rather than in a footnote: that 29.2% is not 29.2% broken. The registry deliberately keeps the routable slug separate from the stored display name, and says so in its own docs, so a real share of those are working as intended. What remains worth knowing is that on nearly a third of published skills, the name the agent registers under is not the name the registry shows you.

The sample is 2,465 drawn uniformly from 69,265 enumerated, seed 20260804. Those 69,265 are the most-recently-updated slice of the registry, not all of it, so every rate above is a rate inside that frame. The harness is published and the seed is fixed; re-run it and you should get the same draw.

The number that mattered was my own

Separately from the registry sample, I audited all 46 skills bundled inside openclaw/openclaw — a repository with 385,000 stars — at a pinned commit.

The first run produced 219 findings. After six precision fixes to my own tools, the same 46 skills produced 59. Every genuine defect appears in both runs. So the 160 that vanished were all mine.

Publishing before that work would have made the census wrong by nearly a factor of four.

What I had wrong

The big one, 139 of 197 path findings: I resolved references against the skill's own folder instead of the repository it lives in. A skill sitting inside a repo and pointing at packages/ or docs/ got reported as pointing at nothing.

The rest, listed plainly because the list is the useful part:

openai/gpt-5.4 read as a file path, because .4 looks like an extension.
An artifact excused on the line that creates it, but reported as a missing reference on the line that reads it back — 16 of 80.
/home/YOUR_USER/… treated as an author's real home directory.
Indented frontmatter parsing to nothing, so a skill with both required keys was reported as having neither.
Sample output inside a fenced code block checked as if it were a live reference — 10.8% of path findings.

Number 5 I found by dogfooding: a skill written to document these linters could not pass them, because it quoted their own example output.

Number 1 has a detail I keep thinking about. The guard that was supposed to stop model identifiers being read as paths already existed, with a comment explaining exactly why it existed. The example in that comment ended in a letter. Anything ending in a version number sailed straight through. The comment was right and the code under it was not, and the comment is what stopped me looking.

A linter dies of false positives, so I treated the false positives as the bugs.

The finding that looks textbook and isn't

One bundled skill hardcodes /Users/steipete/openclaw — a maintainer's home directory, in a file shipped to everybody. That is the textbook portability finding, and it is not a bug. That skill is an explicit single-machine runbook for one canonical checkout, and the path is correct there. It was deliberately left out of the upstream report.

This is the part a percentage cannot carry. A linter produces candidates. Whether a candidate is a defect depends on what the document is for, and that judgement did not survive automation.

One report was closed as `not planned`, and closing it was right

I filed two issues upstream. One was rejected, and I agree with the rejection.

I had argued that a skill could not read the files bundled alongside it. It can: the prompt that hands the skill list to the model spells out the relative-path resolution rule. I had read one file, found no rule in it, and stopped.

That is worth more than the finding would have been. Choosing a resolution rule is choosing a definition of "broken," and that definition has to match runtime behaviour, not the filesystem. Mine was tuned to the permissive side afterwards, which is why every number above is a floor rather than a ceiling.

The other report held up: two bundled skills pointed at files deleted in an earlier refactor, so the reads return nothing and the work proceeds without the rules it was supposed to apply — silently, as usual. The fix is merged and the issue is closed as completed.

What I did not publish, and why

The corpus. 2,465 third-party skill files are other people's work. Re-fetch them with the harness instead.
The 94 author paths. Reported as a count, never as a list, with home-directory segments redacted in the example output. Nobody was contacted. Mass-filing issues against individuals, on the strength of a rule with a known false-positive history, would be the wrong move — see the entire middle of this post.

Reproducing

The harness, the seed, and the results are in agent-skill-audit. Data is CC0, harness is MIT.

The rates above are tied to the tool versions used on the run: tenken 0.2.0, skills-lint 0.7.1, carrylint 0.2.2, reflint 0.8.3. All of them have moved since — precision fixes in some places, new rules in others — so a re-run today would not reproduce the table exactly. That is the reason to publish the harness and not only the numbers.

Corrections to the numbers themselves are welcome, and I would rather have them in public than not.

My linter kept warning the people who did it right. Three times, in the same direction

hyuga — Mon, 17 Aug 2026 09:53:09 +0000

The warning landed on the only people who had done it properly

I maintain a linter that reads agent config files — SKILL.md, AGENTS.md, CLAUDE.md — and fails CI when they bake in something that only works on the author's machine. One of its rules says: if you call an external CLI, declare it, or the next person won't have it.

Declaring it means naming it in frontmatter:

requires: codex

Except that anyone with more than one dependency writes the list form, because that's what YAML is for:

requires:
  - codex
  - gemini

My implementation only read the first shape. So the block list — the normal way, the way you write it the moment you have two of anything — was invisible to the linter, and it warned you for an undeclared CLI that you had, in fact, declared.

Read that back slowly. Authors who ignored the dependency question entirely were never flagged, because they never wrote a requires: key at all. Authors who sat down and wrote the contract properly got a warning telling them they hadn't. The rule was inverted with respect to the thing it was trying to encourage.

I shipped that. It went out in a patch release, and I only found it because a commenter used the phrase "dependency contract" and I went to re-read my own implementation of it.

Then it happened again. Twice, in one release

Two comments on a post of mine turned into new rules. One of them, unverified-write, reports a file that changes external state — git push, npm publish, an INSERT — and never reads that state back anywhere.

Before publishing, I ran it over 586 real skill files pulled from a public registry, found two false-positive shapes in the data, fixed both, and re-measured. Fire rate 0.7%, and every hit I could check by hand was genuine. I felt good about it.

Then I handed the diff to a different model for a pre-publish read, and it produced this input in about a minute:

Never run `git push --force` from this skill.

That is a git push in a code span, in a file with no read-back anywhere. My rule flagged it as an unverified write.

AGENTS.md and CLAUDE.md are full of that sentence. Writing down "don't push without asking" is the single most common act of care in that genre of file. I had built a rule that warns you for pushing, specifically because you wrote down that you must not push.

I fixed it, published, and then noticed the same shape one layer further out. Prohibitions were now excluded, but permissions were not:

- `git push` は明示の指示があるときだけ。
- Only run `git push` when the user asks.
- `npm publish` requires approval from a maintainer.

Nobody who writes those sentences has an unverified write. They have a policy. Three releases, three variations, all pointing the same way: the warning finds the author who wrote the rule down and misses the one who never mentioned it.

Why it points that way

A text-matching rule cannot see actions. It sees mentions. And mentions of a dangerous operation are not distributed randomly across authors — they concentrate in the files of people who thought about it.

The careless author's AGENTS.md doesn't say git push anywhere. There is nothing for the rule to catch. The careful author's file says it three times: once to declare when it's allowed, once to forbid the force variant, once in the actual deploy step. Two of those three are not the thing you're detecting, and both of them are evidence of care.

So the base rate is against you. Among all the files containing the string you match on, the share written by conscientious authors is much higher than in the population — and every false positive you have is drawn from that pool. The people most likely to read your warning carefully, and most likely to uninstall you over it, are the people you are most likely to be wrong about.

Static analysis has a name for the underlying distinction — use versus mention — and my older rule already knew it. Its CLI check ignores a bare `codex` in prose and only fires on codex exec build, an actual invocation with an argument. I wrote that exclusion two releases earlier, in response to the same class of complaint, and then built a new rule without it.

The audit that found nothing here

The part I want to be honest about: running against 586 real files did not catch any of this.

It couldn't. Published, downloadable skills are written to be used; they say "run this" far more often than "never run this." The prohibition shape lives in team-internal AGENTS.md files that nobody uploads to a registry. My corpus was real data, and it was the wrong real data — biased, in exactly the direction that hid the failure.

That's worth separating out, because "test against real data" is advice I've given in writing and still believe:

A real-data audit tells you what your rule does to the corpus you can reach.
An adversarial read tells you what your rule does to the input someone constructs on purpose.

The second one found in one pass what 586 files had not. It cost one prompt. If your detector will run on files you can't see — and a linter always does — you need both, and you should assume the corpus is the more comfortable of the two.

What actually fixed it

Not more keywords. The fix was making the use/mention distinction structural, then checking the price:

A write signal only counts in a code context — inside a fence, or inside a backtick span. Prose saying "after that we push to git" is not a step.
A line carrying a prohibition (never, do not, 禁止) isn't a step.
A line carrying a condition or a permission (only … when, requires approval, 〜のときだけ) isn't a step outside a fence. Inside one, the lines are commands, and excluding them there would drop a real finding over an incidental comment like git push origin main # main only.

Then the part that isn't optional: re-run the corpus and prove the exclusions didn't eat the signal. Four true positives before, the same four after, fire rate unchanged at 0.7%. An exclusion you didn't measure is just a rule you deleted with extra steps.

One more thing, since your linter runs on other people's files

The same review turned up something unrelated but worse. My check for a remote copy looked like this:

\b(?:scp|rsync)\b[^\n]*\s\S+@\S+:

Greedy fill, then a search for something@something:. On a long line containing many @ and no colon, it backtracks quadratically: 77ms at 20k characters, 312ms at 40k, and it keeps going up from there. A minified blob or a base64 payload on one line of somebody's repo is enough.

I had been thinking of the input as "config files people wrote." It isn't. It's arbitrary text from strangers, and a linter that hangs is a linter that stops a stranger's CI. The fix was to stop the filler crossing an @ so there's nothing to backtrack over; an 80,000-character line is now a test case.

The rule of thumb I'm keeping

When a detector matches on text, ask who says that text most often. If the answer is "the people being careful about it," your false positives are not evenly distributed — they're aimed. And the only reliable way to see it is to hand the rule to something that is actively trying to embarrass you, because your own corpus is made of the cases you already knew about.

The tools: carrylint is the linter above; the read-back half it deliberately doesn't attempt is genchi.

That amazing skill you shared only runs on your machine. I built a linter for it — then found I was 85% wrong

hyuga — Mon, 17 Aug 2026 02:09:29 +0000

I mass-produced skills by saying "turn this into a skill." Then I shared them

With Claude Code or Codex, any procedure that worked can become a reusable SKILL.md just by asking "turn what we just did into a skill." I did this for image generation, deploy steps, everything. It was great.

Where it fell apart was handing those skills to teammates. They ran on my machine and silently died on theirs. Every time I dug in, it was the same shape:

Output path hardcoded to C:\Users\atlan\Downloads\out.png (my home directory)
The body calls codex exec ... with no instruction anywhere for installing that CLI
Assumes OPENAI_API_KEY is already set, with no guidance when it isn't
A model id like gpt-image-2 written inline

All of them are the author's environment baked in. None of them raise an error — they just fail quietly in the next person's hands, which is the worst kind.

So I built carrylint, a linter that fails CI on exactly this.

The standard made the format portable. Whether the contents run is a separate question

In December 2025, Anthropic made Agent Skills an open standard. One SKILL.md now runs across Claude Code, Codex, Gemini CLI, Cursor, Copilot, and 20-odd others. Format portability is solved.

But a standard only guarantees the shape of the container. If the inside holds an absolute path or an undeclared CLI, the container is valid and the contents still don't run for anyone else. I couldn't find a tool looking at that.

reflint (also mine) asks: do the references exist?
skills-lint (also mine) asks: do skills collide, is the frontmatter valid?
carrylint asks: do the references resolve in a different environment, under a different model?

The failure class is inverted. The others check "is this correct as specified." carrylint checks "will the next person install this and have it actually run."

What it catches

Run it against a deliberately non-portable sample:

✗ examples/bad/leaky-image-gen/SKILL.md — 3 errors / 3 warnings
  ✗ :16  [abs-path] machine-specific absolute path `C:\Users\alice\Downloads\out.png`
         — will not resolve on anyone else's machine (use a relative path or {baseDir})
  • :16  [undeclared-cli] calls `codex` but never declares or installs it
  ✗ :22  [abs-path] machine-specific absolute path `C:\Users\alice\Downloads\out.png`
  • :25  [provider-env] assumes `OPENAI_API_KEY` is set
  ✗ :27  [placeholder] unresolved placeholder `<FILL_ME>` left in a shipped file
  • :29  [todo] TODO/FIXME marker left in a shipped file

carrylint: 3 errors / 3 warnings
exit code: 1

Rules are split by severity. False positives are what get a linter uninstalled, so only things the next person will hit, with no room for interpretation, are error (which fails the PR). I moved that line a lot after shipping — see below.

Severity	Rules
error	Author-environment absolute paths (`C:\…`, `/Users/<realname>/`); unfinished markers (`<FILL_ME>`, `REPLACE_ME`). `$HOME`, `~`, `YOUR_API_KEY`, `/path/to/` are excluded — they're portable or a documentation convention
warn	Undeclared external CLIs (host commands like `claude mcp add` excluded); raw provider-specific env references; leftover `TODO:`
opt-in	Hardcoded model ids (`claude-`, `gpt-`) — deliberate pinning is legitimate, so off by default

No LLM and no API key at runtime — pure static analysis. Since the criterion is "are you locked into one environment or model", it works identically whether the skill was written by Claude or Codex. The tool embodies the thing it checks for.

Then I ran it against 230 real skills and learned I was 85% wrong

This is the part I most want to be honest about. I had shipped it, but something nagged before I promoted it: my own examples and my own tests passing proves nothing. So I collected 230 real SKILL.md / AGENTS.md files from public GitHub repos and ran carrylint against them untouched.

The good half. Skills that only run for their author are genuinely out there. One PPT-generation skill hardcoded /Users/guohao/Documents/... — the author's Mac path — into the body. Another used C:/Users/vudrk/Desktop/AI Projects/ as the base for every script. None of these error out; they die quietly for the next person. carrylint's reason to exist was sitting in the real data.

The bad half. About 85% of my errors were false positives. I was flagging Bearer YOUR_API_KEY (the standard API-docs convention for "put your key here") as unfinished. I was flagging claude mcp add — the host itself — as an undeclared CLI. I was flagging $HOME/..., which resolves per user and is the portable way to write it, as a machine-specific absolute path. I'd written that false positives are a linter's only cause of death, and then nearly shipped promotion on top of an 85% false-positive rate.

Luckily the real data named exactly what to fix. v0.1.1 corrected four things: $HOME/~/generic names are portable; placeholders narrowed to genuine fill-me markers; host CLI setup commands (claude mcp add etc.) excluded; the home-relative path rule dropped entirely. Re-run against the same 230: error false positives went from ~85% to near zero, and every true positive survived. I then ran it against 70 repos it had never seen to check I hadn't overfit (3% fired, all genuine). The real examples I found are bundled in the repo as regression tests.

One lesson: your own tests passing is not evidence of correctness. You find out by running against real data — and by doing it before you promote, not after.

Honestly: this niche is already crowded

One more admission. Partway through designing this I went looking, and there were already 7+ linters for SKILL.md as of 2026. My own skills-lint is one of them. This was not an empty lot.

But I read the ones I could find, and none of them looked at whether the contents actually run somewhere else. They stop at spec compliance and frontmatter. So carrylint does only that. Teams that mix Claude and Codex and distribute skills to each other are honestly still rare — the demand may be slightly ahead of its time. Still, if "I shared it and it only ran for me" has ever stung you, the low-noise error rules (sharpened by that audit) should earn their place.

Summary

The standard made SKILL.md portable as a format. Whether it runs is a different question. carrylint fails CI on that difference.

- uses: hyuga611/carrylint@v0     # in CI

npx @hyuga/carrylint              # right now, locally

Repo: https://github.com/hyuga611/carrylint

Don't trust "Done." — forcing AI agents to re-fetch reality before they report completion

hyuga — Sun, 16 Aug 2026 02:44:59 +0000

"Inserted the rows. Done." — except not a single row had landed

I hand a lot of my client work to AI agents. Production deploys, report generation, bulk data inserts. Every procedure that works gets turned into a skill, and by now a few dozen skills run my day-to-day.

The one that broke me was a bulk insert. At the end, the agent said:

"Inserted N rows. Done."

A completion report like any other. I almost took it. I opened the admin panel just in case — not one row had been added. The insert command had failed partway through, the error had been swallowed, and the agent had confidently reported completion anyway.

That was a cold moment. What if it had been a number in a report? A deliverable going to a client? How many "done" reports had I already waved through that were never actually done?

This is the nastiest species of hallucination. It isn't a plausible-sounding sentence. It's the fabrication of the fact that the work was completed at all. And humans don't interrogate a confident "Done."

Why agents lie (it isn't malice)

Chase the cause and it isn't that the model is dishonest. It's that acting and checking are the same step.

One turn of an LLM agent looks roughly like this:

Call a tool (run the insert)
Look at the return value, generate the next sentence
Write "Done."

Step 2 is the problem. When the tool's return value is empty, ambiguous, or a timeout, the model writes the most likely continuation — "it probably worked." In the training data, the most natural thing to follow "ran the insert" is "done", so of course that's what comes out.

The agent is claiming completion without ever looking at the world it just changed. Having never looked, it can't notice the failure either.

There's one fix: "done" may only be claimed from re-fetched real state

The countermeasure is simple. An operation with side effects must re-read the state of the world afterward and confirm it, before anything is allowed to say "done." I call this the completion contract and made it a shared rule across all my skills. It's three lines:

Anything with side effects — create, update, delete, upload, insert — re-fetches existence and state with a separate command before saying "done." Inserted rows? Count them again. Uploaded a file? Hit the URL. Created a record? Look it up by id. Never use the return value of the same tool call as evidence — that's exactly where "I think I did it" breeds.
Empty output, errors, and timeouts are never filled in. If a count, path, or id couldn't be read, don't guess it — report "empty" or "failed" as-is.
Numbers that go into a ledger or report must be values confirmed by a re-fetch. Anything unconfirmed doesn't get written.

Put differently: the burden of proof moves to the agent. Not "it should have worked" but "I re-read it just now with my own eyes and it is there."

Here's the difference.

Before (I think I did it)

> Inserting the data
[tool] db_insert(...)   ← empty return value, but not conclusively an error
> Inserted N rows. Done.   ← claimed without looking

After (completion contract)

> Inserting the data
[tool] db_insert(...)   ← empty return value
> The insert returned nothing. Verifying that the rows exist.
[tool] db_count(where=...)   ← re-fetch
> 0 matching rows. The insert did not land (failed).
  Want me to check the error log and retry?

The second agent can't lie. The moment you insert a step that goes and looks at the world afterward, "I think I did it" stops being physically possible.

Why the existing guardrails don't cover this

You might think a guardrail library already handles it. I looked — guardrails-ai, deepeval, promptfoo, a few others. They're good tools. They just guard a different place.

What they validate is essentially the text the LLM produced: is the format right, is it harmful, is it factually consistent, is it close to the expected answer. They grade the content of the output.

But the "I think I did it" hallucination is flawless as text. "Inserted N rows. Done." is grammatical and internally consistent. No amount of grading the text catches it — because what's wrong isn't the text, it's the state of the world after the action.

There's a gap here. There are piles of tools that verify output, but as far as I could find, nothing that re-fetches the world after an action and reconciles the report against reality. We've reached the point where agents actually rewrite the world, and verification stopped at "what did the agent say."

You can do this today, with no library

The most important part first: this contract works right now, with no dependency. Drop the three rules into your agent's system prompt, CLAUDE.md, or AGENTS.md as a single paragraph:

## Completion contract
An operation with side effects (create, update, delete, upload, insert) may not be
reported as complete until a separate command has re-fetched the resulting state and
the raw result has been shown. Empty output, errors, and timeouts are reported as
"empty" or "failed" as-is — never filled in with an imagined id, path, or count.

That alone visibly reduced false completions in my setup. Zero cost, zero dependencies. If you only try one thing, try this.

But prompts don't hold

Here's what I learned running it: discipline written into a prompt gets quietly broken on a busy turn. As context grows, the model steps over that paragraph "by accident" and starts saying "Done." again. Like a human promising to be careful, a declaration gets broken.

At some point you want the discipline enforced, not just written — the same way you replace a verbal note in code review with a linter that mechanically fails CI. I wanted "did you actually re-fetch before saying done?" backed by machinery instead of good intentions.

So I built the piece that enforces it — `genchi`

I published a small piece that backs the completion contract with machinery rather than goodwill: genchi (現地現物 — go and see the actual thing).

npm i @hyuga/genchi

It does exactly one thing: run a probe that re-fetches real state, and issue a verdict from nothing else.

import { gate, expect } from '@hyuga/genchi';

await db.insert(rows);            // the side effect
await gate({
  action: 'insert 45 rows',
  probe: () => db.count({ where: { batch: 123 } }), // ← re-reads real state, not the action's return value
  expect: expect.count(45),
});
// reaching this line means the 45 rows are really there. Otherwise it threw GenchiIncomplete.

The crux is that verify / gate accept nothing but a probe. The evidence has to come from calling something at the moment completion is asserted, not from a value handed in beside the claim. Empty results, errors, and timeouts aren't swallowed; they're reported as failures rather than imagined into successes. A returned count of 0 (nothing landed) counts as incomplete too.

One thing that requirement does not buy, and I claimed it did. Until 0.3.0 the README said this made "I think I did it" structurally unwritable. It is one line:

const result = await doTheInsert();          // suppose nothing landed
await verify({ action: 'insert 45 rows',
               probe: () => result.inserted, // the action's own return value
               expect: expect.count(45) });  // → ok: true

A probe is a function, and nothing in JavaScript can force a function to do I/O. Worse, the CLI printed re-fetched: 45 for a --probe "echo 45" that re-fetched nothing — a tool whose entire subject is "don't report what you didn't check", asserting in its own output a thing it had not checked. Both are corrected in 0.3.0: the wording is now the probe returned, and --help states the limit without being asked.

What requiring a probe actually buys is a place to put the re-read — an expression somebody wrote on purpose — plus refusals that don't get quietly swallowed. That is worth having. It is less than I wrote down, and the difference is the sort of thing you only find by attacking your own package instead of re-reading it.

There's a CLI for agents that don't write JS. Hand it a re-fetch command:

genchi verify --probe "psql -tAc 'select count(*) from t where batch=123'" --count 45
# exit 0=verified / 1=empty or mismatched / 3=probe failed. Raw probe output is always emitted as evidence.

With Claude Code, a Stop hook can block a turn that still has unverified completion contracts on it (adapters/claude-code). No LLM and no API key at runtime — it's a zero-dependency static piece.

Honestly: this may be slightly ahead of demand

Let me be straight. As far as I could find, "re-fetch the world after the action and verify" was a genuine gap. But I don't think that many people have yet been burned specifically by fabricated completion while letting agents rewrite real things. The demand may be a little ahead of its time.

I still built it framework-agnostic on purpose — the Claude Code hook is pushed out to a thin adapter and the core works from any agent. I have a linter (carrylint) whose whole message is "don't bake your environment in", so it would be incoherent for my own tool to be Claude-Code-only. Unlike my static linters (reflint for reference integrity, skills-lint for skill collisions, carrylint for runtime portability), this is the first piece of mine that goes and looks at the world at runtime.

If you've ever had that cold moment over something that was reported done and wasn't, it should land.

Summary

The worst hallucination from an AI agent isn't prose — it's the fabrication of having completed the work. The cause is that acting and checking are the same step.
There's one fix: "done" may only be claimed from re-fetched real state. Empty and failed get reported as empty and failed.
Existing guardrails validate text output. Re-fetching world state after an action and reconciling it is the gap.
The contract works today as a single prompt paragraph. To enforce it in code: npm i @hyuga/genchi.

Go doubt one more "Done."

Repo: https://github.com/hyuga611/genchi

I attacked my own npm package before launching it. It let the proposer approve their own writes

hyuga — Thu, 13 Aug 2026 10:13:32 +0000

I maintain a library whose entire job is one sentence: a person who did not write this statement looked at what it actually does, and said yes.

An LLM proposes an UPDATE. The library runs it for real inside a transaction, reads the before and after values back out of the database, rolls back, and shows a human the measurement. Not a summary the model wrote about its own SQL — the values the database produced when the statement ran.

Last week I found out it was letting the proposer be the person who said yes. And writing "approved" into the audit trail when they did.

Why I attacked it in the first place

I saw someone get taken apart in a shellcheck issue thread. They had posted an alternative tool, and the reply was:

A cursory glance also tells you it's a vibecoded clone of shellcheck.

One glance. That's the bar now, and my GitHub profile fits the pattern that gets you that reply — eleven repositories published in a month. I do have tests. 386 of them, passing, including 161 against real MySQL 8.4 and PostgreSQL 16 in containers.

But a passing test suite only contains the attacks you already thought of. So before launching I installed my own published package with npx, as a stranger, and went looking for a way to break it.

The first thing I tried worked

I used the example from my own README — a privilege escalation riding along inside a quota change:

$ llm-safe-sql plan "UPDATE members SET quota = quota + 10,
    profile = '{\"role\":\"admin\"}' WHERE id = 7" --as kenji

Measured by running the statement and rolling it back
  id = 7
      quota: 5 -> 15
      profile: {"role":"user"} -> {"role":"admin"}

This needs a person. Neither the assistant nor this tool can approve it:
  llm-safe-sql approve <id> --as you@example.com

The tool says it out loud: this needs a person, neither the assistant nor this tool can approve it. So I approved it as the same person who proposed it.

$ llm-safe-sql approve <id> --as kenji
Approved.

$ llm-safe-sql apply <id> --as kenji
Applied: UPDATE on members, 1 row(s).

Committed. quota=15, role=admin, confirmed by querying the database directly rather than trusting the tool's own output.

The commit wasn't the frightening part

The audit table was.

phase=planned    actor=kenji
phase=approved   actor=kenji
phase=applied    actor=kenji

There's an approved row. Anybody reading that trail later concludes a human reviewed this change. Nobody read anything.

A missing audit trail is better than that. A missing one tells you it's missing. A complete one that describes a review which never happened is a record that decisions get built on top of. I had shipped a machine that stores the absence of review as review.

The part I can't excuse

I had thought about this attack. Just not all of it.

My spec document already said:

P5 — This library's own plan and audit tables are refused regardless of configuration. A model that can write the plan table can approve its own writes.

So I knew self-approval was the thing to prevent. I closed the indirect path — tamper with the plan table to mark something approved — and left the front door open. You just run approve.

The README's answer to "the model can't approve" was that approve lives in a separate process the model has no path to. That's true of the deployment I recommend. It is not true of the one npx gives you, and nobody starts at the recommended deployment.

Designing the check was the interesting part

The comparison itself is three lines. The judgement calls were:

Ignore case and surrounding whitespace. A check that --as Kenji walks past is theatre. It refuses the honest caller and waves through anyone who hit shift.

Don't get clever about matching. alice@example.com can approve a plan proposed by alice. Refusing it because it contains the string would lock out a legitimate second reviewer, and a security check that blocks honest use is a security check that gets switched off. I made it deliberately dumb.

Ship an escape hatch. Some people really do hold both roles — a solo operator with nobody to hand the card to. --allow-self-approve exists. It approves the plan and leaves both acts under the one name in the audit trail. It buys you an apply. It does not buy you a tidier story about who reviewed it.

Then I nearly shipped something worse

Fix written, tests green, about to publish. And I realised:

--as is self-asserted.

My new check compares two strings handed to the same process from the same untrusted place. It stops one identity running both halves — an agent and its operator sharing $USER, which is exactly what a single terminal gives you and exactly how the plausible-looking audit trail gets manufactured by accident. It does nothing about a person who types a different name.

And my own README says this, in a section about which guards are real:

Most of what this library does runs inside this process, holding a credential that can write. That is worth saying out loud, because the alternative is an operator believing in a boundary that turns out to be one if statement in a library they have never read.

I had just written an if statement. Shipping it quietly would have meant the release that adds the guard also commits the exact defect that paragraph exists to prevent.

So check now says it every run, in every configuration:

! `--as` is taken at its word: nothing here authenticates anybody.
  So the refusal that stops a proposer approving their own plan catches
  one identity running both halves — an agent and its operator sharing
  $USER, which is what a single terminal gives you — and does not catch
  a person who types a different name.
  Actor separation is a record, not a boundary.
  The boundary is applyConnection: a database account the proposing side
  has no password for.

State what the guard buys and what it doesn't, before anyone asks. What it buys is that a silent non-review becomes a refusal. What it does not buy is authorisation. The only identity here that means anything is a database account the proposing side has no password for. There's a test pinning that line in place, because if it silently disappeared nobody would notice.

One thing I deliberately did not fix

The same audit turned up that anyone can cancel anyone else's plan.

I left it. Cancelling only ever prevents an apply, the MCP surface doesn't expose it, and everyone who can reach it can already write to the plan table directly. Adding a name check there would put a second authorisation-shaped string comparison next to one that already needs a paragraph explaining it isn't authorisation.

It's written into the spec under "out of scope" so it reads as a decision rather than a gap somebody closes by reflex.

If you run something like this

Try approving a plan under the same identity that proposed it. If it goes through, your audit log is recording reviews that did not happen.

If you were on 0.5.2 or earlier of mine, this finds the affected records — verified on MySQL 8.4, PostgreSQL 16 and SQLite:

SELECT p.plan_id, p.actor AS proposed_by, p.logged_at, p.detail
FROM llm_safe_sql_audit p
JOIN llm_safe_sql_audit a
  ON a.plan_id = p.plan_id AND a.phase = 'approved'
WHERE p.phase = 'planned'
  AND LOWER(TRIM(p.actor)) = LOWER(TRIM(a.actor))
ORDER BY p.logged_at;

Rows that come back were committed on one person's word while the trail reads as though two people were involved.

Human-in-the-loop only means anything if the loop checks that the approver is somebody else. Mine didn't, from the first release through 0.5.2, while saying otherwise on the tin. The test count had nothing to say about it — the hole showed up the first time I installed my own package as a stranger and typed the laziest possible thing.

Fixed in 0.6.0.

Repo: https://github.com/hyuga611/llm-safe-sql
npm: @hyuga/llm-safe-sql

Your agent can write to your database. Which layer is actually stopping it?

hyuga — Wed, 12 Aug 2026 07:11:10 +0000

In July 2025, Replit's agent deleted a live production database
during an explicit code and action freeze. Replit's CEO called it unacceptable
and shipped dev/prod separation and better rollback in response.

The freeze was real. It was written down, it was agreed, and the agent had been
told about it — and none of that was in the execution path. The instruction lived
in the model's context, and the model's context is not a place where rules are
enforced.

There is a second detail worth sitting with. Afterwards, the agent reported that
rollback was impossible. That was wrong, and it delayed the recovery. So the
system's own account of what it had done was not merely unhelpful; it was
confidently false in the direction that mattered.

That incident is the shortest version of the question this post is about. Every
tool in this space has a guardrail. They differ enormously in which layer that
guardrail lives at, and almost none of them tell you.

I went looking for a comparison organised that way and could not find one, so
here is mine. I have a tool in this space myself; it is at the end, alternatives
to it are named before it, and I have tried to describe everything else the way
its maintainers would.

An earlier version of this post was reviewed by people looking specifically for
places where I had flattered my own tool or misdescribed someone else's. They
found several. The corrections are in the text rather than in a footnote.

The ladder

There are five places a rule about database writes can be enforced. They are not
equally good, and the difference is not a matter of degree.

Layer	Enforced by	Gets past it
1. Model context	prompt text, tool descriptions	the model, by ignoring it
2. Client	the IDE or chat app's approval dialog	the user, by clicking "always allow"
3. Server process	the MCP server's own code	anyone holding the credential it uses
4. Proxy / gateway	a network hop	connecting directly
5. Database	roles, privileges, engine settings	nothing, short of another credential

The useful property of this ladder: each rung survives the failure of every
rung above it. A database role with no write privileges does not care whether
your allowlist has a parsing bug, whether the user clicked "always allow", or
whether the model was talked into something by a support ticket it read.

Most of the disagreement in this space turns out to be tools sitting at different
rungs while using the same words.

Layer 1: the model's context

This is tool descriptions, system prompts, and instructions returned inside tool
results — "only use this for read queries", "ask the user before applying".

It has no security value. It is worth writing, because it improves ordinary
behaviour and ordinary behaviour is most behaviour. It is not a control, and the
July 2025 incident is what it looks like when someone believes otherwise.

There is a specific version of this worth naming, because it is easy to mistake
for layer 3. Some MCP servers implement a two-step write flow — prepare, then
complete — where the separation between the steps is that the tool description
tells the model to verify in between. If nothing refuses the second call when the
first was never really checked, that is layer 1 wearing layer 3's clothes.

Layer 2: the client

Every serious MCP client shows a confirmation dialog before a tool call, and the
agent frameworks have their own equivalents: LangChain's
HumanInTheLoopMiddleware and interrupt, the OpenAI Agents SDK's
needsApproval with resumable interruptions, Pydantic AI's requires_approval
deferred tools. These are real, well-built mechanisms, and if you are building an
agent that writes to anything, you should use them.

Two honest problems.

What the human is shown is usually the model's own tool input. For a database
write that means a SQL string, rendered as JSON. Not the rows it will touch, not
how many, not what they currently contain. The same clients that render a file
edit as a colour diff render a database mutation as a blob of text. You are being
asked to approve a statement, which means you are being asked to be a SQL
interpreter, in your head, against data you cannot see.

And these gates get turned off. --dangerously-skip-permissions, auto-run
modes, allowlist creep. This is not user error; it is the predictable result of
asking someone twenty times an hour whether they meant it. Any control whose cost
is a click, paid repeatedly, converges on being disabled.

A design that dodges both problems, and is under-credited because it is boring:
have the agent produce an artefact that goes through review you already have.
A migration file in a pull request. A row in a staging table that a trusted job
applies. The human then reviews in a tool built for reviewing, with history and
blame and a second pair of eyes, instead of in a modal that interrupted them.

Layer 3: the server process

This is where most database MCP servers put their guard, and it is where the
interesting failures are, because a server here holds a credential that can write
and is choosing not to use it.

There are at least four distinct mechanisms, and they are not equally strong.

Wrapping the query in a read-only transaction. The original MCP reference
server for Postgres did this: BEGIN TRANSACTION READ ONLY, run the SQL, roll
back in a finally. Datadog Security Labs
showed it was bypassable
by statement stacking: the Postgres driver accepts multiple semicolon-separated
statements in one call, so COMMIT; DROP SCHEMA public CASCADE; ends the
read-only transaction and everything after it runs with full privileges.

Their recommended fix is the whole argument of this post, from a security team
that had just finished breaking the application-layer version of it:

A possible mitigation—one we recommend in any case—is using a Postgres user
with restricted privileges. You should definitely do this.

Their conclusion is the unglamorous one, and worth repeating: "classic
application security vulnerabilities are still very relevant to MCP servers and
other AI tooling."

The lesson generalises, in my words rather than theirs: a read-only transaction
is not a security boundary when the protocol accepts semicolons.

That server was deprecated in July 2025 and archived. It still had 86,941 npm
downloads in the week of 2 August 2026, which I checked while writing this.

Blocking keywords. AWS's database MCP servers default to read-only and reject
INSERT, UPDATE, DROP, session-state statements and a list of dangerous
functions. Their own README is unusually straight about what that buys you:

Treat this as a best-effort, defense-in-depth mechanism, not a security
boundary. A blocklist cannot enumerate every dangerous construct, and a
sufficiently creative query (obfuscation, quoted identifiers, new
server/extension functions, etc.) may bypass it. Do not rely on it as your
only control.

and then says the thing this entire post is about, better than I did:

Combining a minimal-privilege role (database-enforced) with the blocklist
(application-enforced) gives you defense in depth.

Database-enforced and application-enforced. That is the distinction, in a
vendor's own documentation, and it is the one almost nothing else makes explicit.

Parsing the SQL properly. Better than keyword matching, and it moves the
problem rather than removing it: you are now maintaining a SQL parser that must
agree with the database's parser about every dialect quirk, forever. Disagreement
between the two parsers is the whole bug class.

Not registering the write tools at all. Neon's read-only mode restricts which
tools exist. This is clean and easy to reason about — but it is still layer 3: the
credential in the process can write, and anything that reaches it can too.

A trap worth naming, because I nearly filed it under layer 5 myself:
Postgres's default_transaction_read_only looks like engine-level enforcement
and is not. It is a USERSET parameter — a default, not a privilege — so
SET default_transaction_read_only = off; or BEGIN READ WRITE; restores writes
from inside the very session it is supposed to constrain. The same semicolon that
beats a read-only transaction beats this. It is only load-bearing when the role
has no write grants, in which case the grants are the control and the setting is
decoration.

There is also a family of servers that deliberately hands the model transaction
control — begin transaction, commit, rollback as callable tools — or a
general execute_sql that accepts multiple statements. mcp-node-mssql and
mcp-sqlite-tools are examples I verified. This is a legitimate design for a
trusted local development database. It is worth being clear-eyed about what it
means in any other setting: the model holds the commit button.

The choice is not forced, and it is worth naming a server that goes the other
way. mssql-mcp-node is read-only by default, gates writes behind an
MSSQL_ENABLE_WRITES environment variable, and runs every read inside a
transaction it always rolls back. Same protocol, same language, opposite default.

Layer 4: the proxy

An MCP gateway that sits between client and server can gate tool calls centrally,
log them in one place, and apply policy from an engine like OPA or Cedar. It is
the right answer to "which of our forty agents may reach which of our servers",
and it survives a compromised client.

Its limit is structural: it protects the path through it. If the credential is
also reachable directly, the proxy is a convention.

Layer 5: the database

This is the only rung where the enforcement outlives everything above it.

A role without write privileges. Supabase's MCP read-only mode runs queries as a read-only Postgres user — enforcement at the role, not in the process. It is opt-in rather than default, which is a real criticism, but the mechanism is the strong one.
Engine-level read-only. ClickHouse's readonly=1, which also forbids changing settings, and SQLite's read-only connection.
Scoped write authority, which barely appears in agent discussions and is the mature answer to the actual question. Row-level security, column-level grants, updatable views, SECURITY DEFINER functions as the only write path. "May write, but only these rows" is a solved problem with a decade of production use, and it maps onto agents exactly.
No raw SQL at all. Typed, parameterised endpoints — PostgREST, Hasura, stored procedures, an MCP server exposing verbs instead of a SQL box. The agent picks an operation and fills typed slots; the SQL was written by a human in advance. This is the most common answer in production systems and the least discussed in agent writing.

The catch is that layer 5 controls what may happen and cannot tell you what
will. UPDATE orders SET status='shipped' WHERE placed_on < '2026-08-01' is
permitted by any sane privilege model. Whether it touches four rows or forty
thousand is not a privilege question.

Three things worth saying plainly

Read-only is not the same as safe. Removing writes does not remove risk: an
unbounded read is a denial of service and a cost incident, and read access is the
exfiltration half of every serious prompt-injection scenario. Several of the
worst documented MCP incidents involved no write at all.

Approval cannot be the whole answer, because a growing share of agent runs
have no human present — cron jobs, CI, batch runs. For those, the only workable
combination is layer 5 plus caps plus reversibility plus audit.

Reversibility deserves equal billing with prevention, and gets almost none.
Point-in-time recovery, time travel, instant branch restore, temporal tables,
soft deletes. In practice this is what most teams actually rely on, and the
questions that matter — how far back, does it cover DDL, can the agent itself
call the restore, and would you even notice you needed it — are rarely asked
before they are needed.

Measuring before approving is not a new category

I should name the alternatives here too, since this is the one part of the ladder
where I sell something.

Bytebase's SQL review computes affected rows and a risk level and routes the
change to a human before rollout. Branch-based databases — Neon, PlanetScale —
let you apply to a copy and inspect the result before promoting it, which answers
the same question from a different direction and gives you a rollback story for
free. And the cheapest version needs no tool at all: have the agent produce the
matching SELECT, or run the statement in an open transaction with RETURNING
and read it back before you commit.

Where my own tool sits, honestly

I maintain llm-safe-sql, which
attacks the layer-2 problem specifically: that the human is approving a claim.
It runs the proposed UPDATE or DELETE inside a transaction, measures the real
before/after values, always rolls back, and shows the human that measurement.
Applying happens in a separate process, from a separate credential.

Being honest about the ladder, and more honest than my first draft was. The
measurement runs at layer 3. Both the planning credential and the applying
credential can write — a dry run cannot execute without write privileges — so the
database is not enforcing the split between them. The library and the process
boundary are. Calling the separate apply credential "layer 5", as I first wrote,
was promoting my own tool one rung on my own ladder.

The one genuinely layer-5 setting it has is a separate read-only role for the
model's reads, which is also the cheapest thing an operator can do here. And its
check command now prints where each guard actually sits and names the ones that
are fictional, because "apply uses the same credential as plan" is something an
operator deserves to be told rather than to discover.

And the honest limits: this is human-in-the-loop, so unattended agents are out of
scope. It does not solve prompt injection — a measured diff tells you what will
happen, not who chose it, though "what will happen" is a great deal more than the
model's own description of it. And gating database access on a human decision is
not new — privileged-access brokers such as StrongDM have checked commands
against policy mid-session for years, long before anyone was pointing a language
model at a production database.

The question to ask

If you are evaluating anything in this space, the question that separates the
options is not which features it has. It is:

When your guardrail refuses something, what is doing the refusing — and what
would still refuse it if that component had a bug?

If the answer is "the tool description", you have a convention. If it is "the
approval dialog", you have a convention plus a habit. If it is "a role that
cannot write", you have a control.

Most tools are a mix, which is fine. Not knowing which is which is not.

Sources

Everything named above was checked against a primary source while writing. Where
a claim came from a vendor's comparison page about a competitor, I dropped it.

MCP vulnerability case study: SQL injection in the Postgres MCP server — Datadog Security Labs
Replit AI wiped a database and called it a catastrophic failure — Fortune, July 2025
awslabs/mcp — postgres-mcp-server
Supabase MCP server docs
neondatabase/mcp-server-neon
ClickHouse MCP server
crystaldba/postgres-mcp
bytebase/dbhub and Bytebase SQL review
mihai-dulgheru/mssql-mcp-node — read-only by default
PostgreSQL: default_transaction_read_only
LangChain human-in-the-loop middleware
OpenAI Agents SDK — guardrails and human review
Pydantic AI — deferred tools and approval

Download figure for @modelcontextprotocol/server-postgres taken from the npm
registry API for 2026-08-02 to 2026-08-08.

The card said one column. The apply wrote two.

hyuga — Mon, 10 Aug 2026 09:56:46 +0000

I have been building a thing that lets a language model propose an UPDATE, then
executes it for real inside a transaction, measures the actual before and after
values, and always rolls back. A human reads the measurement and decides. Only
then does anything commit.

The pitch is one sentence: what you approve is not the model's description of
its SQL, it is what the database did when the SQL ran.

Last week I found that the thing showing you that measurement was showing you a
subset of it, and had been since the first release.

The failure

Real output, from @hyuga/llm-safe-sql@0.4.0 installed from npm. One row:
name = 'Tanaka', postcode = '00100'.

  UPDATE customers SET name='Sato', postcode='00100' WHERE id=1

What this touches
  customers — Customer records. The postcode is used for billing address and delivery.
  1 row would change, across 1 column: name

Measured by running the statement and rolling it back
  id = 1
      name: 'Tanaka' -> 'Sato'

One row, one column. postcode is not mentioned, and that is correct — it is
being assigned the value it already holds, so nothing about it changes. The card
is describing the diff accurately.

Approve it. Then, before it is applied, somebody else notices the postcode is
wrong and fixes it:

UPDATE customers SET postcode='90210' WHERE id=1;

Now apply the approved plan:

Applied: UPDATE on customers, 1 row(s), at 2026-08-10T09:49:12.049Z.
DB now: [{"name":"Sato","postcode":"00100"}]

The fix is gone. Zero warnings. The word postcode never appeared on the
approval card, never appeared in the audit record, and never appeared in the
comparison the tool makes before it commits.

One variable doing two jobs

The diff was built like this:

const changed: string[] = [];
for (const c of Object.keys(before)) {
  if (same(before[c], after[c])) continue;   // drop what did not move
  if (auto.has(lower(c))) continue;          // drop what the DB maintains itself
  changed.push(c);
}

That is a correct answer to "what should the card show". Showing postcode: '00100' -> '00100' would be noise, and worse than noise — it would be the card
claiming a change where there is none.

The problem is that changed was also the list the apply iterated when it
re-checked that nothing had moved since approval:

for (const c of plan.changed) {
  if (!same(live[c], plan.before[c])) throw new ApplyRefused('ROW_CHANGED', …);
}

postcode is not in changed, so it is not checked. And the statement writes it
on every execution. A column that is written and never verified.

"The set of columns that change" and "the set of columns the statement writes"
are different sets, and I had used one name for both. SET x = <the value it already has> is not exotic SQL — it is zero-padded codes, defensive status
assignments, SET updated_by = 'batch'. Every one of those is this shape.

MySQL was catching it by accident

There is a second version of the same hole, one level up.

When a WHERE matches several rows and one of them already holds the target
value, the card says:

  1 row would change, across 1 column: status
  (1 more match the condition but are already correct.)

That row's changed is empty, so the verification loop runs zero times — before
the write and after it. Nothing about that row's contents was ever checked.

MySQL was saved by an accident of its protocol. It reports "rows matched" and
"rows changed" separately, so a plan that measured one changed row and an apply
that changed two is a detectable disagreement. PostgreSQL and SQLite rewrite a
row even when the new values equal the old ones, so those two numbers are the
same and the comparison says nothing. The reconciliation code knew this:

if (plan.op === 'UPDATE' && plan.rowsChangedIsMeaningful && res.rowsChanged !== plan.rowsChanged) {

rowsChangedIsMeaningful is true on MySQL and false on the other two. A guard
that worked on one of three supported engines was the only thing standing there.
Develop on MySQL, test on MySQL, and you never learn this.

The fix was to stop conflating the two sets: PlanRow now carries covered —
every column the statement assigns — snapshotted before and after even when the
value does not move, included in the tamper digest, and checked at both ends of
the apply. changed still drives the display. Same sequence on 0.4.2:

Refused (ROW_CHANGED): Row id=1 no longer holds the value you approved:
`postcode` was '00100' when the plan was made and is '90210' now.
Nothing was applied — make a new plan against the current values.

The audit that did not open the file

I found this because of how the previous audit failed.

I had run an adversarial review over the codebase — several independent passes by
dimension, each finding verified by separate sceptics whose job was to refute it.
Twenty-one findings survived. Adapters, engine, parser, policy. It felt like a
good day's work.

Then I asked a different question: not "what did you find" but "what did this
review structurally not look at?" The first line of the answer:

src/apply.ts (494 lines at the time) produced zero findings. That should be read as
"nobody opened it", not as "it was clean".

apply.ts is the only code in the library that writes to production. Dry runs
always roll back. Approval only writes a record. Committing happens in exactly
one place, and the review had not been there.

Scoped to that one file, the same process returned twenty-three findings —
more than the first pass found in the whole rest of the codebase. Everything
above came out of it.

The number 21 had felt like progress. It was a record of where I had looked.

Then the examples found two more

This week I wrote worked examples: the four database accounts the design assumes,
with the exact grants, for MySQL and Postgres. I decided to run every line against
a real server rather than write what I knew to be true.

The server disagreed twice.

A database-wide grant cannot be narrowed. I had written the obvious thing —
grant DML on the whole schema, then take it back on the two tables that hold the
approval records:

ERROR 1147 (42000): There is no such grant defined for user 'llm_plan'
                    on host '%' on table 'llm_safe_sql_plans'

MySQL will not revoke a table-level subset of a database-level grant. So
GRANT ... ON shop.* hands the dry-run account write access to the table that
records approvals, permanently. A dry run could forge its own approval. The
examples name each table instead.

And check did not check. The tool has a command whose entire job is to say
whether the environment will work. It verified four connections and never the two
tables the whole approval record lives in. With them missing it reported every
table as ready and exited 0 — and the omission surfaced on the first plan, as
a driver error escaping as an unhandled rejection, after the dry run had already
executed and rolled back.

That is the likeliest mistake anyone makes on day one, and it was invisible to the
command that exists to catch exactly that class of thing. Fixed in 0.4.2: it
reports the missing table and exits non-zero, and a missing store table is a
refusal on every path rather than a stack trace.

There was a third, smaller one. check now asks the catalogue whether the tables
exist rather than issuing SELECT 1 FROM audit WHERE 1 = 0 — because the store
account the examples recommend holds INSERT on the audit table and nothing else,
so the select probe reports the table missing exactly when the credential is as
narrow as it is supposed to be. Writing the recommended configuration is what
made that visible.

And then CI failed, because CI runs the README's own quick start verbatim, and my
fix had made the documented order wrong. The instructions said check then
migrate; check now exits non-zero before migrate has run. Something noticed
before a reader did.

What I actually take from this

Count what you have not looked at. A list of findings looks like evidence of
thoroughness and is evidence of coverage. The file that should obviously have been
at the top of a risk-ordered list did not appear on the list at all, and absence
looked exactly like safety.

Writing the explanation is a test. Twice now, the act of documenting this
thing has found defects in it that reading the code did not. Not because prose is
magic — because writing an example means running it, and running it means the
server gets a vote. Two privilege lists that were obviously right were wrong.

Watch for one name doing two jobs. The bug here was not a missing check. It
was a display set and a verification set sharing a variable, so narrowing the
display for good reasons silently narrowed the check. If you have something that
both shows a human what will happen and decides whether it may happen, those are
two lists, and they should have two names.

A guard that only works on one backend is not a guard. It is a coincidence
with good timing, and it will be silently absent on the day you switch.

If you are building anything that measures before it asks a human to agree, the
question I would now ask it is: what is the interface not showing me, and how
would I know? The answers worth having are specific — which columns does it
compare, what does it do with a value too long to print, what does it do with two
values that look the same and are not.

None of the five layers people usually reach for would have caught this. Not a
role without write privileges, not a proxy, not an approval dialog, not a stricter
prompt. Every one of these was a legitimate, authorised UPDATE on an allowlisted
table by a credential entitled to run it. Nothing about what was permitted was
violated. What was wrong was what the human was told before they agreed.

Code, and the full list of what changed in each version, at
github.com/hyuga611/llm-safe-sql.
The examples/ directory is the part I would read first — it is the only
documentation I have written where every line was executed against a real server
before it was committed.

DEV Community: hyuga

You are in the cockpit now. Nobody gave you instruments.

Am I flying this, or am I a passenger?

The panel

A limiter is not there to slow you down

My own instrument lied to me

Past the limit, it does not stop you

Converging and diverging are opposite jobs

Three CI files that had never run once

Where this actually pays off

Where I think this goes

What none of this buys

cd, type go, and your workspace builds itself — tsumugi 0.3

The premise

1. go builds a workspace in one message

One message, not four

The tree has no motions of its own

The listing is read server-side

It coexists with the Go toolchain

2. Paste a screenshot into what you are telling the agent

3. tsg update starts no external process

Why the download-and-run shape had to go

After the swap, no reopening and no --kill

Install

Also fixed in 0.3.x

I built a terminal because I could never remember the keybindings

One idea: the screen is a document

Selecting "a command and its output" honestly

Every command has a mouse path — enforced by a test

Running several AI agents turned me into a search function

Losing the machine doesn't lose the session

Reviewing what the agent wrote, where you are

Also in the box

What building it actually taught me

Being straight about the state

Your AGENTS.md is lying to your agent. So was my linter.

1. A flag I had advertised in a blog post had never done anything

2. pnpm -r build was reported as "no script named -r"

3. One misplaced fence and it checked nothing, then said everything was fine

If you go quiet, say why you went quiet

The numbers

A different shape, same lesson: only the people who wrote it correctly got warned

The part I actually want to leave here

Try it

I asked one question. All four of my linters failed it.

The question

The worst one was not a false positive

Broken in both directions at once

When a miss flatters your own headline number

One caveat about how this was found

Two questions worth asking your own tool

I audited 2,465 published agent skills. 192 of them cannot be selected the way the spec says

192 skills have no frontmatter at all

The rest of the breakdown

The number that mattered was my own

What I had wrong

The finding that looks textbook and isn't

One report was closed as not planned, and closing it was right

What I did not publish, and why

Reproducing

My linter kept warning the people who did it right. Three times, in the same direction

The warning landed on the only people who had done it properly

Then it happened again. Twice, in one release

Why it points that way

The audit that found nothing here

What actually fixed it

One more thing, since your linter runs on other people's files

The rule of thumb I'm keeping

That amazing skill you shared only runs on your machine. I built a linter for it — then found I was 85% wrong

I mass-produced skills by saying "turn this into a skill." Then I shared them

The standard made the format portable. Whether the contents run is a separate question

What it catches

Then I ran it against 230 real skills and learned I was 85% wrong

Honestly: this niche is already crowded

Summary

Don't trust "Done." — forcing AI agents to re-fetch reality before they report completion

"Inserted the rows. Done." — except not a single row had landed

Why agents lie (it isn't malice)

There's one fix: "done" may only be claimed from re-fetched real state

Why the existing guardrails don't cover this

1. `go` builds a workspace in one message

3. `tsg update` starts no external process

After the swap, no reopening and no `--kill`

2. `pnpm -r build` was reported as "no script named -r"

One report was closed as `not planned`, and closing it was right

So I built the piece that enforces it — `genchi`