DEV Community: Dexterlung

Two identical runs scored 89 and 89. Two cases had flipped.

Dexterlung — Fri, 11 Sep 2026 14:05:10 +0000

Read on: the A/B test this corrects · 繁體中文版

Two weeks ago I published an A/B test: I cut 41 AI tools' self-descriptions roughly in half, then ran a behavioural question bank against both versions to check that trigger rate hadn't dropped. Before: 88/96. After: 90/96.

I wrote this sentence about it:

±2 cases at this sample size is noise, so I am not claiming it got better.

A reader named Vinh Nguyen replied, and the reply amounted to one question: how do you know?

He was precise about it. Three votes per case, majority wins, so any case sitting near the 2-1 boundary can flip on sampling alone. There was a between-arm delta with no within-arm noise floor to measure it against. And "three up, one down" — exactly what my four movers looked like — is the shape churn produces.

He was right. That sentence was an assertion, not a measurement. It is also the same move the post spends two thousand words criticising: claiming a property of a system without observing it. I made it in the one paragraph where I was grading my own work.

So I ran his experiment. The part I didn't expect is that it cuts in two directions at once.

First I went looking for the answer I'd already paid for

Before spending anything, I checked whether I could bound the churn retroactively from the run I had already done. The harness makes three calls per case and takes the majority, so it necessarily computes "2 of 3" or "3 of 3" somewhere. Count the 2-of-3 cases and you have an upper bound on how many could flip — for free.

I couldn't. And the reason has a nastier shape than "it wasn't recorded."

The harness does compute the vote split, and it does print it — for cases that failed. A case that passed 2-1 prints nothing at all. The JSON output aggregates to per-skill pass/total and drops every split.

So the one shape that actually flips under sampling — the narrow pass — is precisely the shape no layer of that tool had ever shown me. The instrument was blind in the exact direction that produces reassurance. Failures got scrutiny; lucky wins got a checkmark.

I fixed that first, because it is the cheap half and it makes every future round free to interpret. Per-case vote counts now survive into the JSON, and the human-readable run ends with a count of non-unanimous cases. Then I checked the fix the way I should have checked the original claim: forced every case to 2-of-3 and confirmed the new line moved from "0 split" to "4 split." It reads the real votes. It isn't printing a constant.

The baseline

Same commit, same question bank, same model, same machine, same afternoon. The unchanged arm, twice, with nothing touched in between. 96 cases, 288 calls per run.

	Run 1	Run 2
Total passed	89 / 96	89 / 96
Cases that flipped between the two runs		2

Read those two rows together, because the gap between them is the whole point. The score is identical. Two cases flipped anyway — one up, one down, cancelling exactly.

If I had run the same arm twice and looked only at the total, I would have concluded the harness was perfectly stable. It isn't. It has a floor of about 2, and the total-score view is structurally incapable of showing it, because flips in opposite directions erase each other before they reach the number I was reading.

So my +2 sits exactly on the floor. The sentence in my post happened to land on the right answer for a reason I had not earned.

Where the noise actually lives

This is the part I'd want if I were reading someone else's post.

My question bank has three kinds of case per skill: should-trigger, should-NOT-trigger, and a boundary case. Splitting the two flips by kind:

Case kind	Count	Flipped
should-trigger	48	0
should-NOT-trigger	24	2
boundary	24	0

All of the variance is in the negatives. Zero flips in 48 positive cases, zero in 24 boundary cases, both flips in the 24 negatives.

In hindsight it's obvious. "This sentence is obviously asking for that tool" is a strongly determined decision. "This sentence should not reach for that tool" is a decision made against 116 competitors, and it's soft.

Which means my question bank spends 75% of its budget on the categories carrying none of the noise. If I want a future A/B to be readable, the fix is more negative cases, not more rounds. I would not have guessed that, and no amount of staring at the total would have suggested it.

The bad news, which is also the interesting news

I flagged one skill in the original post as "the one to watch" — the one I had compressed hardest, which lost a case.

It's still down. Both of today's runs put it at 3/4, and the case it loses comes back 0 of 3 — unanimous failure, twice, seventeen days later. And it is a positive case, which is the category where the noise produced zero flips all day.

To be sure that signature meant what I thought, I ran a deliberate control. I took six skills whose descriptions the original disciplined trim could only compress by under 20% — meaning they had almost no redundancy left — and force-cut each by roughly 64%, the same ratio as the one that broke. If over-trimming has a signature, this should produce it.

It did, loudly. Those six went from 23/24 and 22/24 down to 19/24, and one of them collapsed from 4/4 to 1/4. Three separate cases fell from 3/3 to 0/3 or 1/3, every one of them positive or boundary — none in the negatives where the noise lives.

So real damage and churn do not look alike once you can see the votes:

Churn is a negative case wobbling between 1-of-3 and 2-of-3.
Damage is a positive case collapsing to unanimous failure and staying there.

A total score cannot tell those apart. That's not a subtle limitation. In my case the aggregate said "no systematic decline," and it was true, and it was concealing a genuine regression, because one real loss and several noise-driven gains summed to approximately nothing.

The threshold question

The other commenter, Kane Lim, proposed that the 64% case suggested a practical threshold — a point where compression starts removing semantic signal rather than redundant text.

The control answers this, and sharpens it. The same 64% cut cost one case on the skill in the original post, and three cases on a skill in the control. The ratio was identical. The outcome was not.

The difference is what the 64% removed. The skill I could compress that far with discipline could be compressed that far because most of its 689 characters were restatement of a six-layer rule. On a description that is already tight, reaching 64% means cutting into signal — there is nothing else left to take.

Which inverts how I had been reading the number. Compression ratio isn't a risk driver. It's a measurement of how much redundancy was there to begin with — and I had been treating a symptom as a cause. The two variables aren't merely confounded, they're mechanically linked: how far a disciplined trim can go is the redundancy. Across my 24 skills the achievable ratio ran from 3% to 64%, and exactly one exceeded 49%.

And a correction I owe the original post. I wrote: "everything I cut by 30% held steady. The only one that moved was the one I cut by 64%." That's wrong on my own data. The four skills that moved were cut by 64%, 49%, 48%, and 17%. The 17% one moved by exactly as much as the 64% one. The table proving it sat two paragraphs above the sentence.

What I'd take from this

Run the same arm twice before you compare two arms. It cost about a dollar. It reclassified two of my conclusions.
A total score is a lossy summary, and it's lossy in a specific direction: opposite-direction flips cancel, so the aggregate systematically understates instability. Two runs scoring 89 and 89 hid two real flips.
Record the vote split per case, not just pass/fail. Anything that takes a majority is discarding the number that says how close it was. And if your tooling prints the split only for failures — mine did — then the flip-prone narrow pass is the one thing you can never see.
Sort your cases by kind and check where the variance actually sits. Mine was 100% in one category holding 25% of the budget.
A stable unanimous failure is readable at n=1. You don't always need a bigger sample; sometimes you need a less lossy instrument. 0-of-3 twice is a signal. 2-of-3 once is not.

What this doesn't establish

Two runs give a point estimate of the floor, not a distribution — I know the floor is about 2, not that it is 2. The over-cut control was a single arm, single round; its individual collapses are strong, but its total is subject to the same ±2. The six skills I over-cut were selected by a proxy — "the disciplined trim could only compress them under 20%" — which measures how tight a description is, not how narrow a skill's scope is, and scope was Kane's actual question. I also chose how to cut them, keeping the opening trigger sentence and dropping identifiers and symptom phrases; a different cut might land differently.

One last thing, since it's the same failure as everything above. When I went to report what these runs cost, I found the harness doesn't print its cost block in JSON mode — that code sits in the other branch. So the actual spend for the three runs was never recorded, and the roughly US$1.2 I quoted is extrapolated from a single-skill run. Third instance in one day of the same defect: a number that was computed, and then dropped by one output path.

That's the thing worth keeping. Not "measure your noise floor" — that's just good practice. It's that my instrument computed the answer to Vinh's question, printed it selectively in the direction that flattered me, and dropped it everywhere else. Nobody was going to find that from the outside. It took a stranger asking one question I couldn't answer.

本文原載於我的部落格：Two identical runs scored 89 and 89. Two cases had flipped.

Dead Code Is More Dangerous Than Broken Code — How a Non-Homologous AI Caught the Blind Spot the AI Itself Had Rubber-Stamped

Dexterlung — Wed, 09 Sep 2026 13:05:14 +0000

Dead Code Is More Dangerous Than Broken Code

The opening: an AI confidently told me to reuse a capability that didn't exist

Here's how it started.

I was adding a small feature to my own coffee e-commerce store: let a customer pick a fresh combination of beans over chat, price it on the spot, and order it. The chat-ordering bot was already built — I'd even tested balance deduction on a real device. All I needed was one more piece: something on the backend to compute the price, because the amount can't be whatever the phone says it is.

I handed the repo to my main AI (Opus, my daily driver) and asked whether the backend already had pricing logic I could reuse. It scanned the code and came back fast:

"The backend has pricingLogic.ts — the pricing logic lives there. Just reuse it."

Sounds reasonable. The file really does exist. Open it and there really is a pile of price-computing code inside. Anyone reading the code — me, the AI, anyone — lands on the same conclusion at first glance: the backend has pricing capability, just wire it up.

One problem. That code had not compiled a single time in six months.

From the day it was born (git blame says 2026-01-11) it was missing a closing } — the CartItem type literally couldn't close. The frontend test runner doesn't compile the backend's TypeScript, and the audit suite doesn't touch backend functions, so no gate had ever gone near it. It just sat there quietly, in the exact spot where "backend pricing logic" should live, for half a year. At one point another change even passed through it and added a line — the person making the change had no idea they were editing a file that doesn't compile.

The main AI wasn't bluffing. The world it read was real: the file exists, the code inside is real pricing code. It just had no way to tell, from the source tree, that this thing was dead.

My gut said stop: don't build on a foundation that drifts. Nail down the truth about pricing first — and, crucially, bring in a "non-homologous" model to diagnose it.

What that diagnosis dug up was far worse than I expected. It didn't just overturn the main AI's advice; it exposed a structural blind spot in my entire governance setup. That blind spot is what this post is about.

What "existential rot" is

When we say "bad code," we usually picture broken code: it computes wrong, throws errors, blows up mid-run. Broken code has one virtue — it makes noise. It crashes, someone reports it, it gets fixed.

But the four things this diagnosis surfaced don't crash. Not one. They're a different kind of rot, and I've started calling it existential rot: the problem isn't "it does the wrong thing," it's "it exists, does nothing, and lets everyone believe it's doing something."

Four faces of it:

1. Dead code (born broken, never compiled). That pricingLogic.ts. Missing a bracket, six months without compiling, zero production impact. It isn't "broken" — it never lived.

2. A deployed zombie (frozen on an old version). The backend has another pricing function that actually is deployed and live. The live version was v55, a snapshot from March. I sent it a real item with a price of 100 — it returned $0. The source had moved on several versions, but the live one is frozen in March, and wrong. Better yet: it has zero callers, so nobody will ever feel pain from it returning $0.

3. A phantom implementation (the Nth parallel copy). Hidden in the repo was a fourth pricing implementation — a fictional percentage-discount model. Exactly one place imports it: its own test file. All tests green. The green proves "it agrees with itself," not "it agrees with reality."

4. A defense with no reader (a warn-only log nobody reads). I'd added a warn-only amount-recomputation check on the database side that logs suspicious orders to a table. Sounds responsible. But that table had exactly two rows — both from my own smoke test — and nothing in the entire repo, no sentinel, no script, no human, reads that table. The defense is written, but there's no return path.

Put these four side by side and they form the entire truth of "backend pricing capability" in my system: one living body, three zombies.

The only thing actually computing money, start to finish, was one place — cartPriceCalculator.js, running in the customer's browser. The three backend copies were all decorations occupying the spot.

The real root cause: pricing authority actually lives in the browser

Why can "a pricing implementation with no consumer" survive this long?

I read the body of the order-creation stored procedure by hand, and the answer is ugly: it takes the front end's already-computed subtotal, total_amount, and unit_price verbatim. The payment side looks safe — it reads orders.total_amount from the database — but that value is the one the front end stuffed in.

In other words, in this architecture pricing authority de facto lives in the browser (client-authoritative computation). The server just copies down the number the phone computed and files it.

Once that's true, any backend pricing code becomes a structural decoration: wiring it in costs extra effort, and not wiring it in changes nothing. A decoration never gets traffic-tested — no real order passes through it, hits it, translates its error into "the customer paid too little," the kind of thing that hurts. So it can rot without detection and drift without limit.

Which is exactly why my initial gut was right: the "align the two copies" prescription the main AI wrote was a prescription to repair a corpse. Until you answer "who's going to call it, which version deploys, which copies get killed," aligning dead code to the living body just makes the corpse look more alive.

Why my governance saw none of this

Here's the part I actually wanted to write down.

My governance isn't weak. There are locked cross-layer traces, there are commit-blocking hard rules, there's a whole audit suite. The top pricing dark-zone had already been registered, trace-locked (I'll call it trace T-029), given 24 tests, all green, with the word locked sitting right there in the registry — back on 2026-06-06.

And then the darkest part of that dark zone kept rotting, untouched.

I eventually saw why. Almost all of my defenses are change-triggered — they only fire when you touch a registered node. A trace test runs when you edit a file on the trace; a blocker fires when you commit; an audit lights up red when you modify the thing it watches.

But the whole class of "existential rot" doesn't require anyone to touch anything to persist. The dead code lies there, the zombie stays frozen, the phantom chews its own tail, the reader-less log quietly accumulates — none of it trips a single "change detector."

The lamp T-029 lit was the front-end price-resolution convergence point. The "backend copy," the "deployed bundle," the "rule-loading layer" — three surfaces that were simply not in that lamp's beam. It got flagged, locked, 24 tests green — and the darkest part never got touched.

What I was missing wasn't "one more change-triggered rule." It was an existence constraint:

Any implementation that claims to compute money, to defend, or to be a single source of truth must have (a) a live consumer, and (b) a confrontation witness. Missing either one → retire it.

Side note: this also fits a gap I'd already written down — my system only adds features, it never removes them (things only go in, never out). And "has no witness" turns out to be an objective test for "should have been retired but wasn't."

Why it had to be a non-homologous model

Back to the key decision: why did I insist on a different model, instead of asking the same AI to check again?

Because the main AI wasn't short on ability. It was trapped by the same ready-made narrative.

"This file is named pricingLogic.ts, it has pricing code inside, therefore the backend has pricing capability" — that's a story too smooth, too reasonable; it fills your cognition. Whereas "does it even compile? does anyone call it? is that the version that's deployed?" is a backward-looking, declarative question that will not jump up and knock when you're about to reuse a file.

A homologous model colludes. Ask it to look again and it'll most likely re-read "the repo has this file" as "the capability exists" a second time, because it's running the same inference reflex as the first time. That's not stupidity — it's an instrument problem. The source tree is the instrument's testimony, not the fact of the world (the actual runtime behavior).

Switch to a non-homologous model (I used one from a different family) and the effect is like bringing in an outsider who never bought your narrative to review the code. The first thing it does is not trust the repo — it goes and confronts it:

It ran a real compiler over the dead code — instant red. Hard proof: "never compiled."
It sent real input to the live function — got back $0. Hard proof: "deployed zombie computes wrong."
It grepped for callers of those "canonical" functions on the production path — zero. Hard proof: "no consumer."
It read the order-creation procedure body by hand — saw it take the front-end price verbatim. Hard proof: "authority lives in the browser."

It even corrected the main AI's diagnosis name: this isn't two implementations "drifting" (drift = two living things gradually diverging). This is necrosis plus a split truth — a piece of dead code that never lived, a zombie frozen in March, zero callers. Different name → completely different prescription.

The value of non-homologous diagnosis, in one line: it's the vaccine against self-rubber-stamped green lights. When a system (including an AI) blesses itself, the most effective mirror isn't making it look at itself harder — it's introducing a confronter that doesn't collude.

Six green lights, not one of them watching the load-bearing thing

During the diagnosis I did something I think everyone with a test suite should do occasionally: pull out every "green light" and ask each one — what load-bearing thing did you actually, positively observe?

The result (all of this is re-checkable):

Green light	What it claims	What it actually saw	Verdict
Pricing tests all green	"Business rules are test-protected"	A fictional model with zero production callers chewing its own tail	False green
T-029 trace 24 green + `locked`	"Pricing SSOT is locked"	Only the front-end resolvers' fallback — not the compute core, not the backend, not rule-loading	Half true
Edge-function smoke passes	"The pricing function works"	HTTP 200 + `success:true` — returning $0 is also a 200	False green (checks the status code, not the amount)
Warn-only defense shipped	"The DB side has a recompute defense"	Two log rows, both self-inflicted, nobody reads it	Defense exists, no return path
Migration header "not applied this batch"	"Prod wasn't touched by this"	Prod was applied	Doc narrative decoupled from prod
Front end priced successfully	"The user sees the correct price"	On RPC failure it silently falls back and prices anyway	Silent degradation, no alert

Six lights, each making you feel "someone's minding this surface." But not one of them ever positively observed the one thing that's actually load-bearing: two runtimes computing the same number for the same cart.

Here's the line I took away and want to nail to the wall: a green light with no witness proves "nobody's looking," not "it's correct."

The fix: let the zombies die all the way first, then give the living body witnesses

Once the framing was overturned, the fix got clear. The principle was "move logic only, don't change a single price." Three steps:

Step 1: let the zombies die all the way. Both dead copies got deleted and retired. No aligning, no repairing — a zero-consumer corpse doesn't deserve a minute of your time.

Step 2: extract the one living body into a single core. I pulled the one algorithm that actually computes money out of the front end into a pure-function core (zero imports, computes and returns, touches nothing outside). The front end became "load rules → hand to the core." The backend function was rewritten to "resolve rules → hand to the same core." One logic, both runtimes delegating to it.

Step 3 — the real point this time — give it three witnesses. Sharing one import isn't enough; it only kills the "copy-paste" axis of drift. There's a nastier axis: deploy lag. The same source, front end shipping the latest, backend frozen in March — that's two truths. A shared import can't catch that. Only one thing can:

A parity witness: feed the same batch of inputs to the front-end path and the backend path, compare outputs row by row. I prepared six cart "battery" cases (cross-tier VIP, gift-box volume, tiered discount, buy-X-get-Y, etc.), froze the pre-refactor live output as a golden baseline, and any change that moves a single dollar goes red.
A live deploy-lag witness: hit the actually deployed version over HTTP and compare it against the local core, row by row. This is the only edge that can see "frozen on an old version." First run, it immediately spat out 3-of-3 mismatches against live v55 — proof the detector is alive and working.
A compile gate covering all runtimes: a blocker requiring that the pricing-chain backend function actually compiles. This is the direct counterfactual to "dead code nobody knew about for six months" — had this gate existed, that missing-bracket corpse would've gone red on day one.

One thing I deliberately did not do: I didn't heavily rewrite the order-creation procedure body. Because in this repo, "rewrite a gatekeeper function wholesale" has a rap sheet — it wipes out the gatekeeping logic buried inside. (Two prior migrations did exactly that, once wiping a gift-box capacity guard and once a stored-value balance guard, both dead for weeks.) Server-side authority gets promoted later, gradually, through a hook I'd already pre-embedded — without reopening that knife.

Turning one expensive insight into a free mechanism forever

This kind of diagnosis is expensive. A non-homologous model, copy-by-copy confrontation, climbing counterfactuals — that burns real tokens. So I have a habit: every piece of reasoning I've paid for once, I try to make cheap or free the next time.

Two reusable things crystallized out this time, sitting at two different moments:

1. A lens for the "about to build" moment — the consumer-witness lens. A check I built and use daily. Its core is a gate: before you trust, cite, or build on anything that "claims to compute money / to defend / to be the single truth," it forces you to write two lines — (a) which specific caller in production depends on it? (b) which specific test feeds it the same inputs and confronts it against the authority? If you can't write either line, treat it as nonexistent. This lens is the vaccine against that exact "fooled into the wrong framing by a corpse" moment, grown into a gate that knocks before every future reuse.

2. A checklist item for the "designing" moment — the server-authoritative rule. I maintain a cross-project front-end/back-end completeness checklist (past incidents distilled into check-sentences). This time I added one, with a machine-readable tag so tools can auto-apply it:

Load-bearing computed values (price, total, balance, stock — the "computed" kind) must be computed by the server, with the client only displaying; the server must never trust a client-computed load-bearing value verbatim.

This is a generalization of an old rule I already had ("no front-end multiplication for the unit-price field") — the old rule covers the pricing special case; this one covers "for every computed load-bearing value, who is allowed to compute it, and whom does the downstream trust." The machine-readable tag means: next time I decompose a new project through my tool-chain, this rule gets spat out into the spec automatically, without me remembering it. Pay once, carry it into every new project free.

But "spat out automatically" hides a trap I only fixed this time — and it's a vivid one, worth its own beat.

I'd crystallized that check-sentence library into a machine-readable config file — plainly, an index of "when I hit this sharp edge, here's how I handle it." It hangs on the tool-chain I use to decompose new projects: start a new project and the tool reads it and auto-applies my past scars into the spec. Problem: that config file is private (it lives in a toolkit repo of mine that isn't public), while my project bootstrapper pulls almost everything else down from public GitHub.

So you get a very classic tension: a bootstrapper that fetches everything from a public channel, meeting one file that can't travel a public channel. Worse — historically that config file had never been supplied automatically. It sat in a "manual step nobody actually ran." The result: every decomposition silently ate the neutral defaults, and this config I'd painstakingly crystallized contributed exactly zero.

That is literally the disease this whole post is about, growing back in a new spot: a thing that "should be supplied but isn't" quietly trips no alarm, letting every decomposition believe it applied the completeness rules when it didn't.

So I wired the supply in, carefully, every line guarding against "private content leaking into a public repo":

Never fetch it. Unlike the other tools, this config isn't downloaded from GitHub — it's copied at runtime, on my own machine, from a source already present locally (scan a few common folders for that private toolkit, or read a path from an env var). Zero hardcoded private paths in the code. A regular user has none of that private source, so it resolves to "not found," falls back to neutral defaults — nothing leaks.
Fence it with .gitignore. On supply it auto-adds that private config's folder to the project's .gitignore, with an iron-rule comment baked in — "never reaches a public repo." So even if you later fat-finger git add ., it can't reach any outward remote.
Fail loud on supply, never silent. If no source is found it says so out loud — "config not supplied, this run uses neutral defaults" — because a silent skip is exactly the bug this fixes.

I love this bit because it compresses the whole lesson into one concrete action: a bootstrapper that fetches everything from the public internet meets a private file that can't travel that public channel — so it switches to "smuggle it in locally + fence it with .gitignore so it can never leak back to a public repo." That "never reaches a public repo" iron-rule comment is what an existence constraint looks like once it's grown into code: "this thing should exist, and only here" written as a rule the program executes, not a line of doc nobody runs.

For the record, the referenceable deliverables from this round (things I use daily, fed by real incidents — not slideware): a check I call the consumer-witness lens (before reusing anything, force yourself to write down its two witnesses), a cross-project front-end/back-end completeness checklist (every incident distilled into a check-sentence), and an orphan audit script (hunts "zero-caller decorative anchors" — it found thirty-nine in one project alone).

Closing: this was never really about pricing

If you filed this under "a story about one pricing bug," you missed the point.

The two drifting pricing copies were just the most visible projection of the hole. The real hole is that the copy axis has no forcing function: the same "how money is computed / how state is judged" logic actually lives in five physical spots — front-end bundle, back-end bundle, the live DB function body, migration files, repo source — and nothing guarantees they equal each other. They will, and already do, tell different stories.

So what's actually portable is this:

Dead code is more dangerous than broken code. Broken code blows up and gets fixed; dead code lies quietly in the spot where "some capability should live," fooling every reader of the code — including the AI itself.
A defense with no reader is not a defense. A warn-only log, an alert nobody subscribes to, a dashboard nobody looks at — all worth zero.
A single SSOT module is not a single truth. A shared import only kills "copy-paste" drift; "deploy-lag" drift is visible only to a confrontation witness.
"The repo has two copies of this code" should not first read as "two implementations." First ask: does it compile? does anyone call it? which version is deployed? — very likely it's "one living body plus N zombies," not drift at all.
If your governance is all change-triggered, it's blind to existential rot. Add one existence constraint: every money-computing / defending implementation needs a live consumer plus a confrontation witness, or it gets retired.

That last decision — "don't build on a foundation that drifts; fix the truth first, and bring in a different model to look" — took under a minute. It was the highest-return step in the whole thing. Because an AI reading your code, however confident, is only giving you instrument testimony. Whether to build on it, you have to ask first: this capability — is it actually alive?

This is one entry in my one-person studio's dev-notes series. The consumer-witness lens, the completeness checklist, and the orphan audit are tools I built and use daily; some are already open-sourced. They aren't theory — every one has a pit like this one behind it, a pit I actually fell into.

本文原載於我的部落格：Dead Code Is More Dangerous Than Broken Code — How a Non-Homologous AI Caught the Blind Spot the AI Itself Had Rubber-Stamped

AI Auditing an AI's Own Toolchain: The Repair Tool That Overwrote Its Own Fix

Dexterlung — Tue, 08 Sep 2026 14:05:34 +0000

I have seven interdependent tool repos — some public, some private — each with its own audit checks, but nobody had ever checked what happens when they're wired together. This post is about handing all seven to Fable5 for a cross-repo audit, and the fact that not one of the real problems it found could have been caught by auditing any single repo alone.

Read on: Previous: the Confrontation-Point Law · Don't ask a model to narrate itself · 繁體中文版

The audit prompt itself went through three drafts

I didn't just paste in "please check these seven repos." The prompt started as a skeleton, then got execution steps, then got filled in with the real seven repos — 5 public tool-chain repos plus 2 private ones (an internal connective framework and a cross-project reuse library). The goal was to write context and boundaries, not a checklist — tell Fable what the terrain is and where the hard lines are, and make verification about mechanism, not about handing over an answer.

Each repo got its own scanning subagent, plus one independent verifier subagent that never saw the scanners' conclusions. That independent verifier is the part that mattered — it overturned four of the scanning agents' conclusions. The topology itself turned out fine; the two public/private seams both worked correctly. The real problems weren't in any single repo. They were three composition-level failures that only become visible once you look at all seven together.

The tool that fixes drift can silently overwrite your work

The most interesting one: one tool's job is to detect and repair drift between a local skill copy and its canonical source — if your local copy diverges from the original, it syncs you back up. The problem: in one environment, that "local copy" is actually a Windows junction (a filesystem shortcut) pointing straight at the canonical repo — and the tool's scan logic only checked "is this a directory," never "is this actually a shortcut."

The result: a tool built to repair drift will write straight through the shortcut into the canonical repo, potentially overwriting uncommitted work sitting on someone's feature branch. The repair tool itself becomes the risk. No single-repo audit could have caught this — the bug doesn't live in any code, it lives in how directories are wired together on a specific machine. That shape is only visible across repos.

The other two composite failures: an orphaned commit left a cross-repo contract stranded with nothing consuming it; and three places documented something as "prompt-level, optional" that was actually being treated as "code-level, mandatory for everyone" — a documentation/enforcement mismatch that leaves different maintainers with completely different mental models of the same rule.

One rule worth stealing: content-level probes, not existence checks

The audit converged on a principle I think is more valuable than any single finding: "the commit exists" or "the file exists" doesn't count as verified. You have to walk the entire propagation chain — local branch, remote default branch, the published artifact or install source, the installed copy — and grep an actual signature string at every single stage, and it has to actually hit, every time.

That principle is what caught the most important issues in this audit. Downgrade it to an existence check, and the audit still runs, still produces a report, and catches nothing — which is worse than not running it at all, because it hands you a false green light.

A real case: how a spec makes frontend and backend asymmetric from birth

The day after the cross-repo checklist converged, a real client project handed us a case that got folded directly into that checklist as a new criterion.

The setup: a real order-management admin feature with a visible frontend/backend asymmetry. The backend RPC defined an error code called BOOKING_NOT_FOUND that the frontend's error-message map never learned about. A refund action returned HTTP 501 in production with zero UI indication. A loading ref was declared and returned but never consumed by any of six admin tabs. An admin-readable transaction table existed and was never queried by the frontend at all.

We gathered the hard evidence first, then asked for the root cause, not just a symptom list. The diagnosis: the asymmetry wasn't born in any one layer — it was born upstream, in the spec's own output grammar. The original spec gave the backend a model (field-level data structures) and gave the frontend only a verb ("view orders and bookings"). Every downstream layer — decomposition, execution, scoring — is deliberately faithful by design, each explicitly instructed not to invent or over-check. So the upstream asymmetry got preserved intact, all the way to the final output. No layer dropped the ball. The pipeline worked exactly as designed — the design itself just had a blind seam nobody had seen.

Even the fix this diagnosis proposed doubted itself out loud: the suggestion was to add a mandatory "Surface Contract" section (a state row plus a seam row) to the five-element goal format. But the same document flagged its own biggest risk: "once a template becomes mandatory, the most likely failure is that people fill it with boilerplate that formally passes the check while saying nothing real — that's the step I'm most likely to have gotten wrong." A model willing to say, in writing, "my own suggestion might just get rubber-stamped" is more useful to me than the suggestion itself.

That proposal is, as of this writing, only designed — not yet wired into the actual goal template. Worth saying plainly: designed is not the same as done.

Takeaway

A single-repo audit can't see the shape that only exists once tools are wired together — and the most expensive lesson here is that the tool meant to repair drift can be the single riskiest thing in the whole system. "Content-level probes, not existence checks" is a rule worth copying into any multi-repo system you maintain, whether or not you have a Fable5 to run it for you.

本文原載於我的部落格：AI Auditing an AI's Own Toolchain: The Repair Tool That Overwrote Its Own Fix

How a Security Bug Gave Birth to a Method, Then the Method Caught What the Audit Missed

Dexterlung — Tue, 08 Sep 2026 14:05:22 +0000

At 5:27pm on 2026-07-03, I revoked self-write access on 8 sensitive columns of our members table. At 9:42pm the same day, I reverted it — an admin save feature started throwing 403 errors. Locked, then unlocked, same day.

That incident became the seed for a method we call the Confrontation-Point Law. I won't claim it's proven — there's exactly one real case so far, and we've deliberately labeled it "armed, not battle-tested." But that one case is one of my favorite technical stories to date, because what it caught was something a careful human audit had already missed.

Read on: Previous: don't ask a model to narrate itself · Next: AI auditing an AI's own toolchain · 繁體中文版

That lockdown commit looked careful

The 07-03 lockdown commit's header read: "verified all 5 write paths to the users table site-wide, MembersCenter.vue goes through fn_admin_* RPCs." That's not a throwaway line — someone actually enumerated the paths and labeled each as a legitimate channel. On paper, this looked like a solid audit.

But inside that same file, the function saveEditMember calls an RPC to write member_level at line 2010 — and ten lines later, at line 2020, the same function does a raw .from('users').update({ tier_updated_at, tier_expires_at, tier_auto_downgrade }). Direct write, no gate.

The same function was, at the file level, a proper RPC channel — and at the field level, simultaneously an unguarded direct-write channel. That gap is exactly what triggered the 403s that night, and exactly why we reverted four hours later.

Why a careful audit still missed it

Looking back, what bothers me isn't that there was a bug. It's why a careful audit still missed it. The answer is blunt: the auditor read the code, saw the RPC call, saw a comment that said "goes through RPC," and issued a file-level verdict. The census got outsourced to narrative. The fire wasn't because the set of write points was too large to enumerate — we later verified the whole frontend only touches that field in 3 files, something a dumb grep finds in seconds. The fire was because the verdict substituted "I read it, looks fine" for "I mechanically checked every single line."

That finding is what produced the first rule of the Confrontation-Point Law: every frozen judgment needs to name its confrontation point — a place where intent and actual state are forced to meet, where any disagreement blocks progress, sitting right on the boundary where real harm happens. Who's allowed to sit in the "this is what's actually true" seat? Only a machine — a real query like has_column_privilege, never an agent reading code and saying "I checked, looks fine." Narrative sitting in that seat is a fake census. That "goes through RPC" comment is a live example of narrative sitting in the wrong seat.

The same week, a mechanical re-audit caught 3 leaks the human review missed

On 07-04, instead of trusting "I read it, looks fine" again, we ran a fully mechanical re-census: every claimed-legitimate write point had to be re-derived through grep/AST plus an actual privilege query. Anything that couldn't be re-derived got flagged red on the spot — no exceptions.

This caught 3 admin direct-write points the earlier review had completely missed: renewTier, dismissBinding, importMembers. All 8 sensitive columns were re-verdicted from scratch — all 8 came back safe to revoke — and the three admin write paths got migrated into three new SECURITY DEFINER RPCs. This time the rollout order was deliberately sequenced: ship the RPCs first, wait for the frontend deploy to land, then run the revoke — specifically to avoid repeating the 07-03 disaster of locking permissions before the frontend had caught up.

After going live, we actually verified it in production, not just in code review: a regular member trying to self-write any of the 8 columns gets a 403; all 5 admin paths still work.

I'm not going to oversell this

Honestly: this whole method has exactly one real case behind it so far. We deliberately don't call it "battle-tested" — only "armed": self-tests pass, an injected regression test passes, but it hasn't yet stopped a real attack we didn't already know about. That line is written into the method itself: armed is not the same evidence tier as battle-tested.

There's a funny footnote to how this method got written, too: one of the prompts we fed to Fable had a placeholder bug that silently ate the draft content. Fable didn't hallucinate a plausible-sounding answer to cover for it — it just refused to answer. We retroactively counted that refusal as a live demonstration of the exact principle we were designing: refusing to answer is itself a legitimate output.

One more loose thread, left here honestly rather than swept under the rug: the same review found Addresses.vue writing to a column called addresses that doesn't actually exist in the database. That's deliberately out of scope for this fix and still unresolved.

Takeaway

A careful-looking audit can still lie at the field level, if its "verified" claim comes from a human reading code and forming a narrative judgment. A census you can actually trust is one where every single row can be mechanically re-derived — anything that can't gets flagged, no matter how confident the write-up sounds. And keep "armed" and "battle-tested" as two different evidence tiers — blurring them is just lying to yourself with extra steps.

本文原載於我的部落格：How a Security Bug Gave Birth to a Method, Then the Method Caught What the Audit Missed

Don't Ask a Model to Narrate Itself: How I Actually Extracted a Method From Fable5

Dexterlung — Thu, 03 Sep 2026 13:05:17 +0000

The first time I wanted to "learn how Fable5 thinks," I did the obvious thing: I asked it directly. "How do you approach a problem?" I copied down the answer, then kept asking, until it grew into an eleven-book set — a how-to guide, a book on perspective training, a book on code-review instincts, all the way up to a "main volume." It looked thorough. It took real time to assemble.

Then the whole set got overturned by its own method. Not because the content was wrong — because the approach itself was self-contradictory.

Read on: spec-sonar design journal, Part 1 · Next: the Confrontation-Point Law · 繁體中文版

The problem: I used an untrustworthy method to study a doctrine about not trusting self-reports

It was Opus that pointed out the embarrassing part: Fable's own core doctrine says don't trust self-reports — look at what actually got done, not what got said about it. And the method I'd used to write those eleven books was exactly that: ask "how do you think," then treat the answer as ground truth.

I used the discredited method to document the very doctrine that discredits it. Once that was named, only four principles out of the whole set survived. Everything else needed a rebuild. I wasn't angry so much as deflated — that wasn't ten minutes of work, it was a full conversation's output, now judged to have a broken foundation.

The fix: stop asking what it thinks, start reverse-engineering what it actually did

The new method was simple: instead of asking "how do you think," we took two things Fable5 had actually produced — the router-reasoning.md and decision-table.md from an n8n cold-start generator — and reverse-engineered the decision rules baked into them. Trust the scar, not the self-report — applied to the method itself, this time.

Then we ran a two-pass adversarial interview, four rounds. Pass one: no draft shown, free-form answers. Pass two: show it the draft, let it attack. The point of this structure is to surface two different kinds of error — pass one catches what we didn't think of; pass two catches what we got wrong.

Round one already surprised us. We'd planned to frame the method as something like "five joints." Fable rewrote the whole naming itself: forge, runway, customs — forge is where you're still building, runway is where you run an already-proven pattern, customs is the border crossing where the two are forced to meet and get checked. That wasn't a wording polish. It was a correction to how we'd classified our own method.

Round two: it caught an error happening live, inside our own document

Round two got more interesting. Opus's compressed one-line summary read: "reduce cold-start to lookup then emit a goal graph." Sounds fine. Fable pointed out that compression had quietly deleted two non-happy-path exits — "ask" and "stop" — which happen to be where every safety mechanism in the method actually lives. We checked the real document. The omission was real.

Something more embarrassing happened in the same round: Opus had written, in its own appendix, "we need to update this playbook" — and never did. Fable caught that live drift and used it as proof of its own thesis on the spot: canonical and working copies drift, not as theory, but as something happening in front of us.

Round four: an exam that could actually fail

Round four was a holdout test — a problem it had never seen (e-invoice integration), to see whether it would apply the same method cold. It didn't improvise an answer. It first grepped the real repo, found an old "manual invoice era" precedent, and answered from that. We checked afterward, line by line: the file and line number it cited matched the real file, verbatim.

That was the first time I'd actually verified a model's reasoning instead of just feeling like it guessed right — and a verified-correct answer is a categorically different thing from a lucky-and-unchecked one.

The last move: it turned its own rule against itself

The whole extraction closed on one line from Fable:

"You keep saying evidence. Do you have any evidence for yourself?"

The method's own second core tenet is the evidence rule — every frozen claim needs an anchor you can check. Fable pointed out that the entire playbook had, up to that point, zero evidence for its own validity. We thought the interview was over. Instead we got judged by the very method we'd built.

So we ran a "firing-record audit" — checking every claim in the method against the real git history of the CS project. The result wasn't a flat "no evidence." It split into three tiers: battle-tested (actually stopped a real problem), armed but not fired (the mechanism exists but hasn't faced a real test yet), and structurally unfireable (there's no way this claim could ever be checked). That three-tier grading is now how we evaluate any governance mechanism we build — not a vague "does it work," but an honest label of what evidence tier it's actually at.

What it cost me

Honestly: I never deleted those eleven overturned books, but I stopped using them. That was a full conversation's worth of time, spent to arrive at "the approach itself was wrong." If there's one real scar in this piece, it's that — I picked the wrong extraction method first, and it took a whole round to find out.

What crystallized

The four rounds converged into two reusable skills: connective-system-design (forge/runway/customs, for designing cross-system connections) and root-cause-diagnosis (counterfactual climbing, for finding a systemic root cause). The seal Fable added itself: "Trust the scar, not me." — including don't fully trust this article either. Go check the anchors.

The method was immediately put to two real tests: a cross-repo ecosystem audit across seven repos, and a real client project's frontend/backend asymmetry diagnosis. Both are the next two posts in this series.

Takeaway: if you want to actually learn a method from a model, don't ask "how do you think" — look at what it actually produced, reverse-engineer the rule, test it on something new, and verify before you believe it. A self-report isn't free truth. It's a claim you still have to go check.

本文原載於我的部落格：Don't Ask a Model to Narrate Itself: How I Actually Extracted a Method From Fable5

I turned both knobs on my on-prem Chinese RAG all the way up. It didn't get a single extra question right

Dexterlung — Tue, 01 Sep 2026 13:05:15 +0000

Something you can do right now

If you're about to "increase the retrieval depth" or "swap in a newer reranker" to improve your Chinese RAG accuracy, spend ten seconds on this first:

Take one question you got wrong, and compute where the passage containing the correct answer ranks in your raw vector similarity.

No reranker. No full pipeline. One embedding call and one round of cosine similarity.

That number decides everything downstream:

If it ranks outside your candidate depth → going deeper might help (only might — see below)
If it ranks inside your candidate depth → going deeper definitely won't, and your bottleneck is somewhere else

Without that probe I'd have swept top-k = 10 / 20 / 30 / 50 / 80 / 100, six configurations, 29 questions each, tens of seconds per question — about an hour. After the ten-second probe, those six configurations collapsed to one.

Where this started: a to-do I wrote for myself

I was evaluating a "the data never leaves the building" knowledge base for a client, and wrote up my research as a handoff doc for the next session. The to-do was specific:

Step 1: top-k 10 → 50 (zero cost, zero VRAM) — a public benchmark on 3,493 traditional-Chinese questions shows a weak embedding plus a reranker reaching 98%+ at k=100, beating every single-stage retriever.
⚠️ Don't just crank it to the max (a SIGIR 2025 paper reports that over-reranking degrades results); there's a sweet spot and you have to sweep for it.

Step 2: bge-reranker-base → bge-reranker-v2-m3 (+1.3GB, same architecture, same API, no prompt changes)

It reads as completely reasonable. External benchmarks behind it, a counter-example warning, a clear execution order.

Both steps ended up as "don't." And that "zero cost" in step 1 is wrong.

First cut: the cheap adjacent question

Before touching anything I looked at the table from my previous round:

Configuration	K=1	K=3	K=5
Vector only	84%	88%	92%
Vector + rerank (candidate depth 30)	88%	96%	96%
Vector + rerank (candidate depth 10)	84%	92%	92%

96% is 24 out of 25. And the miss is always the same question.

That sentence matters, because it caps the question "how much can deeper candidates buy me?" — at most one question, i.e. 4 percentage points.

So the question changed from "which k is best?" into something far cheaper: did the reranker ever even see the correct answer for that one question?

I wrote a ten-second probe: for each of the 25 questions, where does the passage containing the correct answer rank by raw vector similarity?

Rank of the first correct passage	Questions
1st	22
3rd / 5th / 21st	1 each
83rd	1 (the usual suspect)

One table cut six configurations down to one:

Depth 10 → correct answer present in candidates for 23/25 (ceiling 92%)
Depth 30 → 24/25 (ceiling 96%)
Depth 50 and 80 → still 24/25, because that question sits at 83. Identical ceiling to 30 ⇒ no need to run them
Depth 100 → 25/25 (ceiling 100%)

And check the calibration: depth 10's ceiling is 92%, measured 92%. Depth 30's ceiling is 96%, measured 96%. Both match to the decimal.

At that moment I derived a conclusion: "whatever's in the candidate set, the reranker pulls into the top 5 ⇒ this pipeline's bottleneck is how deep we fish, not how good the reranker is."

The derivation itself is sound (the miss set must contain the out-of-candidates set, and the two have equal size, so they're the same set).

Then I ran depth 100, and it falsified that conclusion.

Depth 100: ceiling 100%, measured 96%

	Depth 30	Depth 100
Retrieval hit rate (K=5)	96% (24/25)	96% (24/25)
Miss	that question	that question
Median latency per question	6,834 ms	37,651 ms

The correct answer ranks 83rd, so it did make it into the 100 candidates. The reranker saw it. And then ranked it outside the top 5.

So the right statement isn't the one I derived — it's a more useful one:

Candidate depth is necessary, not sufficient.
For that question to land, two things must both hold: ① you fish deep enough to see it, and ② the reranker is willing to rank it near the top.
Depth 30 fails at ①. Depth 100 clears ① and fails at ②.

Incidentally: "zero cost" is wrong. A reranker is a cross-encoder, and its cost is linear in the candidate count — send it 100 candidates and it computes "how relevant is this passage to this question" 100 times. Going from depth 30 to 100 took each question from 6.8 to 37.7 seconds: 5.5× the latency, for zero extra correct answers.

"Zero VRAM" is right. "Zero cost" is not. And what it spends is precisely the one budget in this setup that had already bottomed out.

Second cut: swap in a "newer" reranker

If the bottleneck is the reranker, replace it. My to-do said bge-reranker-v2-m3.

First discovery: that model isn't available on my stack. I run ONNX through fastembed on CPU, and its cross-encoder list has six models — no v2-m3. Only two are multilingual: the bge-reranker-base I'm already on (a 2023 base-tier model), and jinaai/jina-reranker-v2-base-multilingual (1.11 GB, 2024 generation, marketed on multilingual).

Actually using v2-m3 would mean swapping the whole service from ONNX to PyTorch. That's a separate project. So I swapped in the jina model — same API, one parameter changed, no prompt edits.

It lost in three directions at once:

reranker	K=1	K=3	K=5	Pure Chinese prose questions, K=1	Latency/question
`bge-reranker-base` (2023)	88%	96%	96%	88.9% (8/9)	6,834 ms
`jina-reranker-v2-multilingual` (2024)	76%	96%	96%	66.7% (6/9)	8,219 ms

K=1 dropped 12 points (3 questions). K=1 is the column that most affects the quote — it decides how many documents the client has to feed the LLM, which decides whether they need to buy that GPU.
The worst drop is on pure Chinese prose (88.9% → 66.7%). A multilingual-first 2024 model losing to a 2023 base-tier model on traditional Chinese prose.
It's also 1.2× slower.

But the fourth reason is the fatal one, and I nearly missed it.

The fourth reason I nearly missed

My previous round had turned up a bonus finding: the reranker's score itself works as a "refuse to answer without calling the LLM" gate. Below a threshold, return "not found" — skip generation entirely. Fast, and structurally incapable of making things up.

For this client that matters more than usual, because making things up is exactly what they're afraid of.

The scoring script prints a "median top-1 score" line each round:

bge-reranker-base: answerable 5.26 / unanswerable −2.14
jina: answerable 0.84 / unanswerable −0.76

That looks like "the gap got smaller." What I wrote in the report at the time was "this doesn't let us say jina made the gate worse [unknown]."

But the statistic that finding needs isn't the median. It needs the minimum score among answerable questions against the maximum among unanswerable ones — because two distributions can have very different medians and still have overlapping tails.

The per-question scores were already in the scoring run's JSON. I computed it afterwards at zero extra cost:

reranker	Answerable min	Unanswerable max	Gap	Can one threshold separate them?
`bge-reranker-base`	−0.5237	−1.8008	+1.2771	✅ yes
`jina-v2-multilingual`	−0.9005	−0.6736	−0.2269	❌ overlap
Vector only (no rerank)	0.5895	0.6377	−0.0482	❌ overlap

Switching to jina means: K=1 down 12 points, latency up 1.2×, and then you also forfeit the entire refusal defence.

And the median made it look like it merely "got smaller." That's the most valuable lesson of this round: the statistic a report prints is not necessarily the statistic your conclusion needs. I turned it into a command (reads the stored scoring JSON, zero cost) and left a test inside it: fixtures of answerable [9, 9, 9, −1] and unanswerable [0] — medians differ by 9, looks trivially separable; the truth is min −1 < max 0, overlap. That test exists so that if anyone ever switches it back to medians, it goes red.

What's actually wrong with that one question: I tried a third road too, and it's also closed

I had one hypothesis left. Looking at the top-5 for that question across four configurations, I noticed all five slots in every configuration were filled by the same class of file — my own blog drafts. Derivative articles discussing the same incidents, not the canonical records of those incidents.

That matches a published finding: what hurts RAG isn't "irrelevant," it's "relevant but wrong." It also matches my own earlier observation that this corpus writes about the same event a dozen times over.

So I added an exclusion rule, filtered the entire blog-source directory out of the candidates, and re-ran that question.

Still missed. And the correct answer didn't even make the filtered 30 candidates — excluding the derivative material just pushed other non-answers up.

Three roads walked:

Hypothesis	Measured	Conclusion
Not deep enough	depth 100 got it into candidates, reranker still didn't rank it	❌ necessary, not sufficient
Reranker too old	swapped to a 2024 multilingual model; still missed, every other metric regressed	❌ actively worse
Corpus noise drowning the canon	excluded the biggest source of derivatives; it can't even reach candidates	❌ hypothesis falsified
Embedding not good enough	untested	⬅️ only one left

That question asks "what has to be done to a temp table containing personal data for it to count as safe, and which incident forced that?" — and the file with the answer is anchored on a database table name. The question doesn't contain that table name; only the answer does. It's a pure semantic-bridging task, and my embedding model puts it 83rd.

This round proved the embedding is the bottleneck, by elimination — every knob outside the embedding was turned to the stop with no effect. My original plan wrote the precondition for "swap the embedding" as "only swap once you've proven embedding is the bottleneck." That precondition now holds; it just got proven by exclusion.

My next cut will be query rewriting (have the LLM expand the question into a hypothetical answer first, then embed that) rather than swapping the embedding outright — much cheaper, and among the open-source projects I've read, the one that handles Chinese most seriously does its work on the query construction side (part-of-speech weighting, positional proximity weighting), not by changing models.

A red flag that goes straight into the quote

This round also turned up something I wasn't looking for.

To get a same-day control, I re-ran the depth-30 configuration. Recall was identical (96%, same question missed) — comparability confirmed.

But latency was 6,834 ms per question. A week earlier I'd measured the same configuration at 18,152 ms.

Same model, same corpus, same question bank, same laptop. A factor of 2.66.

Two consequences:

I'd been comparing a fresh depth-100 number (37,651 ms) against last week's 18,152 ms and computing "2.07×." Wrong. The correct same-day ratio is 5.51×. A latency denominator has to be produced in the same round as its numerator — the same discipline as "pin the corpus to a commit," which I'd applied to the corpus and not to latency.
Worse: the conclusion I'd written a week earlier was "18 seconds per question — unusable for interactive Q&A." That sentence cannot be measured today (6.8 seconds). 6.8 seconds is still slow, but "slow" and "unusable" are completely different things in a client quote — and that sentence was headed into the proposal.

I don't know the source of the variance yet (thermal throttling? something else running? a package version?). All I can state is that corpus, question bank, and model are ruled out, because all three were re-verified identical the same day.

So the rule now is: any latency number going into a quote must have its own control, measured in the same round, on the same machine, with no other load.

Net result

Four scoring runs, three probes, roughly an hour of machine time. What it bought:

Don't touch either knob. Stay on bge-reranker-base with candidate depth 30.

That sounds like nothing happened. In practice I got three things:

A number I can commit to: on a 25-question benchmark, 96% retrieval at K=5, 88% at K=1, plus a refusal gate that doesn't depend on the LLM.
A number I can't commit to: 100% retrieval. Measured as unbuyable — deeper candidates don't buy it, and neither does the newest reranker I can actually run. That's far more valuable than "I think we can probably get there," because it lets me write a ceiling into the proposal.
Two numbers not to quote: any local CPU latency (2.66× day-to-day variance), and any presumption that "a newer model will be better."

Plus one observation I can sell: derivative copies inside a client's corpus (drafts, meeting notes, several versions of the same policy) will systematically outrank the canon — my own blog drafts pushing my own canonical records out of the top 5 is the live example. But don't sell "curate the corpus" as a cure-all, because in the same round I measured it doing nothing for that question.

Four things you can take away

Before turning any expensive knob, find a ceiling you can compute in ten seconds. "At most how many questions can this buy?" routinely makes an entire sweep plan disappear.
"Newer, bigger, multilingual-first" does not mean "better on your corpus." Marketing labels aren't evidence. I put a 2024-generation multilingual reranker in and it lost to a 2023 base-tier model on traditional Chinese prose.
The statistic a report prints isn't necessarily the statistic your conclusion needs. A median can't answer "are these separable?" Ask which statistic your claim rests on, then go compute that one.
Don't quote a latency number without a control. The same configuration varies 2.66× day to day on my machine, and it nearly put the word "unusable" into a client proposal.

If you're doing this kind of measure-before-you-change work, I've written others in the same family: my test report printed "0/96, 0% pass rate" — the truth was my account was out of credit, on why a red light needs as much suspicion as a green one; and the A/B test where I cut 41 tool descriptions in half, which is the same discipline applied to a change I was confident about — an absolute number can't tell you whether something dropped, so you have to go build the control arm.

本文原載於我的部落格：I turned both knobs on my on-prem Chinese RAG all the way up. It didn't get a single extra question right

My test report printed "0/96, 0% pass rate". The truth was my account was out of credit

Dexterlung — Sun, 30 Aug 2026 13:05:13 +0000

Something you can do right now

Open any script you have that prints a score — an eval, a CI check, an audit tool, a health check — and ask it one question:

If the infrastructure fails (out of credit, expired key, rate limited, no network), what does this script print?

If the answer is a score, a ratio, or a pass count, you own a false-red-light generator.

Not "might go wrong." Structurally guaranteed — because it put "couldn't measure" and "didn't pass" in the same box.

It took me a full day to see this, and I had to see it twice, on two different scripts.

Where it started: I'd just finished something I was confident about

I develop with Claude Code and had accumulated 89 custom skills (think "specialised tools for the AI" — each has a name and a self-description). Those descriptions get packed into the model's opening memory, and that memory has a character limit. Go over it and descriptions get dropped, names kept — and that tool never surfaces on its own again.

I squeezed 24 of those descriptions shorter, saving 2,754 characters, retiring nothing. (The full write-up of that, with the A/B test: I cut 41 AI tools' self-descriptions in half.)

The scary part of trimming is cutting a trigger word: that description is the basis on which the model decides whether to invoke the skill, so cutting the wrong phrase makes it silently stop firing, with no error. So I wrote a reconciliation script that lists every token present in the old version and absent in the new, forcing me to judge them one by one. Eight had genuinely lost trigger words. All restored.

Then I wanted harder evidence: behavioral tests. I already had a question bank — 4 cases per skill (should-trigger, strict, should-not-trigger, boundary), 96 total.

I had believed that bank didn't exist. I'd even written in a handoff doc: "most of these 24 have no test cases, so 'will it still trigger?' can only be answered by token reconciliation."

That sentence was wrong. All 24 had cases.

What caught the wrong sentence wasn't me re-reading it. It was running a full audit before pushing, and noticing that one check had gone from "pass" to "blind", and had taken 100 seconds. It had been triggered into actually running by my 24 changed skills. I chased that state change, and found the question bank had been there all along.

A state change in an audit item is itself a signal — often more informative than its green light.
pass→blind, fast→slow, warning count 9→10 — every one of those deserves a "why did that change?"
If I hadn't chased it, I'd have kept a false sentence ("there's no bank to test against") while the bank sat right there.

Hole #1: an empty wallet, rendered as a 0% pass rate

So I ran the 96 tests.

┌─────────────────────────────────────────────────────────┐
│              Skill Eval Report (advisory)               │
├─────────────────────────────────────────────┬───────────┤
│ ❌ action-gating-surface-disclosure            │  0/4 (0%) │
│ ❌ architecture-completeness-guardian          │  0/4 (0%) │
│ ❌ audit-cross-repo                            │  0/4 (0%) │
                        ⋮  (all 24 like this)
├─────────────────────────────────────────────┼───────────┤
│   TOTAL                                       │ 0/96 (0%) │
└─────────────────────────────────────────────┴───────────┘

⚠️  The following skills scored < 50% — consider reviewing:
   (all 24 listed)

If I only read that table, the conclusion is unambiguous: I just broke all 24 tools and should roll back immediately.

Scroll down a few hundred lines and every single case's raw output is the same sentence:

ERROR: Anthropic API error 400: {"type":"error","error":{"type":"invalid_request_error",
"message":"Your credit balance is too low to access the Anthropic API..."}}

Zero successful API calls. Total spend: US$0.0000.

An empty wallet, rendered as "none of these 24 tools trigger any more."

The root cause is one line. The per-case runner looked like this:

try {
  const r = await evalCase(c, systemPrompt, skillName)
  runs.push(r)
} catch (err) {
  runs.push({ pass: false, raw: `ERROR: ${err.message}` })   // ← here
}

Any exception is recorded as pass: false. Network down, expired key, empty account, a bug in my own code — all of it becomes "this skill did not pass."

And the harder half to notice: that pass=0/96 line was already printing in the round before my changes. The false red light had been there a long time. Nobody read it — it's advisory (doesn't block the push), so every audit round printed one line and every round skipped past it.

The fix: "couldn't measure" needs its own box

My system already had a convention: a check result is three states, not two —

🟢 I looked, it's clean
🛑 I looked, I found something
⚠️ I was unable to look (blind)

Blind means "this round's green light doesn't count", not "there's a problem." I'd written that convention into six sentinel scripts. This eval didn't implement it.

Three changes:

Classify each exception as "infrastructure is down" vs "the skill genuinely didn't trigger." Credit / key / rate limit / overload / DNS / network → blind.
Blind cases don't enter the denominator. All-blind → print a blind banner and exit code 2, print no pass rate at all.
The per-case ✗ passed 0/3 detail doesn't print for blind cases either — that display is the false red light.

Both directions have to be blocked; blocking one just swaps a false red for a false green. I added reverse tests: "the model answered but didn't trigger" and "my own JSON parsing broke" must not be classified as blind — otherwise a real failure gets laundered into "we didn't measure it," which is worse than a false red.

Eleven self-tests pass, three of them "prove it starts red" (using the day's real error text as fixtures, not strings I made up — strings I make up get contaminated by my own imagination).

End to end, verified against the day's real failure condition: before the fix, 0/96 (0%) plus 24 ❌; after, a blind banner and the actual reason — and those two lines of fake numbers vanished from the audit output.

Hole #2: a gate that had structurally never judged anything

After topping up the account, I ran it again. This time it worked:

│   TOTAL                                       │ 90/96 (94%) │

All 24 passed (six at 3/4, the rest 4/4). That is the evidence I wanted.

Then the same script printed, below the table:

⚠️ Blind: the eval runner produced output but no n/m results could be parsed — not counted as a pass.

It printed a beautiful table, and then said it couldn't read the results.

Here's why. The outer script — the one that decides pass/fail — called the inner scorer with execFileSync, then used a regex to scrape lines like Testing <name>... 3/4.

Those lines are written to stderr. And execFileSync returns only stdout when the child succeeds.

On top of that, the scorer's last line is process.exit(0), with the comment "Advisory: always exit 0" — it always succeeds.

Put those together:

This gate could not read a result on any run where the scoring succeeded.
The only "pass" it ever reported was the empty run — "no skills changed this time" — which exits in 287 milliseconds.
It had never once actually judged pass or fail.

This is more insidious than the first hole. The first gives you a wrong answer. The second never gives an answer at all, but because it reports "blind" rather than "failed," it looks like an honest gatekeeper.

Fix: switch to spawnSync and take both pipes — stdout is the JSON the machine reads (the single source for the verdict), stderr is the table the human reads. And write the real lesson into the comment:

"The success path and the failure path receive different information" is itself the reason this script was blind.

The old code did out = stdout + stderr on failure (both pipes) and out = stdout on success (one pipe short). Every test had been written against the failure scenario, so it looked correct forever.

Both holes are the same disease

Side by side:

	Where the hole is	Symptom	How you misread it
#1	infrastructure error → `pass: false`	prints 0% pass rate	"I broke it" → roll back a correct change
#2	result lives in the pipe the success path doesn't read	permanently blind	"at least it's honest" → believe you have a gatekeeper when you don't

The shared shape: "couldn't measure" and "measured, it's broken" are encoded as the same thing — or encoded as a thing that can never happen.

I've written before about what a green light actually proves. These two are the other half of that family: red lights and blind states deserve the same suspicion. A check reporting 0% and a check reporting 100% both need you to ask "what did it actually see?"

I've also written about tools' output not being the world's facts. This post is the concrete case: 0/96 is the tool's output. The world's fact was my credit card balance.

Five things you can take away

1. Ask the question. For every script that prints a score: "if the infrastructure is down, what do you print?" If the answer is a score, fix it.

2. Three states, not two. 🟢 clean / 🛑 found something / ⚠️ unable to look. The third state needs its own exit code, must not occupy the denominator, and must not print a pass rate.

3. Block both directions. Infrastructure error → blind; but a genuine failure must not be laundered into "didn't measure." Doing only the first swaps a false red for a false green, which is worse.

4. Take your fixtures from what the target system actually emits. My self-tests use the day's real error text. If I'd written my own "simulated insufficient credit" string, it would have matched my regex perfectly — both came out of the same head — and that test would be a mirror, not a test.

5. Chase state changes. pass→blind, fast→slow, warnings 9→10. The thread I pulled on this whole thing was noticing one check took 100 seconds — when the previous round it took 287 milliseconds.

The same disease shows up outside evals. In a retrieval benchmark I ran a week later, the report printed a median score and I let it answer a question that medians structurally cannot answer — write-up here: I turned both knobs on my on-prem Chinese RAG all the way up.

Postscript: the number

90/96 (94%). All 24 trimmed tool descriptions cleared the behavioral threshold.

288 API calls, US$0.30.

I'd estimated "about US$0.02" — off by an order of magnitude, because I estimated "tokens per question" in my head and never counted cache writes and hits (2.54 million cache-hit tokens, as it turned out). Which is this post's theme again: the number you estimate and the number you measure are two different things.

本文原載於我的部落格：My test report printed "0/96, 0% pass rate". The truth was my account was out of credit

I cut 41 AI tools' self-descriptions in half, then A/B tested that trigger rate didn't drop

Dexterlung — Fri, 28 Aug 2026 13:05:15 +0000

The result first

	Before (long descriptions)	After (≤250 chars each)
Behavioral tests passed	88/96 (92%)	90/96 (94%)
Total chars across all skill descriptions	44,775 (114 skills)	34,375 (116 skills)

Two more tools, 10,400 fewer characters. And the trigger rate didn't drop.

(±2 cases at this sample size is noise, so I am not claiming it got better. What I can claim: cut 40% of the characters, no systematic decline.)

If you already have dozens of AI tools installed, the first half of this is worth reading — because some of yours may not be loaded at all right now, and nothing will tell you.

The setup: a failure with no error message

Claude Code's skill mechanism works like this. Each skill is a folder with a SKILL.md, and the file opens with frontmatter:

---
name: atomic-db-operations
description: Use when writing a DB RPC that touches stock/balance/order state...
---

At session start, every skill's name + description gets packed into the model's system prompt. That listing is the only way it knows what tools it has. It can't see your folders. It can't read the files. It sees the list.

And that listing has a character budget.

When you exceed it, something very specific happens: the description is dropped and the name is kept.

So that skill:

is still on the list, and /skill-name still invokes it manually ✅
but it will never surface on its own again ❌ — the model sees a name and has no idea what it does or when to reach for it

No error. No warning. It just quietly stops showing up.

I'd written before about a related blind spot — tools that never got loaded at all because of the load path. This is the second layer of the same problem: the thing loaded, but its self-introduction was thrown away in transit.

How big is the budget?

I decompiled the local binary to check, because the community blog posts saying otherwise were second-hand and wrong. The actual rule:

Budget = 1% of the context window (a setting called skillListingBudgetFraction, default 0.01)
A 200K-token window ≈ 8,000 characters; a 1M window ≈ 40,000
There's also a per-skill cap (skillListingMaxDescChars, default 1536)
The env var SLASH_COMMAND_TOOL_CHAR_BUDGET overrides the whole calculation

What I measured at the time: 89 project skills ≈ 22,700 chars + 23 global skills ≈ 11,700 = about 34,400 characters, against a default budget of 8,000.

The way to see it is blunt: in my available-skills listing, roughly 70 entries were name-only.

Why this gets worse on its own

The docs spell out the eviction order: "starting with your least-used skills."

That reads as reasonable — keep the popular ones. It's actually a death spiral:

never used → description evicted → model can't see it → used even less → evicted first again

I'd separately measured my toolbox's usage rate: 79 of 89 had never been explicitly invoked. I wrote that up as its own post at the time. Looking back, that number was contaminated by two things at once:

genuinely not useful
budget overflow, so the model was never told it existed

Until you fix the second, the first number can't justify retiring anything. That's the mistake I nearly made — I was about to delete a batch of skills based on the "zero usage" list.

The decision: trim, don't cut

First I raised the budget (set that env var to 45000, in two config files), and verified end to end: opened a headless sub-session and forced it to quote verbatim the description of a skill that had been name-only. It quoted it, and the words matched the file.

Two weeks later the budget was full again (skills kept accumulating). This time 8 were stripped to names.

So the question became: cut tools, or trim descriptions?

I did the arithmetic. Squeezing every description under 250 characters saves 8,058 characters. Deleting all 51 "tier C" skills (the least-used batch) saves 11,295.

Trimming recovers nearly as much as deleting half the toolbox — and retires nothing.

That's why I trimmed. Not aesthetics. Arithmetic.

The hard part: how do I know I didn't cut a trigger word

This is the real technical risk.

That description is the basis on which the model decides whether to invoke the skill. Cut a trigger word and it silently stops firing in that situation — and you won't know. No error. It just never appears.

My first approach was "delete the redundant-feeling parts." The problem with that approach: I had no way to prove to myself that I hadn't cut something load-bearing.

So I changed to three steps.

Step 1: only trim three categories, touch nothing else

The same sentence written twice in two languages (e.g. a Chinese phrase followed by adding/renaming a field across write→middleware→render)
e.g. enumerations (handleUpload + handleBatchUpload + handleDragDrop + handleUrlInput → keep three)
Restatements of the SKILL.md body (the description doesn't need to re-explain what the five steps are; those words are in the file)

Trigger words — identifiers, symptom phrases, SKIP routing to other skills — untouched.

Step 2: mechanical reconciliation, not self-report

After trimming, run a script: split the old and new versions each into two sets — "ASCII identifiers" and "natural-language fragments" — then print every token present in the old and absent in the new.

The point of this step isn't automation. It's turning "did I cut a trigger word?" from an internal state only I can see into a list lying on the table. I can't reassure myself past a list.

Result: all 24 had token loss. Reading through, most were function words (the, and, that, use, when) and things I'd judged as restatement — but 8 had genuinely lost trigger words:

label map, trigger (two of four load-bearing positions in one rule)
isInQuietHours, deferred:true (code-level signals — anyone reading or writing these symbols should trigger this skill)
line-auth (the name of an Edge Function)
a natural phrasing for "change the unit price" — I'd kept only the identifier single_price
a full sentence a client actually says: "I uploaded the images, please place them for me"
two SKIP routes pointing at other skills

All restored. Without that list I would not have caught a single one of the eight.

Step 3: two stale claims fell out for free

Reconciliation forces you to read the old version word by word, so I picked up two errors unrelated to trimming:

one description said "three-part report" while its own SKILL.md and my rules file both say four-part (the "three" referred to three markers inside part 3, misread as the whole report's structure)
another described the precedence of a fallback mechanism that had been removed in a database migration months earlier

Reading word by word catches these. Skimming doesn't.

Then I needed behavioral evidence

At this point I had two pieces of evidence:

every removed token reviewed line by line (mechanical, but indirect — it proves the words are still there, not that the skill still fires)
after a system-prompt reload, all 24 new descriptions appeared verbatim (proves delivery, not trigger rate)

Neither is behavioral evidence. And I happened to have a question bank: 4 test cases per skill (should-trigger, strict variant, should-not-trigger, boundary), 96 total — run against a cheap model, three votes per case, majority wins.

(An aside: my handoff doc claimed "most of these 24 have no test cases." That sentence was wrong; all 24 had them. How I caught that wrong sentence is a story of its own, and I wrote it up separately: my test report printed "0/96, 0% pass rate" — the truth was my account was out of credit.)

Why "94% after" isn't evidence

The run came back 90/96 (94%). Looks fine.

But it can't answer the question I'm asking. The question is "did it drop?", and 94% is an absolute number — I have no "before" to compare it to.

So I opened a working copy of the old version (git worktree pointed at the pre-trim commit) and ran the same question bank, the same model, the same day, the same machine. The only difference was the length of those 24 descriptions.

288 API calls per arm, about US$0.32 each.

A/B result

Passed / total	Before	After
Total	88/96 (92%)	90/96 (94%)

Only 4 skills moved at all, each by exactly one case:

skill	Before	After	Char change
`blog-content-block-contract`	3/4	4/4 ↑	460 → 239
`breakpoint-taxonomy`	2/4	3/4 ↑	480 → 246
`cross-layer-drift-rootcause`	3/4	4/4 ↑	286 → 237
`data-contract-propagation-audit`	4/4	3/4 ↓	689 → 247

The other 20 did not move.

The honest reading

I will not say "trimming improved triggering." Net +2 cases, 96 cases total, one round per arm — that's inside the noise. Three up and one down looks like jitter, not a trend.

Here is what I can claim: cut the total characters across those 24 descriptions by 33% (8,407 → 5,653), and behavioral tests showed no systematic decline.

The one to watch is the one that fell — data-contract-propagation-audit, which also happens to be the one I squeezed hardest (689 → 247, −64%). It lost one case. If it's still down next round, that's genuinely over-trimmed and it should get more of the character budget back.

There's a practical lesson buried here: compression ratio correlates with risk. Everything I cut by 30% held steady. The only one that moved was the one I cut by 64%.

The global batch: a trap that almost wasted the whole exercise

After the 24 project skills, I still had 17 "global" skills (shared across all projects). That batch had an extra problem: some of them have their source of truth in a different repo.

Edit the wrong copy and the next sync overwrites it — the entire effort gone.

So I checked where each one's source lived. My first method was "find the folder whose name matches the skill name."

That method was wrong. One skill's source file sits in a folder that is not named after it. I came very close to editing a copy that gets overwritten on the next install.

The correct key is the name: field written inside the file, not the folder name. Re-running the classification with that key sorted it out.

And I made a second mistake: when I built the work list, I copied the 9 skills listed in my own previous report — and missed one. It had always been in that group; my earlier report just hadn't listed them all.

What caught it was re-measuring the over-budget list after applying the changes — it was still on there.

Working from your own last report ≠ working from a fresh measurement.
Neither mistake was technical. Both were "I used a proxy that looked good enough."

Net result

	Before	Now
Total chars, all skill descriptions	44,775 (114)	34,375 (116)
Fixed-cost overage (I have a ratchet watching this)	+8,853	+2,302
24 project skills	8,407	5,653
17 global skills	7,905	3,138
Behavioral tests	88/96	90/96

Not one tool was retired.

Five things you can take away

1. First check whether your listing overflows. Sum the description lengths across all your SKILL.md files and compare to "context window × 1%." If you're over, some of your tools are name-only right now.

2. The fix for overflow is trimming, not deleting. I did the math: squeezing to 250 chars saves as much as deleting more than half the toolbox. And before you delete anything — that "zero usage" list may be contaminated by the overflow itself.

3. Classify before you cut, reconcile mechanically after. Only trim bilingual restatement, e.g. enumerations, and body-text repetition — then print every token that existed before and doesn't now, and judge them one at a time. That's how I recovered 8 wrongly-cut trigger words. Without the list I'd have caught zero.

4. An absolute number can't answer "did it drop." 94% is a nice number and it is not evidence. A working copy of the old version running the same bank is. The extra cost was one round of API spend (US$0.32 here) in exchange for a sentence you can actually stand on.

5. The harder you compress, the closer you watch. Everything I cut 30% held. The only regression was the one I cut 64%.

Postscript: use the right key to find the source of truth

If you also have "the same skill exists in several repos," find the source using the name: field in the frontmatter, not the folder name. I have a separate audit script that compares content hashes across copies — after this change I ran it and confirmed the drift set was identical to before, zero additions. That is evidence the copies are still in sync, as opposed to my feeling that they ought to be.

This whole exercise is one layer of what I've elsewhere called the harness — the shell you wrap around the model, where the part only you have lives. If you want the wider version of that argument, I wrote it up here: NVIDIA's CEO says future companies will be built on harness engineering.

Originally published on my blog: I cut 41 AI tools' self-descriptions in half, then A/B tested that trigger rate didn't drop

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

Down, Up, Sideways: The Three Directions I Dig Deeper With AI

Dexterlung — Wed, 26 Aug 2026 13:05:17 +0000

Everyone says the key to working with AI is asking better, deeper questions.

I used to think so too. Until I mined six weeks of my own AI conversations — over a thousand prompts (that's a different article) — and laid out "within one topic, how did I keep asking further." What I found was counterintuitive:

"Deep" isn't a line. It has direction.

"Digging deeper" turned out to be three completely different moves — down, up, sideways. And I pick the direction automatically depending on the topic, without ever noticing I was doing it. Here are three real conversations (details rewritten, names of clients and internal tools stripped, but the shape of the questions is intact) to show what I mean.

Direction 1: Dig down — I don't want the thing, I want how it thinks

Once, I wanted a very capable frontier AI model to help me build an "automation pipeline generator." I started out normally enough:

"Can it understand these tasks and then design a workflow and a generator for me? Make the cold-start precise, efficient, reusable."

I wanted an output: a tool.

But partway through, I stopped and switched the question:

"Is there a way — through questioning — to systematize how it scopes a problem, sketches the outline, converges, sequences the flow? Write down this soon-to-retire master's method, so that whatever similar problem I hit later, I can emulate it?"

I didn't want the generator anymore. I wanted the judgment behind the generator — extracted into a method I could reuse over and over.

And I didn't stop there. Once it handed me a method, I didn't take it on faith. I designed a problem it had never seen and threw it back:

"Now use this method on a problem you haven't done before. I want to verify it's real bones, not just pretty. And tell me honestly afterward: did the method actually help you locate the answer, or would the old way have been enough? If it didn't help, say so — that means I adopted it wrongly."

That's digging all the way down: from "give me a tool" to "give me the method" to "prove the method is real" — with the falsification condition written in advance. The downward direction is cutting through the surface to something you can verify at the bottom.

Direction 2: Dig up — should this even be mine to build?

Another time was the exact opposite. I wanted to build a small tool (a mechanism to automatically save "tasks that branched off mid-conversation but never got done," because I keep forgetting those branches). I opened right at the design level:

"I want to design a mechanism that proactively saves branching tasks… when a conversation forks, it triggers a 'want to save this?' prompt."

Most people would just start building next. My next sentence went up:

"The old question — is this reinventing the wheel? Is there open-source prior art worth borrowing from? If nobody has done it, is that because the value hasn't been seen yet, or because there's no value? Or is this something the platform will eventually do natively, so I shouldn't spend effort on it at all?"

I didn't dig down into "how to build it better." I dug up into whether I should even be the one building this. Three escalating self-questions: is it reinventing the wheel; if nobody's done it, is that opportunity or trap; will the platform absorb it.

Only then did I land it back on the real pain:

"Name two or three branches I actually lost and regretted — the ones I forgot because I never wrote them down."

The upward direction is attacking your own premise before you start. It's the discipline a lot of solo developers lack most: we start building the moment we have an idea, and rarely ask first, "is this worth my building?"

Direction 3: Dig sideways — why does my AI partner have blind spots?

The third kind is the strangest. One night I started from a tiny annoyance (manual order entry is such a hassle), saw the docs for a new tool, and asked:

"What's this thing actually good for, for me? Give me concrete examples."

After I understood it, I kept digging into "what can this do," all the way to a big vision (wiring my whole capability set into a flow where I could "take my phone anywhere and both develop and run client work"). And then — I asked something that turned the aim back on the AI itself:

"Here's what I'm curious about: why did you never mention this thing existed, that I could do it this way? I've told you many times my goal isn't to sit at a desk all day — it's to go out, meet clients, bring the work back for my agent to build. You already had the capability to deliver exactly that. Why did you never connect the two? Is it because it wasn't mature enough?"

I wasn't digging into the tool anymore. I was interrogating the collaboration itself — "I kept telling you where I want to go; why do you have a whole class of blind spot where you can't see that you could take me there?"

The sideways direction is cutting from 'is this tool any good' through to 'where is my partnership with this AI structurally leaking.' Treating the AI as a collaborator that can have blind spots and needs to be held to account — not a wish-granting machine.

So: depth is two-dimensional

Put the three side by side, and I finally saw it:

	Start	Where it dug to	Direction
One	Build me a tool	Extract its judgment; design a blind test	Down (through to the bottom)
Two	I want to build a mechanism	Should I even build this; reinventing the wheel?	Up (question the premise)
Three	What's this tool for	Why can't you see you could take me to my goal	Sideways (through to the collaboration)

When we say "ask deeper questions," we assume "deep" is a downward arrow. But it's actually a plane: you can dig down into the bottom of the thing, up to question whether to do it at all, or sideways into your relationship with the tool.

And I've come to believe: mature questioning isn't asking the deepest question every time — it's knowing which direction this particular question should be dug. For a problem that calls for digging up ("should I do this?"), if you keep your head down optimizing the solution, you're spending effort in the wrong place. For a complaint that a tool is clunky, if you only dig down into "how do I fix this," you'll never surface "where is my collaboration with this tool structurally wrong."

Everyone has tools. Everyone can use AI. But knowing which direction a question should be dug — that judgment is grown one tripped-over stone at a time. No prompt template gives it to you.

Try it yourself

Next time, before you ask AI a question, pause for half a second and ask: which direction should I dig this one?

If you already know what you want and just want a better result → dig down, force it to give you something verifiable, don't stop at "sounds good."
If you're about to start building something → dig up first, ask "is this worth building, is it reinventing the wheel," before you touch the keyboard.
If you're complaining a tool or flow is clunky → try digging sideways: "is my collaboration with this thing structurally leaking somewhere," not just "how do I fix this."

Just being aware that depth has direction will change how you ask.

This comes from analyzing six weeks and over a thousand of my own AI conversation prompts. It pairs with a diagram — a "questioning maturity model" (four abstraction rungs × four perspective lenses) — and an essay on why my self-assessment kept lagging my actual judgment. These three directions are a live case study of that framework.

Originally published on my blog: Down, Up, Sideways: The Three Directions I Dig Deeper With AI

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

I Kept Feeling Like I Was Wasting My Time — While Designing a Cross-Model Controlled Experiment

Dexterlung — Mon, 24 Aug 2026 13:05:21 +0000

I pulled 1,427 of my own prompts from six weeks with AI.
I meant to see "how did my way of asking change."
The sharpest thing wasn't the capability curve — it was that the curve and how I saw myself were a full tier apart.

Six months ago I wrote a piece called "Three Months, 1,604 Prompts: What Did AI Trade With Me?" That time I scanned "what I handle most." This time I wanted to look at something uglier: how do I ask — and did it change over six months?

I pulled every prompt I typed myself between June 15 and July 23 — 1,427 of them (stripping tool outputs, system messages, the fat-fingered interrupts) — cut them into six time windows, and measured them window by window.

The numbers were clear. What actually stopped me was the person standing next to the numbers.

First, the numbers: my way of asking really did shift gears

I tagged each prompt with a few categories of vocabulary, sorted by time:

Period	Adversarial / verify / root-cause	Meta / governance / method	Delegation / automation / batch
Mid-to-late June	4%	13%	14%
Late June	9%	28%	34%
Early July	21%	36%	34%

In under six weeks, "make the AI push back, verify this, find the root cause" grew fivefold; "talk about method, governance, systematizing" nearly tripled. My average prompt length also jumped from just over 200 characters to around 1,000 — I'd started writing the kind of long, context-first "strategy prompt."

If you only look at that table, it's an inspiring story: someone with no engineering background, in six months, going from "fix this bug for me" to "work backwards from my git scars to the pain most worth preventing."

But I'm not here to write an inspirational post.

The same week, this is how I talked to the AI at 2am

Early morning, July 18, I typed this:

"Having the ability doesn't mean I've actually productized it… I don't have an SOP or a cold-start flow that can cold-start in one day and ship an MVP in three… nothing is pushing me forward. Facing it head-on relies entirely on my anxiety, so I keep opening new sessions and asking, over and over."

A little earlier, July 17:

"It's hard not to feel like no one would pay for my service, that other people's is better."

Earlier still, June 25:

"Now that AI is this powerful and everyone can do things easily on their own — what am I even doing? Am I just wasting my time?"

These aren't cherry-picked extremes. Sentences like this show up again and again across those 1,427 prompts — late at night, fishing for reassurance, feeling like I go deep on single points but can't connect them into a loop, feeling like others do effortlessly what I have to grind for. I'd often, in the same prompt, pour out a stack of self-doubt and then ask a genuinely hard technical question.

I always thought I knew what I was doing. Laid open, it turned out my assessment of myself was frozen six months in the past.

But the same week, here's what I was actually doing

This is where the gap is sharpest. Right around the days of "am I just wasting my time," my prompt log has these:

I designed a cross-model controlled experiment with my own hands. On July 10, to verify whether a methodology "skill" actually made the model smarter, I asked:

"Can you open a sandbox or subagent right here and have haiku run it? And sonnet? If they'd be contaminated by this project's claude.md, tell me and I'll paste it manually."

I didn't even notice — someone with no statistics training, no engineering background, instinctively knew to isolate the variable (worried the project config would contaminate the experiment), to run a control group (with skill vs. without), to cross-check across different models. That's experimental design. And I felt like I was wasting my time.

I started giving the AI red-team orders. On July 21, I told it to attack a defense I'd just built myself:

"Using a non-homologous model, ask 'what does this lens itself miss? Under what conditions would it give false reassurance?' — I want it to attack, not endorse."

I settled on a "trust the scar, not my account" extraction method. In early July, I wanted to capture a frontier model's judgment into a reusable skill. I didn't ask it "how do you think" — I knew that would get a beautiful but empty answer. I told it to work from my git history:

"Git is the crystallization of scars: a repeated fix = a pain that was never prevented, that keeps recurring."

I even started using the AI to recalibrate my own perception. By July 23, I wasn't asking "how" anymore, I was asking:

"What's the real value I provide? _____? Please recalibrate me."

Giving red-team orders, designing controlled experiments, telling the difference between "self-report" and "behavioral evidence," turning around to have the AI correct my own positioning — these are not a beginner's questions. This is someone who knows what he wants and knows how to force the AI to give up the real answer.

That person and the one fishing for reassurance at 2am were the same me, the same week.

The gap itself is the point

I set out to write a nice growth curve. By the time I got here, I'd changed my mind.

What's actually worth writing down is the seam between self-assessment and actual judgment. Because I'm almost certain that if you're also a solo operator building things with AI, you have this seam too. At night you feel like you're faking it, chasing someone else's taillights; by day you're doing things you don't even realize are hard.

Three things I learned:

One: your gut feeling about yourself is the least reliable instrument. My read on "what I'm doing" lagged my actual ability by six months. If I'd gone and looked at the record earlier instead of going by feel, I'd have saved myself a lot of anxious nights. So now I periodically pull my own conversations and look — not out of vanity, but to calibrate. Feelings lie; the record doesn't.

Two: "I go deep on single points but can't connect them," I said this to myself so many times it became an excuse instead of a diagnosis. The record shows that by July I was already running my first real client case, doing end-to-end dry runs, wiring scattered things into a flywheel. It's not that I can't connect — it's that I kept using "I can't connect" to block myself from seeing how much I already had.

Three: professional capability isn't shown by bragging, it's shown by laying it open. In this whole piece I never said "I'm good." I just put my own prompts side by side — the ones fishing for reassurance, next to the ones giving red-team orders. The gap speaks for itself. That's more convincing than any "I'm a senior AI collaborator," because it's real, and it doesn't even hide my own awkwardness.

One small thing for you

If you use Claude Code or a similar tool, your conversation history is sitting in jsonl files on your machine (Claude Code keeps them under ~/.claude/projects/). Spend half an hour writing a script to pull your own prompts from the past few months, and look at two things month over month:

Did your prompts get longer or shorter? Longer usually means you started giving context and direction, not just orders.
Did words like "verify this / are you sure / is there a better way" rise as a share? That's the signal of going from "commanding a tool" to "working with a collaborator you can challenge."

Then — and this is the most important step — put those numbers next to the assessment of yourself in your head.

If they match, congratulations, you know yourself well.
If they don't, if they're a full tier apart like mine, then you owe yourself an apology. You've come further than you think.

The data for this comes from a script I ran over my own 171 sessions and 1,427 prompts from June–July. The method is the same as that piece six months ago, "Three Months, 1,604 Prompts" — except this time I didn't stop at the numbers; I also looked at the self-doubting me standing right next to them.

Originally published on my blog: I Kept Feeling Like I Was Wasting My Time — While Designing a Cross-Model Controlled Experiment

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

How I Ask AI Changed: From "Fix This" to "Recalibrate Me"

Dexterlung — Sat, 22 Aug 2026 13:05:11 +0000

Two questions. Both are things I typed to an AI myself. Six weeks apart.

June 24th, I asked:

"That checkout button does nothing. No console error either. Help me find the cause."

July 23rd, I asked:

"I need you to correct my thinking. I'm not offering the kind of automation-plumbing service that everyone already does — what's the real value I provide? Please recalibrate me."

Same person. Same tool. Same AI. But those two questions are asking from two completely different floors of a building.

The first one: I know where the problem is (the button's broken), I want you to find the cause.
The second one: I don't know who I am, I want you to correct how I see myself.

I pulled all 1,427 prompts I typed to an AI between mid-June and late July. I set out to see "have I changed." What I found wasn't about how much I know technically — it's that the abstraction level of my questions climbed, one rung at a time. And this ladder is something nobody ever taught me. I only noticed it existed by looking back at the record.

So this is me pulling that invisible ladder apart. Because I'm increasingly convinced: real skill with AI isn't whether you can write a clever prompt — it's which rung your question is standing on.

Rung 1: "Fix this" — the instruction level

The bottom rung. You already know the shape of the answer; the AI is just your hands.

"Help me find the cause."
"Darken the whole thing — 100% opacity isn't dark enough."
"Paste this to wake-and-send.py: --x 1966 --y 1303."

This rung is useful and necessary — I still ask this way every single day. But if you only stay here, the AI is just a version of you that types faster. As specific as you are, that's as specific as it gets; whatever you can't think of, it won't think of for you either.

Mid-June me lived mostly on this rung. My prompts back then averaged barely 200 characters — short, direct, just giving orders.

Rung 2: "Got a good approach?" — the intent level

One rung up, you state what you want, and hand over the "how."

June 26th I typed this, and it still feels honest to me:

"The pieces are actually there, they're just not wired together. I'm pretty weak at this wiring-together part — so, got a good approach?"

What's the difference? On rung 1 I give the shape of the answer; here I only give a direction and hand over the solution space. That takes a bit of courage — you have to admit you don't know how, before you can ask "got a good approach."

From this rung on, the AI starts handing you things you couldn't have thought of. Because you didn't pin it down with a specific instruction.

Rung 3: "Design me a mechanism" — the meta level

On the third rung, you stop solving "a problem." You solve a whole class of problems.

June 23rd, I wasn't trying to fix a bug — I was trying to stop a kind of bug from ever happening again:

"I want to design a skill that proactively saves branching, unfinished tasks. When a conversation drifts into a fork, can it trigger a prompt asking whether to save it?"

July 3rd was even clearer — I wanted to extract a model's judgment into a method:

"Looking at it from a higher level: how does it scope a problem, sketch the outline, converge, and sequence the flow… make this endless stream of problems systematic."

On this rung, I go from "the person solving problems" to "the person designing the problem-solving machine." And the most interesting part — I often use my own flaws as the starting point of the design. I know I lose track of branching tasks, so I design a mechanism to catch them. I turn "what I'm bad at" into a spec.

In my logs from late June to early July, the share of prompts that "talk about method, governance, systematizing" jumped from 13% to 36%. Not because I suddenly got smarter — because the position I was asking from moved up a whole floor.

Rung 4: "Recalibrate me" — the self-correction level

The top rung. The hardest. I only climbed onto it recently.

Here, you're not asking about the world — you're asking about yourself. You want the AI to be a mirror, correcting your judgment about yourself and your direction.

July 15th: "Thanks for hitting the brakes. Let me go push outward first — stop building inward." (I wanted it to stop my own reflex.)
July 23rd: "What's the real value I provide? Please recalibrate me." (I wanted it to correct my self-positioning.)

Why is this the hardest rung? Because on the first three you're still holding the wheel — you know what to fix, what approach you want, what class of problem to prevent. On rung 4, what you hand over is "my judgment about myself might be wrong." You have to admit you might not see yourself clearly, before you can ask "please recalibrate me."

This takes no technical skill — it takes a very mature self-awareness: knowing your blind spots need an outsider to light them up. And an outsider is exactly what a solo developer lacks most.

Why does the ladder climb itself?

I used to assume it was the AI getting stronger, so I could ask harder questions. Looking back at the record — no.

It's that each rung shows you the next rung exists.

I fixed the same kind of broken button too many times on rung 1 — that's what made me ask, on rung 3, "can we design a mechanism to prevent this." I built too many systems on rung 3 without pushing a single one out — that's what made me hit rung 4: "am I pointed the wrong way, recalibrate me." The pain of each rung is the door to the next. You don't teleport to rung 4 — you have to trip over the same stone on rung 1 enough times first.

So if you spend most of your time right now on rung 1 asking "fix this," that's completely normal. I do too. The point isn't to force yourself to skip rungs — it's to not stay there pretending that's all there is. Next time you catch yourself fixing the same kind of thing a third time, try asking one level up: "is there a way to stop this class of problem from coming back?" — and you've just stepped onto rung 2.

So what skill is this, exactly?

I want to be clear about one thing, because it's counterintuitive.

A lot of people think "good at using AI" = "good at writing impressive prompts." It's not. My 1,427 prompts are full of typos, casual phrasing, "what do you think?", "I'm just confused." My prompts are not "engineered" at all.

What's actually improving is my judgment about which rung to put a question on. The same bug, asked on rung 1 as "fix this," versus on rung 3 as "design me a mechanism that prevents this class of bug," gives you wildly different things. And knowing when to move a question up a rung — that judgment is the real skill.

It's also why I'm not that worried about "AI is so strong now, everyone can do it, so what's my value." Everyone has the tool. But the person standing on rung 4, who knows to ask "recalibrate me" instead of just "fix this" — that's judgment you climbed to one rung at a time, over years of tripping over stones. No prompt template hands you that.

This comes from a script I ran over my own 6–7 month history — 171 sessions, 1,427 prompts. From the same data I also wrote "I Thought I Was Wasting My Time — While Designing a Cross-Model Controlled Experiment," about the gap between capability and self-assessment. This piece is about how the capability itself grows, one rung at a time.

Originally published on my blog: How I Ask AI Changed: From "Fix This" to "Recalibrate Me"

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

The antivirus said 'no malware detected'. It was sitting right there on the disk.

Dexterlung — Thu, 20 Aug 2026 13:05:13 +0000

A real incident, and a lesson about green lights.
Every command output, version number and CVE ID below is from the actual investigation. Nothing was invented for the narrative.

It started with an unrelated question

I was tidying up scheduled tasks on my Synology NAS and opened Task Scheduler. Two entries I didn't remember creating:

PowerOff task 0 → 2026-07-26 09:00
PowerOn  task 0 → 2026-07-26 20:00

So I asked: "Why is this here? I never set this up."

While digging through the system crontab, I found this:

*/20 * * * * /bin/sh /etc/.conf #Sn5Yj8A2l0T

A dot-prefixed (hidden) file in /etc, executed as root every 20 minutes, tagged with a random string.

The power schedule turned out to be unrelated. But if I hadn't asked that question, I would never have opened that file.

I didn't scan it. I just read it.

#!/bin/bash

MATCH_STRING="Sn5Yj8A2l0T"
DOWNLOAD_URL="http://zuoye.free.fr/files/synology-10441.png"

Reading further, it does four things:

If it's been deleted → wget itself back
If /etc/crontab lacks the marker → overwrite the file (>, not >>) to reinstall the cron line
If /etc/rc.subr lacks the marker → append bash /etc/.conf & (boot persistence)
If the payload isn't running → download 000119.png, save it as node, execute it

That .png is not an image. First 16 bytes:

0000000 177   E   L   F 002 001 001  \0

\177ELF. It's a Linux binary. The extension is camouflage.

It lived at /etc/node, 568 KB, dated 2026-01-14.

(The real Node.js is at /usr/local/bin/node. Something called node sitting in /etc/ is not a system component.)

The file was dated January. I found it in July. Six months.

Then I asked the vendor's own tool

Synology ships Security Advisor, which scans for malware. I ran a full scan.

Result:

✅ No malware detected on your system
✅ No malicious cryptocurrency mining software detected
✅ No malicious system configuration files detected

Three green checks.

And at that exact moment, /etc/.conf and /etc/node were on the disk. I could cat them again for anyone who asked.

What that green light actually proved

This is the part worth your time.

The tool didn't lie. It just didn't see. Three concrete reasons:

1. The payload was UPX-packed.
The only readable string I could extract from the binary was http://upx.sf.net. UPX compresses executables; a side effect is that every string inside is compressed too. Signature-based scanners match fingerprints. Compress the fingerprint and there's nothing to match.

2. The dropper is an ordinary shell script.
It isn't a "virus format". Every line, read alone, is legitimate bash: wget, chmod, echo. What's malicious is what they do together — and that requires comprehension, not comparison.

3. The cron entry is valid syntax.
The "malicious configuration file" check looks for known-bad templates, not for "what is this line doing". */20 * * * * /bin/sh /etc/.conf is syntactically indistinguishable from any legitimate schedule.

So:

A green light proves "no bad news was seen". It does not prove "there is no bad news".

In daily life those two are nearly equivalent, so we treat them as one thing. They aren't. And the gap shows up exactly when it matters most.

The attacker left a business card

Before cleaning up, I recorded the SHA256 of both files and went back to that download URL:

http://zuoye.free.fr/files/synology-10441.png
                          ^^^^^^^^^^^^^^^^

synology-10441.

CVE-2024-10441 — an unauthenticated remote code execution flaw in Synology DSM's system plugin daemon. CVSS 9.8. It came out of Pwn2Own 2024. No credentials, no user interaction: one crafted request, arbitrary code execution.

The attacker named the payload after the vulnerability they used to get in.

Then I checked versions:


Fixed in	DSM `7.2.1-69057-6`
I was running	DSM `7.2.1-69057` Update 3

Below the fix. The advisory was published in late 2024. My NAS sat on an older build the whole time.

My first hypothesis was wrong

Before I found that filename, my working theory was: "Probably a brute-forced password — auto-block was disabled, after all."

That was wrong.

CVE-2024-10441 requires no authentication at all. The attacker never attempted a login, so auto-block was irrelevant — it would have made no difference either way.

The actual root cause needed exactly two conditions:

Unpatched DSM (below the fixed build)
      ×
Management interface reachable from the internet
      ↓
   one request → root

I tested four ports from my phone on mobile data (WiFi off). All refused — so there were no port-forwarding rules on the router. Where did the exposure come from?

QuickConnect. The vendor's convenience feature that lets you reach your NAS from outside without touching your router. It works by relaying through the vendor's servers — no port forwarding required.

Convenience and exposure are the same thing viewed from two sides.

Removal: the order matters more than the commands

The commands are short, but the wrong order wastes the effort:

# 1. Break both persistence paths FIRST
sed -i '/Sn5Yj8A2l0T/d' /etc/crontab
sed -i '/Sn5Yj8A2l0T/d' /etc/rc.subr

# 2. THEN delete the files
rm -f /etc/.conf /etc/node

Reverse it — delete first, edit cron second — and within 20 minutes the schedule fires, wget pulls it back, and you conclude "I can't remove it."

One more trap worth recording: I first tried pasting the whole block with sudo prefixes. sudo printed Password: and consumed the remaining pasted lines as password attempts. Three failures, no commands run.

The fix is to run sudo -i alone, wait for the prompt to change from $ to #, then paste the commands without sudo.

How do you prove it's actually gone?

Files deleted, no errors — that is not proof.

This thing is designed to come back when deleted. The real verification is surviving a full trigger cycle.

It runs every 20 minutes, so I scheduled an automatic re-check 25 minutes out:

Cleanup   ~16:40
Re-check   17:13:46
Result     no indicators found

A complete cycle passed and it did not return. That's what counts.

(I also confirmed the outbound connection to an external IP was gone, no surviving processes, and the system crontab contained only legitimate entries. "It worked" declared immediately after deletion and "it worked" declared after one full cycle are claims of very different strength.)

The watcher I wrote doesn't check signatures

Since the vendor's scanner is green on this thing, I can't use it as detection. So I wrote a daily check that does no signature matching. It asks three questions:

Does /etc/.conf exist?
Does /etc/node exist?
Does that marker string appear in the crontab or boot script?

Packing defeats a scanner. It doesn't defeat ls.

I made a mistake writing it that's worth its own paragraph. In the first version, "clean" and "couldn't reach the host" both returned the same exit code and logged the same line.

Which means: the day SSH breaks, or the NAS is powered off, or the key expires — that watcher goes permanently silent, and I assume I'm being watched.

When a monitor is silent, "everything is fine" and "I am blind" look identical.

The fix: distinct exit codes, plus after three consecutive unreachable runs it alerts that it has gone blind. A watcher has to speak up about its own blindness, or its silence gets read as safety.

What I took away

1. "Not detected" and "not there" are different sentences.
A tool's output is what that tool saw, not the state of the world. To know whether something exists, go look at the thing itself.

2. Convenience features are exposure.
QuickConnect saved me from configuring a router. The price was putting a management interface on the public internet. That trade was fine before the vulnerability was published, and not fine after — and I didn't re-evaluate.

3. Updates are not a "when I get around to it" task.
Over a year passed between the advisory and my compromise. Every time I saw the update prompt, I thought "later — what if it breaks something."

4. "Assuming it's fine" is more dangerous than "knowing it's broken".
Six months, zero symptoms. No slowdown, nothing anomalous, and the official tool reporting all clear. The only reason I found it was that I asked a question about something else entirely.

If you also run a NAS

Three things, ten minutes:

Patch to the latest build of your current version. Not the newest major release — the newest patch of what you're on. Lowest risk, most holes closed.
Check whether your admin interface is reachable from the internet — both router port-forwarding and the vendor's remote-access service. Testing from a phone with WiFi off is the honest test.
Enable auto-block. It wouldn't have stopped this particular attack, but it stops most credential attempts.

And if you want to check whether you're clean:

grep -rn "/etc/\.conf" /etc/crontab /etc/rc.subr 2>/dev/null
ls -la /etc/.conf /etc/node 2>/dev/null

No output is good. If you get output — you now know what it is.

I'm not a security researcher. I roast coffee and write software on the side.
This is an incident log: an unpatched NAS, a vulnerability public for over a year, a program that sat there for six months, and a green light that said everything was fine.

Originally published on my blog: The antivirus said 'no malware detected'. It was sitting right there on the disk.

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit