DEV Community: Dexterlung

I ran claude plugin eval on 89 skills. The trigger rate fell from 100% to 55.6%.

Dexterlung — Mon, 28 Sep 2026 13:05:19 +0000

The problem with skills is that failure is silent

My coffee e-commerce project has 89 skill directories under it. Each one exists because I got burned: remember the GRANT on that migration, keep those three layers in sync when you rename a column, query the database before you guess at the code.

Skills work by having the model read their descriptions and decide, on its own, whether to invoke one. That's the whole problem. When it doesn't invoke, nothing happens. No error, no warning, no log line. You just watch the AI start working, and find out two days later that it skipped the rule.

I measured this by hand once, back in July: four natural-language prompts, four sessions, counting how many times the skill name showed up in the reply. Sonnet hit 3 of 4, Haiku hit 1 of 4. I wrote the numbers into my project rules and never measured again, because doing it by hand is expensive and gives you a one-off number that goes stale.

Last week Claude Code quietly shipped claude plugin eval.

The mechanism is simple. One design choice isn't.

You write test cases — realistic sentences a user might type — plus graders that score the result. Each case runs in a fresh isolated session, three times by default because the model is non-deterministic.

Standard stuff. What surprised me: it runs a second arm by default, with your plugin not loaded at all.

The difference between the two scores is Δ. The docs put it bluntly:

If a case scores 1.0 both with and without the plugin, the plugin isn't what made it pass.

Vendors don't usually ship a default that proves their users' work is useless. This design concedes something: your skill might be doing nothing, and you'd have no way to know.

First measurement: 100% with one skill loaded

I reused the four prompts from July rather than writing new ones. Writing your own test corpus has a trap — you unconsciously write sentences your skill happens to catch. The old corpus was written before I knew what I'd be measuring.

Three positive cases (add a back-in-stock notification, add a ship-date column, add a daily-summary page for staff) and one negative (change the homepage headline and button colour). One grader: did that skill get invoked, yes or no.

Result: 12 for 12. All three positives fired, the negative correctly didn't. $2.49, 273 seconds.

I knew immediately the number was useless, because the test plugin contained exactly one skill. When there's only one candidate and its name matches the task, picking it costs the model nothing. My real environment has 89 competing for the same attention.

Second measurement: 55.6% with 89 loaded

I wrote an assembly script that builds a throwaway plugin from all 89 skills at run time. (Deliberately no skill copies in version control — copies drift from the original, and this project has already paid that bill.)

Same prompts, same model. The only variable is 1 skill versus 89:

Case	1 skill	89 skills
Back-in-stock notification	100%	67%
Ship-date column	100%	33%
Staff daily-summary page	100%	67%
Homepage copy (should NOT fire)	100%	100%

Positives combined: 5 of 9, 55.6%. The negative case stayed correct in both.

That lands in the same range as my July hand-measurement (Sonnet 75%, Haiku 25%), which is mildly reassuring — two completely independent methods, one manual and one automated, agreeing.

I had a conclusion forming: too many skills dilutes attention, time to prune.

Then I did something stupid

Before writing that conclusion I wanted one more thing: in the runs where the skill didn't fire, what did the model invoke instead?

If it picked a more appropriate skill, that's correct delegation, not a defect. If it picked one with an overlapping description, the fix is editing descriptions, not deleting skills. Both leave fingerprints in the trace.

Then I realised I had deleted every trace ten minutes earlier.

The tool leaves a sandbox directory per run and prints "remove it when you're done." I cleaned them all up in one sweep. Those directories held the only thing that could answer the question.

This was the expensive lesson of the day, and it's a dumb one: I deleted something far more valuable than tidiness, in order to be tidy. The tool tells you to remove them because they may contain agent-written content. That is not the same as telling you to remove them before you've looked.

Re-run with --keep-temp: $0.82.

The traces: it didn't fire the skill, but it did the skill's job

All three re-runs failed to invoke. Here's every tool call each one made:

run 1: Glob, Glob, Glob, Glob, Glob, Grep
run 2: Glob, Glob, Glob, Glob, Glob
run 3: Glob, Glob, Glob, Glob, Glob

Not one alternative skill invoked. It went looking through files instead.

So "correct delegation" and "overlapping descriptions" are both ruled out. But the thing that actually changed my conclusion was the final reply. All three looked like this (excerpt, translated):

Based on this project's skill descriptions, here's what I expect to touch (this is expectation, not fact — I'll confirm by reading files once you give me the path):

DB: new migration adding orders.estimated_ship_date, with the tenant_id / RLS / GRANT boilerplate

Write path: whitelist the field in the admin order edit payload (don't spread formData raw)

Middle layer: if there's an RPC jsonb_build_object or a normalizeOrder whitelist, add it there too, otherwise you get "value in DB, null in frontend"

Presentation: orderPresentation.js SSOT, plus the admin order list…

Those four points are what that skill would have produced. GRANT boilerplate, no raw form spreading, the middle-layer whitelist that silently nulls your field, the five order surfaces that drift apart. It even labelled its own confidence level.

It never invoked the Skill tool. It read the description and reasoned from it.

So my 55.6% measures invocation, not effect. Those are separable in the trace, and I had assumed they were the same thing.

A flaw in my test design, while I'm being honest

All three traces show the model stuck in the same place: the working directory is empty, so it burns five Glob calls hunting for project files and ends with "please tell me where the project is."

That's my fault. Every eval run starts in an empty workspace, and the tool supports seeding one with a scaffold script. I skipped it. An empty directory pushes the model's attention toward "find the files" instead of "is this an architectural change?"

So the honest framing is: these numbers measure whether the model formally invokes a skill in an empty workspace. That's some distance from the question I care about.

The 0.00 that never ran

One more trap, because it will fool anyone who doesn't read the surrounding columns.

My first attempt at proving the grader could actually fail produced this:

run 1/1 [with]: score 0.00  $0.00  error: exit 1: Credit balance is too low
  ✗ skill-fired: Skill called 0x (expected 1..∞)

I wanted a red light, and it handed me a red light.

But that 0.00 isn't "the grader caught something." The run never happened. API credit ran out, the session never started, the grader searched an empty transcript, found no skill invocation, and rendered 0.

(The credit thing turned out to be a separate wallet from my subscription — eval uses whatever credentials your normal sessions use, and ANTHROPIC_API_KEY in my environment was routing it to the API account. Unsetting it for the child process fell back to subscription auth.)

The real one, once it ran:

run 1/1 [with]:    score 1.00  $0.30
run 1/1 [without]: score 0.00  $0.15   ← this 0.00 is real
Δ +1.00

Two identical-looking zeros. The discriminator is the columns next to them: the fake one cost $0.00 and carried an error string; the real one cost $0.15 with a null error.

Why would Anthropic make this a first-party command?

This is the part I find most worth thinking about, and I can only offer speculation.

They could have published a "how to write good skill descriptions" guide for a fraction of the cost. Instead they built a measuring instrument — one whose default behaviour is to prove your work contributed nothing.

Three signals I can read from that:

One: this isn't a documentation problem. If unclear descriptions were the issue, a style guide would fix it. Building a measurement tool implies they think this needs per-case empirical testing, and that results shift with model versions. The docs say to pin your model in CI so "a model rollout isn't mistaken for a plugin regression." That sentence concedes the same skill performs differently across models.

Two: the ablation arm is the default, not a flag. That's the strongest signal in the design. It assumes your plugin might not be contributing and computes the answer whether you asked or not.

Three: this is a tool that only becomes necessary at a certain ecosystem size. With five skills you don't need to measure. Past some threshold the question shifts from "do I have a skill for this?" to "will it win the attention contest?" I have 89. That number didn't exist two years ago.

My guess is they saw this curve before I did. That's a guess — I have no inside information.

What I know and what I don't

Known:

Same model, same prompt: going from 1 skill to 89 drops formal invocation from 100% to 55.6%
In the runs that didn't invoke, the model didn't pick a different skill — it read descriptions and reasoned from them
The grader has discriminating power (Δ = +1.00 against the no-plugin arm, so it does go red)
Total spend: $8.79, about 15 minutes

Not known:

Whether "smarter models need fewer skills" holds. Every run used the same model. That variable was never manipulated. Testing it needs two models of different strength, same suite.
How much effect is lost when invocation is lost. The traces show descriptions still reaching the reasoning; how far "reached the reasoning" sits from "executed the skill's full checklist" is unmeasured.
What these numbers become once the workspace is properly seeded.

If you've got a pile of skills, prompts or rule files, my suggestion isn't to prune. It's to measure once — and when you do, remember that "was the tool invoked" is the easy grader to write, and probably not the question you care about.

As for whether I should cut my 89 down: I have no evidence either way yet. Next step is seeding the workspace and running two models. That's when I get to have an opinion.

Related, from the same toolchain: Why does the AI always leave a few blocks out? — the other half of this problem, where the verifier checks whether your declarations are internally consistent and never whether they match the world. And compiling AI reasoning into scripts (in Chinese), on paying for frontier reasoning once and replaying it for free.

Originally published on my blog: I ran claude plugin eval on 89 skills. The trigger rate fell from 100% to 55.6%.

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

Google crawled 270 of my articles and indexed none — the body it received was 0 characters

Dexterlung — Sun, 27 Sep 2026 00:22:02 +0000

Read on: the acceptance check that proved 1 = 1 · 繁體中文版

I assumed my content wasn't good enough. Google had never read my content. What it received inside <div id="app"> was zero characters.

The numbers that started it

On 13 August 2026 I opened Google Search Console and exported two reports, 427 rows. Three lines stood out:

Status	Rows
Crawled – currently not indexed	270
Soft 404	157
Duplicate, Google chose a different canonical	95

"Crawled – currently not indexed" has an unambiguous meaning: Google came, looked, and declined.

That is far worse than not being crawled. Not crawled means it hasn't arrived yet. Crawled and declined means it read the page and judged there was nothing worth indexing.

My first instincts were keywords, depth, backlinks. All three were the wrong direction, and all three would have cost me another three months of writing articles nobody could find.

There was only one correct first move: stop guessing what it sees and go be it.

First move: fetch your own page as Googlebot

I requested my own article URL with Googlebot's User-Agent, saved the raw HTML, and counted two things:

characters inside <div id="app"> → 0
JSON-LD blocks → 0

Repeated it with GPTBot. Identical. So besides Google, the AI-search side was blind too.

The cause is unsurprising to anyone who has shipped a single-page app: the first HTML the server sends is a shell, and the body appears only after JavaScript runs. Google says it executes JavaScript, which is true — but "will execute" and "will always wait for you to finish before scoring" are different promises.

The part I want to emphasise isn't the conclusion. It's how long it took me to run that command. That HTML was one curl away the entire time. I wrote 270 articles before I fetched it once.

Fix 1: put the body in the first-hand HTML

My site already had a Netlify edge function doing og-tag injection, which means it already reads the article from the database for social previews. So body injection needed no new architecture — three more things on the same trip:

render the article body to semantic HTML and place it inside <div id="app">
emit JSON-LD (BlogPosting)
emit hreflang, and only when the other language version is actually published

The human path is untouched; the branch is bot-only, so real readers pay nothing.

One technical decision deserves its own paragraph: I did not reuse the frontend's markdown renderer. It sanitises with DOMPurify, DOMPurify needs a DOM, and the Deno edge runtime has none. So I wrote an escape-first renderer on the edge side: escape every character, then apply structure. That ordering makes XSS structurally impossible rather than "unlikely because I was careful."

The cost lands on raw-HTML blocks, which degrade to plain text. I didn't estimate whether that mattered — I asked the database. Of 1,112 text blocks in published articles, 1,111 use markdown and 1 uses raw HTML.

I'm pointing at that because it is the most useful discipline I picked up this year: when you are about to make a judgement about "all of the X", derive the scope from a query, not from memory. Had I written "most of them are probably markdown" I would likely have been right — and I would not have known the 1 existed. (The same habit, learned the hard way, is what fixed a privilege sweep that reported zero residue while missing seven views.)

Verification: 28 unit tests, 12 cross-layer contract tests, all green. Green isn't the evidence, though — I also built 9 deliberately broken variants and confirmed each turned exactly the check it should red, and that restoring made everything green again. 9/9.

One self-inflicted trap worth half an hour of your life: I used a sentinel character to mark inline-code boundaries, and the first version used a literal NUL byte. Git immediately classified the file as binary (Bin 0 -> 21694 bytes) — no diff, no line-by-line review. An escape sequence carries identical semantics and stays readable.

Fix 2: 157 soft 404s, because the site answered 200 to everything

The second number is worse, because it grows by itself.

I picked two URLs out of the GSC export and requested them by hand:

/product/panama-esmerlada-geisha-washed/  → 200 + homepage HTML
/product-tag/washed-processed/            → 200 + homepage HTML

Both are dead WordPress-era URLs. But the last line of my _redirects was:

/*  /index.html  200

Every SPA deployment guide tells you to add that line — without it, refreshing any sub-path 404s. Its side effect: every dead URL is also caught by it, and answered 200.

Google receives a 200 and something that looks like a homepage, and files it as a soft 404. Every mistyped or mis-pasted link manufactures another one.

Two layers of fix:

edge function: for /content/:slug, /product/:id, /works/:slug, when the record definitively does not exist, return status 404. The body is still the SPA, so the visitor still sees the in-site "page not found" screen; only the status code becomes honest.
_redirects: legacy WooCommerce/WordPress archive paths return 410 Gone. Not a 301 to /shop — in Google's eyes that is still a soft 404, just reported somewhere else. Seven legacy .md URLs get a 301 stripping the extension.

The load-bearing part isn't the 404, it's fail-open

This is the easiest thing in the batch to get wrong, and getting it wrong produces no visible symptom.

My data helper fetchRows returned an empty array for both "query failed" and "zero rows found". While it only did injection, that was harmless — both cases meant "don't inject."

The moment "no record → return 404" existed, the same code meant something else entirely:

database hiccups for 3 seconds → my real article is declared nonexistent → that 404 lands in Google's index.

The fix is to return { ok, rows } and treat only ok-and-zero as genuinely missing.

/product/:id needed one more layer: decide locally whether the string even looks like a uuid. A non-uuid sent to PostgREST returns 400, and 400 is "query failed", which fail-open lets through. The 82 legacy WooCommerce permalinks I needed to block are all non-uuid. They must be judged locally, not shipped out as a question.

Four mutations for this piece: remove fail-open, remove the uuid check, drop a header cleanup in the 404 branch, revert fetchRows to a bare array.

The third one did not go red the first time.

My assertion sliced the file contents to the end, so it matched a headers.delete further down in the injection path rather than the one in the 404 branch. Classic "the substring had another source" — the test appears to check one thing and is being fed by somewhere else. Slicing to just the 404 branch made it 4/4.

I did not think that defect up. The mutation exposed it.

Fix 3: one stylesheet costing 2,222 ms

Indexing handled, on to speed. Local Lighthouse against one of my article pages:

Render-blocking resource	Delay	Size
`fonts.googleapis.com/css2?Noto+Sans+TC`	2,222 ms	98 KiB
`assets/index-*.css`	341 ms	26 KiB
`assets/vendor-formkit-*.css`	—	6 KiB

One Google Fonts stylesheet, 2,222 ms on its own. Of those 98 KiB, 92 KiB are unicode-range subset declarations I never use (CJK fonts split into dozens of subsets, one declaration each).

I thought I had handled this, because the HTML contains <link rel="preload" as="style">. But the load-bearing line is the rel="stylesheet" right beneath it — preload only makes the browser fetch earlier; it does not take the stylesheet off the critical rendering path. With both present, the slow one is still slow.

The fix swaps rel to stylesheet on load. No flash of invisible text, because the font URL already carries display=swap and the splash screen uses system fonts. JavaScript-disabled clients get the ordinary tag inside <noscript>.

And afterwards I went back and confirmed all three injection anchors from Fix 1 were still intact. Touching fonts shouldn't affect body injection — but "shouldn't" and "confirmed" are different words.

Fix 4: a 531K PDF bundle preloaded on every page

The live index.html had a modulepreload line pointing at vendor-pdf-*.js, 531K.

The three pages that use PDF (poster studio, sticker generator, cupping station) are all lazy routes. Their dependencies have no business in the first screen.

Reading the build output made it clear: every page chunk in dist — including "page not found" and the newsletter page — carried a bare import"./vendor-pdf-*.js", and that chunk exports three identifiers nobody binds. So this wasn't "somebody needs pdf." It was my manual chunking rule, matching on strings, fighting Rollup's own splitting.

Fix: return undefined for those three packages and hand the decision back to Rollup.

⚠️ The first version just deleted the line, and made things worse: the return 'vendor' fallback below caught them and moved them into the core vendor bundle, which loads earlier. Core vendor went 499K → 1,057K and precache 2,398 → 2,930 KiB. You have to return undefined explicitly, before the fallback.

That was caught by verifying the build output, not by thinking.

⚠️ The first verification script was also wrong. It asserted "the pdf chunk must have no static importer", which flagged two legitimate cases — a lazy route's chunk statically importing a shared chunk is normal, because that whole subtree only loads on navigation. The real criterion is whether the transitive static-import closure from the entry point reaches it. Computing the closure fixed it.

A guard that barks at legitimate change is as bad as one that never barks.

Final numbers:

	Before	After
modulepreload	6 (incl. 531K vendor-pdf)	5
core vendor	499K	419K
precache	2,398 KiB	2,319 KiB
first-screen static closure	—	6 chunks / 1,461K, pdf not among them

Then I shipped a new defect: the title, three times

With four layers done, I thought the batch was finished.

The next day I looked at a real screenshot of the live article page. It opened with:

title → date → summary → title (as a figure caption) → title (h1) → summary

Title three times, summary twice. Shipped by my own previous commit.

The cause is simple and unflattering: I designed a layout without reading the frontend component first. Its <article> contains exactly one line, a <BlockRenderer> — the title, hero image and summary are already inside the blocks. My injector then added an h1, a date, a summary and a hero image with a caption. All four duplicates.

The fix is to render the blocks and nothing else — faithfully mirroring the existing renderer.

The test was an accomplice

I had a test asserting "the output contains exactly one h1". It was green throughout, while production grew three.

The reason was the fixture: its blocks happened to contain no title block. So the assertion held forever, over data that didn't resemble reality.

When the fixture doesn't look like the real thing, the test will vouch for the defect.

I gave the fixture a hero block (the real shape) and added a regression test specifically for duplication. Incidentally, my mutation tool caught its own stale config in that round — the string it was supposed to mutate no longer existed, so it reported "this proof is invalid" instead of counting it as a pass. That is what a guard should do.

Final acceptance: 34 unit tests, prove-red 11/11.

Two sentences carry this section:

An injection layer's job is to mirror the existing renderer faithfully, not to redesign the layout.

"Measuring" is not "looking." No Lighthouse score will ever show you a title printed three times. A screenshot showed it in three seconds.

A footnote: a byline can't be a shell

In the JSON-LD I changed the author type from Organization to Person to feed E-E-A-T person signals.

But I left the brand name in the name field. Google now saw "a person named after a brand." Right type, wrong load-bearing field.

I changed it to my real name and added an assertion that can actually fail: author.name !== SITE_NAME. Before that, the test only checked @type === 'Person' — which stays green if you put the brand name back, and that is exactly the defect this change existed to fix.

If you run an SPA content site, check in this order

Twenty minutes, roughly:

Fetch one of your own article pages with Googlebot's User-Agent and count characters inside <div id="app">. Cheapest cut, and the origin of everything else.
Export the soft 404 list from GSC and request two of them by hand. If your _redirects has /* → /index.html 200, you already know the answer.
Run Lighthouse and read only the render-blocking section. On a CJK site the top entry is very likely a font.
Open the live index.html and read the modulepreload lines. Ask each one whether the first screen truly needs it.
When you're done, look at a real screenshot. Not the score. The picture.

The last thing I'll leave here: what actually saved me in this batch was not any individual fix — it was prove-red, deliberately breaking the code and watching whether the check turned red. It caught three false green lights in this batch alone (an assertion matching a string from elsewhere, a guard flagging legitimate cases, a fixture that didn't look real), and not one of those was avoidable by being more careful.

Where the numbers come from

GSC exports, 2026-08-13, 427 rows: crawled–not-indexed 270 / soft 404 157 / duplicate 95
Live fetches as Googlebot and GPTBot: <div id="app"> 0 characters, JSON-LD 0
Local Lighthouse 13, target /content/the-mechanism-i-never-checked
Build output: modulepreload 6→5, core vendor 499K→419K, precache 2,398→2,319 KiB
Prove-red totals: body injection 9/9, 404 honesty 4/4, layout duplication 11/11

Originally published on my blog: Google crawled 270 of my articles and indexed none — the body it received was 0 characters

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

RLS says yes and Postgres still says permission denied: the 403 family I only understood on the second one

Dexterlung — Sat, 26 Sep 2026 05:05:10 +0000

RLS says yes and Postgres still says `permission denied`: the 403 family I only understood on the second one

Read on: the layer of privileges RLS cannot reach at all · 繁體中文版

An admin tops up a customer's balance in the back office and the screen returns permission denied for table users. My RLS policy explicitly allows admins. The door is open and I can't get through — because the thing blocking me was not the door I was looking at.

The symptom: the admin is locked out of his own system

My coffee shop runs on stored value: customers top up, then draw down. Besides customers topping up themselves, an admin can top up or deduct on a customer's behalf from the POS.

Testing the admin path, I got:

permission denied for table users

First reaction: impossible. My RLS policy has an is_admin() clause. The admin's identity check passes. The door is open — why can't I walk through?

I spent a while fiddling with the policy. Nothing helped. I was looking at the wrong door.

The cause: RLS governs which rows, GRANT governs which columns

The sentence that unlocked it: RLS allowing you is not the same as you being allowed to write.

Postgres permissions are two independent gates:

Table / column level GRANT — may this role touch this table, these columns, at all?
RLS policy — given that it may, which rows may it touch?

I had been staring at gate 2. Gate 1 was the one stopping me.

My top-up RPC was SECURITY INVOKER, meaning it runs as the caller. In the frontend the admin is the authenticated role. And the privileged columns on users — balance, wallet, member_level, the tier_* fields, annual_spent — were deliberately never granted to authenticated. I had restricted them to service_role and a few DEFINER functions precisely so that no ordinary logged-in user could ever write their own balance or tier.

So the chain was: admin → authenticated → INVOKER RPC → the RPC writes users.balance still as authenticated → no column grant → permission denied.

is_admin() cleared the admin at the row level. Execution never got that far; it died one layer below, at the column level. The error message says permission denied for table users and tells you nothing about which of the two gates rejected you. That is the worst part of it.

The fix: a DEFINER wrapper with the guard on the first line

What I actually needed: admins can write these columns, ordinary users cannot.

Granting the privileged columns to authenticated is the wrong fix — that opens balance writes to every logged-in user. I wanted "a call that has passed an admin check temporarily gains the ability to write privileged columns."

Postgres's tool for that is SECURITY DEFINER: the function runs as its owner, not the caller. My DEFINER functions are owned by postgres, which can write anything.

So, wrap:

create function fn_admin_topup_balance(...)
returns ...
language plpgsql
security definer          -- runs as owner (postgres), can write privileged columns
as $$
begin
  if not is_admin() then                        -- guard FIRST
    raise exception 'not authorized';
  end if;
  perform fn_topup_balance(...);                -- call the original INVOKER function
end;
$$;

revoke all on function fn_admin_topup_balance from anon;
grant execute on function fn_admin_topup_balance to authenticated;

The ordering is the whole design. is_admin() must be the first statement. A DEFINER function is superuser-powered from the moment control enters its body, so the guard has to run before anything touches data. With that in place, authenticated may execute the function, while the body admits only admins. The frontend's admin call site points at the new function; the customer self-service path and the service_role path are untouched.

I fixed two this way: fn_admin_topup_balance and fn_admin_deduct_balance.

The turn: fixing the second one told me it was a family

I fixed top-up. The same afternoon I hit the identical 403 on deduction. Identical fix.

That is when I stopped and thought about something else: if two identical bugs surface on the same day, it isn't two bugs, it's a family. There must be other INVOKER functions writing privileged users columns as authenticated, sitting there untriggered because nobody has walked that path yet.

Memory won't enumerate that, and neither will grep. I wrote a scanner that asks the catalogue directly — pg_proc — with three conditions:

prosecdef = false (INVOKER, not DEFINER)
the function source writes one of the privileged users columns
authenticated has EXECUTE on it

Four hits. Two were the ones I had just fixed (into the allowlist, with the reason recorded: "guarded by fn_admin_*"). The other two were cron maintenance functions: fn_expire_bonuses and fn_reconcile_balances.

Not every member of a family gets the same fix

Those two cron functions did not get the DEFINER + is_admin() treatment, because they should never be called from the frontend at all. They are scheduled tasks; the scheduler runs as superuser and needs no authenticated privileges. The one frontend call site I found for expireBonuses() turned out to be dead code with no callers.

So their fix is the opposite direction: REVOKE from authenticated, leave service_role. Narrow the grant instead of adding a gate. After deploying I queried the live ACLs to confirm only postgres and service_role can reach them.

That is the part worth underlining: a bug family does not imply a single remedy. What admins legitimately need gets a guarded privilege escalation. What nothing outside should ever touch gets its privileges taken away. The scanner's job is finding every member; deciding the remedy is still one judgement per member.

Three things to take away

On Supabase/Postgres, when RLS allows it and you still get permission denied, check the column-level GRANT before you touch the policy. They are two independent gates and the error message will not tell you which one rejected you.
When you need "admins can write this, users can't", wrap it in SECURITY DEFINER with is_admin() as the first statement. Inside a DEFINER body you are superuser — no unauthorised call may reach the line that touches data.
The second identical bug is the signal to write a scanner. Querying a system catalogue like pg_proc converts "are there others?" from an anxiety into a question with an answer. Allowlist the known-good with a written reason; whatever remains is the real work.

I still run that scanner as part of my health check. The next time somebody — including me — writes an INVOKER function that quietly writes a balance, it raises its hand before the deploy instead of after the 403.

Originally published on my blog: RLS says yes and Postgres still says permission denied: the 403 family I only understood on the second one

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

My most important test had never failed — because what it proved was 1 = 1

Dexterlung — Tue, 22 Sep 2026 13:05:14 +0000

Read on: measuring the noise floor before trusting a delta · 繁體中文版

Do this first — it is uncomfortable and it takes five minutes

Pick the test or check script you have the most confidence in. Then break the thing it claims to protect, on purpose.

Not something adjacent. The actual load-bearing thing: make the function return null forever, push the threshold somewhere unreachable, empty the data source it reads.

If it stays green, what you own is not an acceptance check. It is a decoration — and it is worse than having nothing, because it is why you stopped worrying.

The setup

I keep a notebook of production potholes — a year and a bit of markdown files recording what broke and how I fixed it. The problem with a notebook is that it only helps when somebody thinks to open it, and the most expensive class of mistake is defined by nobody feeling any need to check: the wrong road is smooth the whole way down.

So I built a tool. Give it a task description, it surfaces the relevant potholes.

Its most important feature is the red-flag list — marking which entries are the ones that produce no signal at the moment you get them wrong. I wrote an acceptance check for it and called it, in the commit message, "the most important check in this tool":

The notebook declares 48 red flags → the tool must be able to display 48.

Reasonable, right? 48 against 48. If the numbers match, no red flag got dropped.

Then I handed it to a different model

I asked a model from a different lab to review it. (Different lab matters. Models from the same family share blind spots; three of them looking together is still one pair of eyes.)

The first thing it did was not read my code. It ran a mutation:

make the fuzzy-match function return null unconditionally
→ selftest still 14 passed, 0 failed
→ the reconciliation still reads "48 / 48 ✅"

I re-ran it myself. Reproduced exactly.

Why it could never go red

Unfolded, it's obvious:

// numerator: how many rows are flagged red in the notebook
const declared = rows.filter(r => r.flag === '🔴').length

// denominator: how many the tool can actually display
let shown = 0
for (const file of files) shown += computeRedFlags(file).length

And computeRedFlags is:

for (const row of rows) {
  if (row.flag !== '🔴') continue
  const hit = findSection(row)      // found → carries a line number
  allRed.push({ ...row, line: hit ? hit.line : null })   // pushed either way
}

It pushes every row unconditionally. Found gets a line number, not-found gets null — but both count as "can display."

So shown and declared are two spellings of the same filter. They are mathematically incapable of differing. I had not written an acceptance check. I had written 1 === 1.

And the failure mode is at its most toxic here: it was always green, and green is what stopped me looking.

I made the same shape of mistake three times in one day

First: the tautology above.

Second: I rewrote it as an end-to-end assertion — run a real query, check the output string contains the red flag and its line number:

say(out.includes('TRUNCATE') && out.includes(':257'))

Ran the mutation. Still green.

Because :257 was arriving from a different line of the output. My output has two sections: the red-flag list (affected by fuzzy matching) and "closest entries by literal match" (unaffected). With fuzzy matching broken, the red-flag line did lose its line number — and the section below printed :257 anyway.

I was asserting on "does this whole blob of output contain this substring", and the substring had two possible sources.

Third: while fixing the counting logic I found a bug (a last-wins map dropping two entries), fixed it in the main code — and then made the identical mistake inside the test itself.

What the working version looks like

Three things.

1. Bind the assertion to the load-bearing line, not to the whole output.

const redLine = out.split(/\r?\n/)
  .find(l => l.trimStart().startsWith('·') && l.includes(needle))
const ok = !!redLine && /`:\d+`/.test(redLine)

2. Do not bind to something that legitimately changes.

My first version asserted the line number equals 257. Then I added a few lines to the top of that file, it became 261, and the test went red.

That was not catching a defect. That was an assertion bound to something allowed to move. A guard that barks at legitimate change is as bad as one that never barks — both train you to ignore it.

Assert that there is a line number, not that it is a particular one. That still survives the mutation: fuzzy match broken → that row prints a cross-entry label → no line number → red.

3. Every check must name the mutation that turns it red.

I ended up labelling each assertion with the mutation it guards:

What I broke on purpose	Which check goes red
fuzzy match returns null forever	end-to-end #3 (16 pass / 1 fail)
put the `if (!x.length) continue` back	the P0 check (16 pass / 1 fail)
push the first-stage threshold to 9999	13 failures
delete the plain-language line in the file header	real-corpus 7/7 drops to 2/7 (19 pass / 5 fail)
change nothing	24 pass / 0 fail

A check whose red-making mutation you cannot name is a candidate tautology.

One more trap: a prove-it-goes-red run must control its input

Later the same day I wrote a sentinel monitoring whether the tool's zero-hit rate had gotten too high. To prove it fires, I appended 16 fake zero-hit records to the real log.

It did not go red.

The file already held 241 records from my own testing. The ratio moved from 46.9% to 49.0% — nowhere near the 70% threshold.

A prove-red stacked on real data gets diluted by the existing denominator. The right move is to lift the judgement into a pure function and feed it fully controlled input:

const mk = (n, zeros) => Array.from({length: n}, (_, i) => ({ hits: i < zeros ? 0 : 2 }))
// ★ red: 13 of 16 at zero hits (81%) → must exit 1
// ★ boundary: exactly 70% does not count as over → exit 0; 75% → exit 1

Why having the rule written down did not save me

The part that bothers me most: my project documentation already contained this rule, verbatim:

A thing that emits a green light must positively observe the load-bearing thing itself.

That day I quoted it, believed I had followed it, and then got the same shape wrong three times.

Written down, correctly placed, remembered — and still no help.

What did help was: a model from a different lab, breaking my code with mutation, and watching whether my check made a sound.

The lesson isn't "be more careful." It is:

"I feel confident about this check" carries no evidential weight whatsoever. The only thing that does is "I broke what it protects and it went red."

The one-minute version

Pick the check you trust most
Break the thing it claims to protect, on purpose
Still green → it is a decoration, fix it now
Then label each check with the mutation that makes it red
Bind assertions to the load-bearing line — not the whole output, not something allowed to change
Run prove-red on fully controlled input, never stacked on real data

By the way — after step 3 you may find more than one. I found three.

Originally published on my blog: My most important test had never failed — because what it proved was 1 = 1

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

31 articles vanished from my site. No error, no log — one hardcoded .limit(80).

Dexterlung — Sun, 20 Sep 2026 13:30:11 +0000

31 articles vanished from my site. No error, no log — one hardcoded `.limit(80)`.

Read on: the test that never failed because it proved 1 = 1 · 繁體中文版

I had not deleted a single article, and yet 31 of them were completely gone from the site. Not in the series cards, not reachable through the category filters, not findable in site search. No error. No log entry. No trace.

The symptom: the data is there, the people are gone

My coffee brand's site has a content section with a few years of accumulated writing — early coffee notes from 2020 through recent dev journals.

One day, tidying up, I noticed the earliest batch of coffee posts seemed to be missing from the front end. The series card count was short. Clicking into the category showed nothing. Searching a keyword I knew was in the title returned nothing.

I was quite sure I had never deleted them. I queried the database: all present, status published, nothing unusual.

The rows exist, the front end can't see them. That is a worse feeling than "the data was deleted", because there is nowhere to start. No error message to search for, no stack trace to follow. It isn't broken. It is quietly short.

The root cause: `.limit(80)` truncates in silence

I opened the query in my list page component and found this line:

.limit(80)

Written long ago, when I had maybe sixty articles. I counted the real number of published posts that day: 111.

There it is. The frontend fetches 80, and the other 31 — mostly the 2020–2023 coffee posts, because of the sort order — were never fetched at all. They weren't "fetched but not rendered." They were absent from the frontend's data from the first millisecond. Series counts, category bucketing, and site search were all operating correctly over a corpus that was missing a chunk.

The reason .limit(80) is so nasty is that it does not error. It isn't "too much data, give up, throw." It is "here are the first 80, pretend the rest don't exist." That is silent truncation, and it belongs to the most dangerous bug family there is — every component looks like it is working, because every component is working. The input was already wrong.

Stopgap: change 80 to 200 and move on, for now

Having found the cause, I did not immediately do the correct, larger fix. I stopped the bleeding:

.limit(200)

One line, thirty seconds, 31 articles back.

I knew 200 is not a cure — today it is 111, and someday it is 201 and the identical bug returns identically. But stopping the bleeding and curing the disease are two different jobs: 31 articles were invisible to real visitors right now, and the priority in that moment was getting them back, not shipping perfect architecture an hour later.

An expedient bump isn't a sin. The sin is bumping it and forgetting, which converts it into the next time bomb. So I queued the real fix at the same moment — and did it the same day.

The real fix: delete the premise that the frontend loads everything

Three changes.

1. A paged RPC on the server. A new fn_get_content_list_page taking limit / offset. The frontend loads the first 30 and appends as you scroll. Whether the total is 111 or 1,111, the frontend never tries to swallow it whole. While I was there, the same RPC took over full-text search (including matching legacy English slugs) and computed reading time in the database instead of in the browser.

2. Counts separated from the list. The old count was pages.value.filter(...).length — deriving "how many posts are in this category" from the articles already loaded into the frontend. That is two bugs in one: it couples load volume and count correctness to the same ceiling. Truncate the load and the count silently follows. I replaced it with an independent countRows query that asks the database directly, selecting two narrow columns and moving no content. "How many are there" and "which ones do I load" are two different questions and should never share a data source.

3. The list stopped carrying full article bodies. The old list query dragged the whole content jsonb for every article, so the index page was hauling the full text of every post on the site. Now it returns summary, cover image and reading time only. Payload per batch went from 348,954 bytes to 22,114 — about 94% less.

Verification: "they're back" is not a number

Fixing vanished data has a trap: it is very easy to glance at the page, see more articles, and call it done. I wanted the numbers to match exactly, not to "look fuller."

Two independent lines, crossed:

SQL counts — ask the database how many per category and per series.
Playwright on a real browser — boot the dev server, actually click the category buttons, actually scroll to trigger the next page, and count what is rendered.

Before the fix, the category buttons summed to 8 + 51 + 6 = 65, against a database total of 111. After: 23 + 70 + 13 = 106, plus 5 known cross-category exceptions — exactly 111, matching to the unit. Playwright also confirmed the "30 then 60 more" path produced no duplicates and no gaps.

Numbers matching is what let me say it was fixed.

Three things to take away

Any frontend .limit(N) over a list needs an answer to "what happens past N." Usually the answer is silent truncation: no error, just quietly less data. It is the hardest class to diagnose precisely because every part of the system looks healthy.
"How many" and "which ones" are different questions. Never use the length of the loaded array as a count. It goes wrong in lockstep with the load ceiling. Ask the database for COUNT, page the list, keep them apart.
After restoring missing data, reconcile per bucket against database counts. Exact match or it isn't fixed — "looks like more now" isn't a verification. If you can, drive the real load path with a browser rather than trusting a unit test that seeds its own array.

That .limit(80) sat in my code for years and only detonated when the article count quietly crossed 80. Which is the actual lesson: a hardcoded number never tells you it has expired. It just starts lying while you aren't looking.

Originally published on my blog: 31 articles vanished from my site. No error, no log — one hardcoded .limit(80).

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

Supabase had granted anon TRUNCATE on 96 tables. I never wrote that line, and every check I own was green.

Dexterlung — Sat, 19 Sep 2026 14:00:58 +0000

Read on: the blind spot a non-homologous model caught · 繁體中文版

Do this first, it takes fifteen seconds

Paste this into your Supabase SQL editor:

SELECT grantee, privilege_type, count(*) AS n
FROM information_schema.role_table_grants
WHERE table_schema = 'public'
  AND grantee IN ('anon', 'authenticated')
  AND privilege_type IN ('TRUNCATE', 'TRIGGER')
GROUP BY 1, 2 ORDER BY 1, 2;

If it comes back non-empty, welcome. Mine came back with anon holding TRUNCATE on 96 tables, authenticated on 109.

I had not written a single one of those grants.

How I found it

I was building a podcast feature and idly checked the privileges on my articles table — I only wanted to confirm anon could read published posts. What I saw in anon's list was INSERT, UPDATE, DELETE, TRUNCATE.

The first three didn't worry me much. They're governed by RLS, my policies are fail-closed, and anon has no write policy at all. Ugly, but unreachable.

TRUNCATE is a different animal. PostgreSQL's row-level security applies to exactly four verbs: SELECT, INSERT, UPDATE, DELETE. TRUNCATE is not one of them. It empties an entire table in one statement and it answers only to table-level privileges. The whole RLS wall I spent a year building sits above this one. TRIGGER is the same — whoever holds it can attach a trigger, which is a path to executing code as the table owner.

Where the grants came from

Supabase's public schema ships with a default ACL entry (pg_default_acl) that says: every table created in this schema from now on automatically grants arwdDxtm to anon and authenticated. The D in that string is TRUNCATE. The t is TRIGGER.

So nobody slipped. The platform made the decision before I wrote my first line of SQL. And revoking on existing tables isn't enough — the next table you create grows it back.

Let me be honest about severity before this reads like a scare piece: this is not "your database gets wiped tomorrow." PostgREST has no TRUNCATE verb and no create-trigger endpoint, so a public anon key can't reach it over the API. You need an actual database connection. What it is, is a hole in the foundation of your defence in depth: you believe the worst case is bounded by RLS, and two verbs are not governed by RLS at all.

Why every check I own was green

I'm not without defences. I have a pre-commit checker whose whole job is verifying that every migration creating a table also writes its three GRANT lines. It has a self-test. It has a blindness signal. It was green every single time.

It was green correctly. What it checks is what I wrote: it reads migration files as text and confirms the GRANT lines are present. And the entirety of this hole is what I never wrote. Reading your declarations cannot, in principle, see past their edge.

The sentence I ended up writing on the wall:

A checker reads your declarations. The nastiest defects live in what you never declared — platform defaults, upstream package behaviour, the difference between your laptop and production.

That's the same family as an antivirus reporting "no malware detected" while the file sat right there on the disk: the tool answered the question it was asked, and the question had a blind side.

The more embarrassing half: I made the same mistake again while fixing it

I wrote the repair migration the same day. A loop over every table in public, revoking TRUNCATE and TRIGGER from anon and authenticated, with a self-verification at the end: count the residue, roll back unless it's zero.

It ran. Verification reported 0 rows of residue. Green.

The next day I looked again with an independent query: seven views still held TRUNCATE.

The reason is small and horrible. My revoke loop scanned relkind IN ('r','p') — ordinary and partitioned tables, the list my head produces when it hears "table". My verification query joined pg_tables, a system view that also lists only ordinary tables. The fix missed views, and the verification was structurally incapable of seeing views. They shared one blind spot, so the verification could never falsify the fix.

Notice how fine the trap is: both the fix and the verification really did query the live database. "Go ask the real world" sounds like the cure, and it is not sufficient — when I asked the world, the scope of the question was still typed from memory. The query was real. The range was invented.

The three rules I took out of it

1. Derive scope from a query, never from memory. Before acting on "all the tables", ask the database what actually exists:

SELECT DISTINCT relkind FROM pg_class WHERE relnamespace = 'public'::regnamespace;

Generate the repair loop from that output. Don't type ('r','p') because that is what "table" feels like.

2. The acceptance query may not contain a single word the invariant doesn't contain. My invariant was "anon must hold no TRUNCATE." That sentence contains no concept of kind of relation — so the acceptance query had no business joining pg_tables at all. Asking information_schema.role_table_grants directly (it naturally includes views) is the correct answer. Every extra predicate in an acceptance query is an assumption smuggled in from memory.

3. Every "too little" check has a "too much" twin — ask for it while you're there. My green checker guarded against missing grants causing a 403 in the frontend. Its dual — is anybody holding too much? — had never been assigned to any checker in over a year. That pairing exists nearly everywhere: guard against missed sends and the twin is double sends; guard against running late and the twin is running early. Asking costs almost nothing at the moment you write the first one.

The fix, copy-pasteable

-- 1. Revoke on what exists (including views! check your own relkinds first)
DO $$
DECLARE r RECORD;
BEGIN
  FOR r IN
    SELECT c.relname FROM pg_class c
    WHERE c.relnamespace = 'public'::regnamespace
      AND c.relkind IN ('r', 'p', 'v', 'm')
  LOOP
    BEGIN
      EXECUTE format('REVOKE TRUNCATE, TRIGGER ON public.%I FROM anon, authenticated', r.relname);
    EXCEPTION WHEN OTHERS THEN NULL;  -- extension-owned objects you cannot touch
    END;
  END LOOP;
END $$;

-- 2. Stop new tables from growing it back
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public
  REVOKE TRUNCATE, TRIGGER ON TABLES FROM anon, authenticated;

-- 3. Acceptance: zero joins, zero extra predicates — exactly as wide as the invariant
SELECT grantee, privilege_type, count(*)
FROM information_schema.role_table_grants
WHERE table_schema = 'public'
  AND grantee IN ('anon', 'authenticated')
  AND privilege_type IN ('TRUNCATE', 'TRIGGER')
GROUP BY 1, 2;
-- expected: empty

Two footnotes. First, pg_default_acl usually holds two entries — one owned by postgres, one by supabase_admin. You cannot alter the second (I get permission denied), but in my project supabase_admin owns zero user objects, so that dirty ACL has never actually applied. Check yours with SELECT relname, pg_get_userbyid(relowner) FROM pg_class WHERE relnamespace='public'::regnamespace. Second, cleaning once is not staying clean — I turned the acceptance query into a scheduled check that runs before every deploy, because the only thing that catches a privilege which regrows is asking again on a timer.

What it cost me to learn

This hole existed in my project for over a year. During that year I wrote dozens of governance rules, a dozen checkers, thousands of tests. All green. All honest. All inspecting things I had written.

It was found because I glanced at something while building an unrelated feature. That is part of the lesson: for the class of defect that lives in what you never wrote, there is no checker you can build by reading your own source. Only two moments catch it — the moment you adopt a platform, when you enumerate the decisions it made on your behalf, and a scheduled question asked of the live system rather than of your files.

96 is 0 now.

Originally published on my blog: Supabase had granted anon TRUNCATE on 96 tables. I never wrote that line, and every check I own was green.

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

Two identical runs scored 89 and 89. Two cases had flipped.

Dexterlung — Fri, 11 Sep 2026 14:05:10 +0000

Read on: the A/B test this corrects · 繁體中文版

Two weeks ago I published an A/B test: I cut 41 AI tools' self-descriptions roughly in half, then ran a behavioural question bank against both versions to check that trigger rate hadn't dropped. Before: 88/96. After: 90/96.

I wrote this sentence about it:

±2 cases at this sample size is noise, so I am not claiming it got better.

A reader named Vinh Nguyen replied, and the reply amounted to one question: how do you know?

He was precise about it. Three votes per case, majority wins, so any case sitting near the 2-1 boundary can flip on sampling alone. There was a between-arm delta with no within-arm noise floor to measure it against. And "three up, one down" — exactly what my four movers looked like — is the shape churn produces.

He was right. That sentence was an assertion, not a measurement. It is also the same move the post spends two thousand words criticising: claiming a property of a system without observing it. I made it in the one paragraph where I was grading my own work.

So I ran his experiment. The part I didn't expect is that it cuts in two directions at once.

First I went looking for the answer I'd already paid for

Before spending anything, I checked whether I could bound the churn retroactively from the run I had already done. The harness makes three calls per case and takes the majority, so it necessarily computes "2 of 3" or "3 of 3" somewhere. Count the 2-of-3 cases and you have an upper bound on how many could flip — for free.

I couldn't. And the reason has a nastier shape than "it wasn't recorded."

The harness does compute the vote split, and it does print it — for cases that failed. A case that passed 2-1 prints nothing at all. The JSON output aggregates to per-skill pass/total and drops every split.

So the one shape that actually flips under sampling — the narrow pass — is precisely the shape no layer of that tool had ever shown me. The instrument was blind in the exact direction that produces reassurance. Failures got scrutiny; lucky wins got a checkmark.

I fixed that first, because it is the cheap half and it makes every future round free to interpret. Per-case vote counts now survive into the JSON, and the human-readable run ends with a count of non-unanimous cases. Then I checked the fix the way I should have checked the original claim: forced every case to 2-of-3 and confirmed the new line moved from "0 split" to "4 split." It reads the real votes. It isn't printing a constant.

The baseline

Same commit, same question bank, same model, same machine, same afternoon. The unchanged arm, twice, with nothing touched in between. 96 cases, 288 calls per run.

	Run 1	Run 2
Total passed	89 / 96	89 / 96
Cases that flipped between the two runs		2

Read those two rows together, because the gap between them is the whole point. The score is identical. Two cases flipped anyway — one up, one down, cancelling exactly.

If I had run the same arm twice and looked only at the total, I would have concluded the harness was perfectly stable. It isn't. It has a floor of about 2, and the total-score view is structurally incapable of showing it, because flips in opposite directions erase each other before they reach the number I was reading.

So my +2 sits exactly on the floor. The sentence in my post happened to land on the right answer for a reason I had not earned.

Where the noise actually lives

This is the part I'd want if I were reading someone else's post.

My question bank has three kinds of case per skill: should-trigger, should-NOT-trigger, and a boundary case. Splitting the two flips by kind:

Case kind	Count	Flipped
should-trigger	48	0
should-NOT-trigger	24	2
boundary	24	0

All of the variance is in the negatives. Zero flips in 48 positive cases, zero in 24 boundary cases, both flips in the 24 negatives.

In hindsight it's obvious. "This sentence is obviously asking for that tool" is a strongly determined decision. "This sentence should not reach for that tool" is a decision made against 116 competitors, and it's soft.

Which means my question bank spends 75% of its budget on the categories carrying none of the noise. If I want a future A/B to be readable, the fix is more negative cases, not more rounds. I would not have guessed that, and no amount of staring at the total would have suggested it.

The bad news, which is also the interesting news

I flagged one skill in the original post as "the one to watch" — the one I had compressed hardest, which lost a case.

It's still down. Both of today's runs put it at 3/4, and the case it loses comes back 0 of 3 — unanimous failure, twice, seventeen days later. And it is a positive case, which is the category where the noise produced zero flips all day.

To be sure that signature meant what I thought, I ran a deliberate control. I took six skills whose descriptions the original disciplined trim could only compress by under 20% — meaning they had almost no redundancy left — and force-cut each by roughly 64%, the same ratio as the one that broke. If over-trimming has a signature, this should produce it.

It did, loudly. Those six went from 23/24 and 22/24 down to 19/24, and one of them collapsed from 4/4 to 1/4. Three separate cases fell from 3/3 to 0/3 or 1/3, every one of them positive or boundary — none in the negatives where the noise lives.

So real damage and churn do not look alike once you can see the votes:

Churn is a negative case wobbling between 1-of-3 and 2-of-3.
Damage is a positive case collapsing to unanimous failure and staying there.

A total score cannot tell those apart. That's not a subtle limitation. In my case the aggregate said "no systematic decline," and it was true, and it was concealing a genuine regression, because one real loss and several noise-driven gains summed to approximately nothing.

The threshold question

The other commenter, Kane Lim, proposed that the 64% case suggested a practical threshold — a point where compression starts removing semantic signal rather than redundant text.

The control answers this, and sharpens it. The same 64% cut cost one case on the skill in the original post, and three cases on a skill in the control. The ratio was identical. The outcome was not.

The difference is what the 64% removed. The skill I could compress that far with discipline could be compressed that far because most of its 689 characters were restatement of a six-layer rule. On a description that is already tight, reaching 64% means cutting into signal — there is nothing else left to take.

Which inverts how I had been reading the number. Compression ratio isn't a risk driver. It's a measurement of how much redundancy was there to begin with — and I had been treating a symptom as a cause. The two variables aren't merely confounded, they're mechanically linked: how far a disciplined trim can go is the redundancy. Across my 24 skills the achievable ratio ran from 3% to 64%, and exactly one exceeded 49%.

And a correction I owe the original post. I wrote: "everything I cut by 30% held steady. The only one that moved was the one I cut by 64%." That's wrong on my own data. The four skills that moved were cut by 64%, 49%, 48%, and 17%. The 17% one moved by exactly as much as the 64% one. The table proving it sat two paragraphs above the sentence.

What I'd take from this

Run the same arm twice before you compare two arms. It cost about a dollar. It reclassified two of my conclusions.
A total score is a lossy summary, and it's lossy in a specific direction: opposite-direction flips cancel, so the aggregate systematically understates instability. Two runs scoring 89 and 89 hid two real flips.
Record the vote split per case, not just pass/fail. Anything that takes a majority is discarding the number that says how close it was. And if your tooling prints the split only for failures — mine did — then the flip-prone narrow pass is the one thing you can never see.
Sort your cases by kind and check where the variance actually sits. Mine was 100% in one category holding 25% of the budget.
A stable unanimous failure is readable at n=1. You don't always need a bigger sample; sometimes you need a less lossy instrument. 0-of-3 twice is a signal. 2-of-3 once is not.

What this doesn't establish

Two runs give a point estimate of the floor, not a distribution — I know the floor is about 2, not that it is 2. The over-cut control was a single arm, single round; its individual collapses are strong, but its total is subject to the same ±2. The six skills I over-cut were selected by a proxy — "the disciplined trim could only compress them under 20%" — which measures how tight a description is, not how narrow a skill's scope is, and scope was Kane's actual question. I also chose how to cut them, keeping the opening trigger sentence and dropping identifiers and symptom phrases; a different cut might land differently.

One last thing, since it's the same failure as everything above. When I went to report what these runs cost, I found the harness doesn't print its cost block in JSON mode — that code sits in the other branch. So the actual spend for the three runs was never recorded, and the roughly US$1.2 I quoted is extrapolated from a single-skill run. Third instance in one day of the same defect: a number that was computed, and then dropped by one output path.

That's the thing worth keeping. Not "measure your noise floor" — that's just good practice. It's that my instrument computed the answer to Vinh's question, printed it selectively in the direction that flattered me, and dropped it everywhere else. Nobody was going to find that from the outside. It took a stranger asking one question I couldn't answer.

本文原載於我的部落格：Two identical runs scored 89 and 89. Two cases had flipped.

Dead Code Is More Dangerous Than Broken Code — How a Non-Homologous AI Caught the Blind Spot the AI Itself Had Rubber-Stamped

Dexterlung — Wed, 09 Sep 2026 13:05:14 +0000

Dead Code Is More Dangerous Than Broken Code

The opening: an AI confidently told me to reuse a capability that didn't exist

Here's how it started.

I was adding a small feature to my own coffee e-commerce store: let a customer pick a fresh combination of beans over chat, price it on the spot, and order it. The chat-ordering bot was already built — I'd even tested balance deduction on a real device. All I needed was one more piece: something on the backend to compute the price, because the amount can't be whatever the phone says it is.

I handed the repo to my main AI (Opus, my daily driver) and asked whether the backend already had pricing logic I could reuse. It scanned the code and came back fast:

"The backend has pricingLogic.ts — the pricing logic lives there. Just reuse it."

Sounds reasonable. The file really does exist. Open it and there really is a pile of price-computing code inside. Anyone reading the code — me, the AI, anyone — lands on the same conclusion at first glance: the backend has pricing capability, just wire it up.

One problem. That code had not compiled a single time in six months.

From the day it was born (git blame says 2026-01-11) it was missing a closing } — the CartItem type literally couldn't close. The frontend test runner doesn't compile the backend's TypeScript, and the audit suite doesn't touch backend functions, so no gate had ever gone near it. It just sat there quietly, in the exact spot where "backend pricing logic" should live, for half a year. At one point another change even passed through it and added a line — the person making the change had no idea they were editing a file that doesn't compile.

The main AI wasn't bluffing. The world it read was real: the file exists, the code inside is real pricing code. It just had no way to tell, from the source tree, that this thing was dead.

My gut said stop: don't build on a foundation that drifts. Nail down the truth about pricing first — and, crucially, bring in a "non-homologous" model to diagnose it.

What that diagnosis dug up was far worse than I expected. It didn't just overturn the main AI's advice; it exposed a structural blind spot in my entire governance setup. That blind spot is what this post is about.

What "existential rot" is

When we say "bad code," we usually picture broken code: it computes wrong, throws errors, blows up mid-run. Broken code has one virtue — it makes noise. It crashes, someone reports it, it gets fixed.

But the four things this diagnosis surfaced don't crash. Not one. They're a different kind of rot, and I've started calling it existential rot: the problem isn't "it does the wrong thing," it's "it exists, does nothing, and lets everyone believe it's doing something."

Four faces of it:

1. Dead code (born broken, never compiled). That pricingLogic.ts. Missing a bracket, six months without compiling, zero production impact. It isn't "broken" — it never lived.

2. A deployed zombie (frozen on an old version). The backend has another pricing function that actually is deployed and live. The live version was v55, a snapshot from March. I sent it a real item with a price of 100 — it returned $0. The source had moved on several versions, but the live one is frozen in March, and wrong. Better yet: it has zero callers, so nobody will ever feel pain from it returning $0.

3. A phantom implementation (the Nth parallel copy). Hidden in the repo was a fourth pricing implementation — a fictional percentage-discount model. Exactly one place imports it: its own test file. All tests green. The green proves "it agrees with itself," not "it agrees with reality."

4. A defense with no reader (a warn-only log nobody reads). I'd added a warn-only amount-recomputation check on the database side that logs suspicious orders to a table. Sounds responsible. But that table had exactly two rows — both from my own smoke test — and nothing in the entire repo, no sentinel, no script, no human, reads that table. The defense is written, but there's no return path.

Put these four side by side and they form the entire truth of "backend pricing capability" in my system: one living body, three zombies.

The only thing actually computing money, start to finish, was one place — cartPriceCalculator.js, running in the customer's browser. The three backend copies were all decorations occupying the spot.

The real root cause: pricing authority actually lives in the browser

Why can "a pricing implementation with no consumer" survive this long?

I read the body of the order-creation stored procedure by hand, and the answer is ugly: it takes the front end's already-computed subtotal, total_amount, and unit_price verbatim. The payment side looks safe — it reads orders.total_amount from the database — but that value is the one the front end stuffed in.

In other words, in this architecture pricing authority de facto lives in the browser (client-authoritative computation). The server just copies down the number the phone computed and files it.

Once that's true, any backend pricing code becomes a structural decoration: wiring it in costs extra effort, and not wiring it in changes nothing. A decoration never gets traffic-tested — no real order passes through it, hits it, translates its error into "the customer paid too little," the kind of thing that hurts. So it can rot without detection and drift without limit.

Which is exactly why my initial gut was right: the "align the two copies" prescription the main AI wrote was a prescription to repair a corpse. Until you answer "who's going to call it, which version deploys, which copies get killed," aligning dead code to the living body just makes the corpse look more alive.

Why my governance saw none of this

Here's the part I actually wanted to write down.

My governance isn't weak. There are locked cross-layer traces, there are commit-blocking hard rules, there's a whole audit suite. The top pricing dark-zone had already been registered, trace-locked (I'll call it trace T-029), given 24 tests, all green, with the word locked sitting right there in the registry — back on 2026-06-06.

And then the darkest part of that dark zone kept rotting, untouched.

I eventually saw why. Almost all of my defenses are change-triggered — they only fire when you touch a registered node. A trace test runs when you edit a file on the trace; a blocker fires when you commit; an audit lights up red when you modify the thing it watches.

But the whole class of "existential rot" doesn't require anyone to touch anything to persist. The dead code lies there, the zombie stays frozen, the phantom chews its own tail, the reader-less log quietly accumulates — none of it trips a single "change detector."

The lamp T-029 lit was the front-end price-resolution convergence point. The "backend copy," the "deployed bundle," the "rule-loading layer" — three surfaces that were simply not in that lamp's beam. It got flagged, locked, 24 tests green — and the darkest part never got touched.

What I was missing wasn't "one more change-triggered rule." It was an existence constraint:

Any implementation that claims to compute money, to defend, or to be a single source of truth must have (a) a live consumer, and (b) a confrontation witness. Missing either one → retire it.

Side note: this also fits a gap I'd already written down — my system only adds features, it never removes them (things only go in, never out). And "has no witness" turns out to be an objective test for "should have been retired but wasn't."

Why it had to be a non-homologous model

Back to the key decision: why did I insist on a different model, instead of asking the same AI to check again?

Because the main AI wasn't short on ability. It was trapped by the same ready-made narrative.

"This file is named pricingLogic.ts, it has pricing code inside, therefore the backend has pricing capability" — that's a story too smooth, too reasonable; it fills your cognition. Whereas "does it even compile? does anyone call it? is that the version that's deployed?" is a backward-looking, declarative question that will not jump up and knock when you're about to reuse a file.

A homologous model colludes. Ask it to look again and it'll most likely re-read "the repo has this file" as "the capability exists" a second time, because it's running the same inference reflex as the first time. That's not stupidity — it's an instrument problem. The source tree is the instrument's testimony, not the fact of the world (the actual runtime behavior).

Switch to a non-homologous model (I used one from a different family) and the effect is like bringing in an outsider who never bought your narrative to review the code. The first thing it does is not trust the repo — it goes and confronts it:

It ran a real compiler over the dead code — instant red. Hard proof: "never compiled."
It sent real input to the live function — got back $0. Hard proof: "deployed zombie computes wrong."
It grepped for callers of those "canonical" functions on the production path — zero. Hard proof: "no consumer."
It read the order-creation procedure body by hand — saw it take the front-end price verbatim. Hard proof: "authority lives in the browser."

It even corrected the main AI's diagnosis name: this isn't two implementations "drifting" (drift = two living things gradually diverging). This is necrosis plus a split truth — a piece of dead code that never lived, a zombie frozen in March, zero callers. Different name → completely different prescription.

The value of non-homologous diagnosis, in one line: it's the vaccine against self-rubber-stamped green lights. When a system (including an AI) blesses itself, the most effective mirror isn't making it look at itself harder — it's introducing a confronter that doesn't collude.

Six green lights, not one of them watching the load-bearing thing

During the diagnosis I did something I think everyone with a test suite should do occasionally: pull out every "green light" and ask each one — what load-bearing thing did you actually, positively observe?

The result (all of this is re-checkable):

Green light	What it claims	What it actually saw	Verdict
Pricing tests all green	"Business rules are test-protected"	A fictional model with zero production callers chewing its own tail	False green
T-029 trace 24 green + `locked`	"Pricing SSOT is locked"	Only the front-end resolvers' fallback — not the compute core, not the backend, not rule-loading	Half true
Edge-function smoke passes	"The pricing function works"	HTTP 200 + `success:true` — returning $0 is also a 200	False green (checks the status code, not the amount)
Warn-only defense shipped	"The DB side has a recompute defense"	Two log rows, both self-inflicted, nobody reads it	Defense exists, no return path
Migration header "not applied this batch"	"Prod wasn't touched by this"	Prod was applied	Doc narrative decoupled from prod
Front end priced successfully	"The user sees the correct price"	On RPC failure it silently falls back and prices anyway	Silent degradation, no alert

Six lights, each making you feel "someone's minding this surface." But not one of them ever positively observed the one thing that's actually load-bearing: two runtimes computing the same number for the same cart.

Here's the line I took away and want to nail to the wall: a green light with no witness proves "nobody's looking," not "it's correct."

The fix: let the zombies die all the way first, then give the living body witnesses

Once the framing was overturned, the fix got clear. The principle was "move logic only, don't change a single price." Three steps:

Step 1: let the zombies die all the way. Both dead copies got deleted and retired. No aligning, no repairing — a zero-consumer corpse doesn't deserve a minute of your time.

Step 2: extract the one living body into a single core. I pulled the one algorithm that actually computes money out of the front end into a pure-function core (zero imports, computes and returns, touches nothing outside). The front end became "load rules → hand to the core." The backend function was rewritten to "resolve rules → hand to the same core." One logic, both runtimes delegating to it.

Step 3 — the real point this time — give it three witnesses. Sharing one import isn't enough; it only kills the "copy-paste" axis of drift. There's a nastier axis: deploy lag. The same source, front end shipping the latest, backend frozen in March — that's two truths. A shared import can't catch that. Only one thing can:

A parity witness: feed the same batch of inputs to the front-end path and the backend path, compare outputs row by row. I prepared six cart "battery" cases (cross-tier VIP, gift-box volume, tiered discount, buy-X-get-Y, etc.), froze the pre-refactor live output as a golden baseline, and any change that moves a single dollar goes red.
A live deploy-lag witness: hit the actually deployed version over HTTP and compare it against the local core, row by row. This is the only edge that can see "frozen on an old version." First run, it immediately spat out 3-of-3 mismatches against live v55 — proof the detector is alive and working.
A compile gate covering all runtimes: a blocker requiring that the pricing-chain backend function actually compiles. This is the direct counterfactual to "dead code nobody knew about for six months" — had this gate existed, that missing-bracket corpse would've gone red on day one.

One thing I deliberately did not do: I didn't heavily rewrite the order-creation procedure body. Because in this repo, "rewrite a gatekeeper function wholesale" has a rap sheet — it wipes out the gatekeeping logic buried inside. (Two prior migrations did exactly that, once wiping a gift-box capacity guard and once a stored-value balance guard, both dead for weeks.) Server-side authority gets promoted later, gradually, through a hook I'd already pre-embedded — without reopening that knife.

Turning one expensive insight into a free mechanism forever

This kind of diagnosis is expensive. A non-homologous model, copy-by-copy confrontation, climbing counterfactuals — that burns real tokens. So I have a habit: every piece of reasoning I've paid for once, I try to make cheap or free the next time.

Two reusable things crystallized out this time, sitting at two different moments:

1. A lens for the "about to build" moment — the consumer-witness lens. A check I built and use daily. Its core is a gate: before you trust, cite, or build on anything that "claims to compute money / to defend / to be the single truth," it forces you to write two lines — (a) which specific caller in production depends on it? (b) which specific test feeds it the same inputs and confronts it against the authority? If you can't write either line, treat it as nonexistent. This lens is the vaccine against that exact "fooled into the wrong framing by a corpse" moment, grown into a gate that knocks before every future reuse.

2. A checklist item for the "designing" moment — the server-authoritative rule. I maintain a cross-project front-end/back-end completeness checklist (past incidents distilled into check-sentences). This time I added one, with a machine-readable tag so tools can auto-apply it:

Load-bearing computed values (price, total, balance, stock — the "computed" kind) must be computed by the server, with the client only displaying; the server must never trust a client-computed load-bearing value verbatim.

This is a generalization of an old rule I already had ("no front-end multiplication for the unit-price field") — the old rule covers the pricing special case; this one covers "for every computed load-bearing value, who is allowed to compute it, and whom does the downstream trust." The machine-readable tag means: next time I decompose a new project through my tool-chain, this rule gets spat out into the spec automatically, without me remembering it. Pay once, carry it into every new project free.

But "spat out automatically" hides a trap I only fixed this time — and it's a vivid one, worth its own beat.

I'd crystallized that check-sentence library into a machine-readable config file — plainly, an index of "when I hit this sharp edge, here's how I handle it." It hangs on the tool-chain I use to decompose new projects: start a new project and the tool reads it and auto-applies my past scars into the spec. Problem: that config file is private (it lives in a toolkit repo of mine that isn't public), while my project bootstrapper pulls almost everything else down from public GitHub.

So you get a very classic tension: a bootstrapper that fetches everything from a public channel, meeting one file that can't travel a public channel. Worse — historically that config file had never been supplied automatically. It sat in a "manual step nobody actually ran." The result: every decomposition silently ate the neutral defaults, and this config I'd painstakingly crystallized contributed exactly zero.

That is literally the disease this whole post is about, growing back in a new spot: a thing that "should be supplied but isn't" quietly trips no alarm, letting every decomposition believe it applied the completeness rules when it didn't.

So I wired the supply in, carefully, every line guarding against "private content leaking into a public repo":

Never fetch it. Unlike the other tools, this config isn't downloaded from GitHub — it's copied at runtime, on my own machine, from a source already present locally (scan a few common folders for that private toolkit, or read a path from an env var). Zero hardcoded private paths in the code. A regular user has none of that private source, so it resolves to "not found," falls back to neutral defaults — nothing leaks.
Fence it with .gitignore. On supply it auto-adds that private config's folder to the project's .gitignore, with an iron-rule comment baked in — "never reaches a public repo." So even if you later fat-finger git add ., it can't reach any outward remote.
Fail loud on supply, never silent. If no source is found it says so out loud — "config not supplied, this run uses neutral defaults" — because a silent skip is exactly the bug this fixes.

I love this bit because it compresses the whole lesson into one concrete action: a bootstrapper that fetches everything from the public internet meets a private file that can't travel that public channel — so it switches to "smuggle it in locally + fence it with .gitignore so it can never leak back to a public repo." That "never reaches a public repo" iron-rule comment is what an existence constraint looks like once it's grown into code: "this thing should exist, and only here" written as a rule the program executes, not a line of doc nobody runs.

For the record, the referenceable deliverables from this round (things I use daily, fed by real incidents — not slideware): a check I call the consumer-witness lens (before reusing anything, force yourself to write down its two witnesses), a cross-project front-end/back-end completeness checklist (every incident distilled into a check-sentence), and an orphan audit script (hunts "zero-caller decorative anchors" — it found thirty-nine in one project alone).

Closing: this was never really about pricing

If you filed this under "a story about one pricing bug," you missed the point.

The two drifting pricing copies were just the most visible projection of the hole. The real hole is that the copy axis has no forcing function: the same "how money is computed / how state is judged" logic actually lives in five physical spots — front-end bundle, back-end bundle, the live DB function body, migration files, repo source — and nothing guarantees they equal each other. They will, and already do, tell different stories.

So what's actually portable is this:

Dead code is more dangerous than broken code. Broken code blows up and gets fixed; dead code lies quietly in the spot where "some capability should live," fooling every reader of the code — including the AI itself.
A defense with no reader is not a defense. A warn-only log, an alert nobody subscribes to, a dashboard nobody looks at — all worth zero.
A single SSOT module is not a single truth. A shared import only kills "copy-paste" drift; "deploy-lag" drift is visible only to a confrontation witness.
"The repo has two copies of this code" should not first read as "two implementations." First ask: does it compile? does anyone call it? which version is deployed? — very likely it's "one living body plus N zombies," not drift at all.
If your governance is all change-triggered, it's blind to existential rot. Add one existence constraint: every money-computing / defending implementation needs a live consumer plus a confrontation witness, or it gets retired.

That last decision — "don't build on a foundation that drifts; fix the truth first, and bring in a different model to look" — took under a minute. It was the highest-return step in the whole thing. Because an AI reading your code, however confident, is only giving you instrument testimony. Whether to build on it, you have to ask first: this capability — is it actually alive?

This is one entry in my one-person studio's dev-notes series. The consumer-witness lens, the completeness checklist, and the orphan audit are tools I built and use daily; some are already open-sourced. They aren't theory — every one has a pit like this one behind it, a pit I actually fell into.

本文原載於我的部落格：Dead Code Is More Dangerous Than Broken Code — How a Non-Homologous AI Caught the Blind Spot the AI Itself Had Rubber-Stamped

AI Auditing an AI's Own Toolchain: The Repair Tool That Overwrote Its Own Fix

Dexterlung — Tue, 08 Sep 2026 14:05:34 +0000

I have seven interdependent tool repos — some public, some private — each with its own audit checks, but nobody had ever checked what happens when they're wired together. This post is about handing all seven to Fable5 for a cross-repo audit, and the fact that not one of the real problems it found could have been caught by auditing any single repo alone.

Read on: Previous: the Confrontation-Point Law · Don't ask a model to narrate itself · 繁體中文版

The audit prompt itself went through three drafts

I didn't just paste in "please check these seven repos." The prompt started as a skeleton, then got execution steps, then got filled in with the real seven repos — 5 public tool-chain repos plus 2 private ones (an internal connective framework and a cross-project reuse library). The goal was to write context and boundaries, not a checklist — tell Fable what the terrain is and where the hard lines are, and make verification about mechanism, not about handing over an answer.

Each repo got its own scanning subagent, plus one independent verifier subagent that never saw the scanners' conclusions. That independent verifier is the part that mattered — it overturned four of the scanning agents' conclusions. The topology itself turned out fine; the two public/private seams both worked correctly. The real problems weren't in any single repo. They were three composition-level failures that only become visible once you look at all seven together.

The tool that fixes drift can silently overwrite your work

The most interesting one: one tool's job is to detect and repair drift between a local skill copy and its canonical source — if your local copy diverges from the original, it syncs you back up. The problem: in one environment, that "local copy" is actually a Windows junction (a filesystem shortcut) pointing straight at the canonical repo — and the tool's scan logic only checked "is this a directory," never "is this actually a shortcut."

The result: a tool built to repair drift will write straight through the shortcut into the canonical repo, potentially overwriting uncommitted work sitting on someone's feature branch. The repair tool itself becomes the risk. No single-repo audit could have caught this — the bug doesn't live in any code, it lives in how directories are wired together on a specific machine. That shape is only visible across repos.

The other two composite failures: an orphaned commit left a cross-repo contract stranded with nothing consuming it; and three places documented something as "prompt-level, optional" that was actually being treated as "code-level, mandatory for everyone" — a documentation/enforcement mismatch that leaves different maintainers with completely different mental models of the same rule.

One rule worth stealing: content-level probes, not existence checks

The audit converged on a principle I think is more valuable than any single finding: "the commit exists" or "the file exists" doesn't count as verified. You have to walk the entire propagation chain — local branch, remote default branch, the published artifact or install source, the installed copy — and grep an actual signature string at every single stage, and it has to actually hit, every time.

That principle is what caught the most important issues in this audit. Downgrade it to an existence check, and the audit still runs, still produces a report, and catches nothing — which is worse than not running it at all, because it hands you a false green light.

A real case: how a spec makes frontend and backend asymmetric from birth

The day after the cross-repo checklist converged, a real client project handed us a case that got folded directly into that checklist as a new criterion.

The setup: a real order-management admin feature with a visible frontend/backend asymmetry. The backend RPC defined an error code called BOOKING_NOT_FOUND that the frontend's error-message map never learned about. A refund action returned HTTP 501 in production with zero UI indication. A loading ref was declared and returned but never consumed by any of six admin tabs. An admin-readable transaction table existed and was never queried by the frontend at all.

We gathered the hard evidence first, then asked for the root cause, not just a symptom list. The diagnosis: the asymmetry wasn't born in any one layer — it was born upstream, in the spec's own output grammar. The original spec gave the backend a model (field-level data structures) and gave the frontend only a verb ("view orders and bookings"). Every downstream layer — decomposition, execution, scoring — is deliberately faithful by design, each explicitly instructed not to invent or over-check. So the upstream asymmetry got preserved intact, all the way to the final output. No layer dropped the ball. The pipeline worked exactly as designed — the design itself just had a blind seam nobody had seen.

Even the fix this diagnosis proposed doubted itself out loud: the suggestion was to add a mandatory "Surface Contract" section (a state row plus a seam row) to the five-element goal format. But the same document flagged its own biggest risk: "once a template becomes mandatory, the most likely failure is that people fill it with boilerplate that formally passes the check while saying nothing real — that's the step I'm most likely to have gotten wrong." A model willing to say, in writing, "my own suggestion might just get rubber-stamped" is more useful to me than the suggestion itself.

That proposal is, as of this writing, only designed — not yet wired into the actual goal template. Worth saying plainly: designed is not the same as done.

Takeaway

A single-repo audit can't see the shape that only exists once tools are wired together — and the most expensive lesson here is that the tool meant to repair drift can be the single riskiest thing in the whole system. "Content-level probes, not existence checks" is a rule worth copying into any multi-repo system you maintain, whether or not you have a Fable5 to run it for you.

本文原載於我的部落格：AI Auditing an AI's Own Toolchain: The Repair Tool That Overwrote Its Own Fix

How a Security Bug Gave Birth to a Method, Then the Method Caught What the Audit Missed

Dexterlung — Tue, 08 Sep 2026 14:05:22 +0000

At 5:27pm on 2026-07-03, I revoked self-write access on 8 sensitive columns of our members table. At 9:42pm the same day, I reverted it — an admin save feature started throwing 403 errors. Locked, then unlocked, same day.

That incident became the seed for a method we call the Confrontation-Point Law. I won't claim it's proven — there's exactly one real case so far, and we've deliberately labeled it "armed, not battle-tested." But that one case is one of my favorite technical stories to date, because what it caught was something a careful human audit had already missed.

Read on: Previous: don't ask a model to narrate itself · Next: AI auditing an AI's own toolchain · 繁體中文版

That lockdown commit looked careful

The 07-03 lockdown commit's header read: "verified all 5 write paths to the users table site-wide, MembersCenter.vue goes through fn_admin_* RPCs." That's not a throwaway line — someone actually enumerated the paths and labeled each as a legitimate channel. On paper, this looked like a solid audit.

But inside that same file, the function saveEditMember calls an RPC to write member_level at line 2010 — and ten lines later, at line 2020, the same function does a raw .from('users').update({ tier_updated_at, tier_expires_at, tier_auto_downgrade }). Direct write, no gate.

The same function was, at the file level, a proper RPC channel — and at the field level, simultaneously an unguarded direct-write channel. That gap is exactly what triggered the 403s that night, and exactly why we reverted four hours later.

Why a careful audit still missed it

Looking back, what bothers me isn't that there was a bug. It's why a careful audit still missed it. The answer is blunt: the auditor read the code, saw the RPC call, saw a comment that said "goes through RPC," and issued a file-level verdict. The census got outsourced to narrative. The fire wasn't because the set of write points was too large to enumerate — we later verified the whole frontend only touches that field in 3 files, something a dumb grep finds in seconds. The fire was because the verdict substituted "I read it, looks fine" for "I mechanically checked every single line."

That finding is what produced the first rule of the Confrontation-Point Law: every frozen judgment needs to name its confrontation point — a place where intent and actual state are forced to meet, where any disagreement blocks progress, sitting right on the boundary where real harm happens. Who's allowed to sit in the "this is what's actually true" seat? Only a machine — a real query like has_column_privilege, never an agent reading code and saying "I checked, looks fine." Narrative sitting in that seat is a fake census. That "goes through RPC" comment is a live example of narrative sitting in the wrong seat.

The same week, a mechanical re-audit caught 3 leaks the human review missed

On 07-04, instead of trusting "I read it, looks fine" again, we ran a fully mechanical re-census: every claimed-legitimate write point had to be re-derived through grep/AST plus an actual privilege query. Anything that couldn't be re-derived got flagged red on the spot — no exceptions.

This caught 3 admin direct-write points the earlier review had completely missed: renewTier, dismissBinding, importMembers. All 8 sensitive columns were re-verdicted from scratch — all 8 came back safe to revoke — and the three admin write paths got migrated into three new SECURITY DEFINER RPCs. This time the rollout order was deliberately sequenced: ship the RPCs first, wait for the frontend deploy to land, then run the revoke — specifically to avoid repeating the 07-03 disaster of locking permissions before the frontend had caught up.

After going live, we actually verified it in production, not just in code review: a regular member trying to self-write any of the 8 columns gets a 403; all 5 admin paths still work.

I'm not going to oversell this

Honestly: this whole method has exactly one real case behind it so far. We deliberately don't call it "battle-tested" — only "armed": self-tests pass, an injected regression test passes, but it hasn't yet stopped a real attack we didn't already know about. That line is written into the method itself: armed is not the same evidence tier as battle-tested.

There's a funny footnote to how this method got written, too: one of the prompts we fed to Fable had a placeholder bug that silently ate the draft content. Fable didn't hallucinate a plausible-sounding answer to cover for it — it just refused to answer. We retroactively counted that refusal as a live demonstration of the exact principle we were designing: refusing to answer is itself a legitimate output.

One more loose thread, left here honestly rather than swept under the rug: the same review found Addresses.vue writing to a column called addresses that doesn't actually exist in the database. That's deliberately out of scope for this fix and still unresolved.

Takeaway

A careful-looking audit can still lie at the field level, if its "verified" claim comes from a human reading code and forming a narrative judgment. A census you can actually trust is one where every single row can be mechanically re-derived — anything that can't gets flagged, no matter how confident the write-up sounds. And keep "armed" and "battle-tested" as two different evidence tiers — blurring them is just lying to yourself with extra steps.

本文原載於我的部落格：How a Security Bug Gave Birth to a Method, Then the Method Caught What the Audit Missed

Don't Ask a Model to Narrate Itself: How I Actually Extracted a Method From Fable5

Dexterlung — Thu, 03 Sep 2026 13:05:17 +0000

The first time I wanted to "learn how Fable5 thinks," I did the obvious thing: I asked it directly. "How do you approach a problem?" I copied down the answer, then kept asking, until it grew into an eleven-book set — a how-to guide, a book on perspective training, a book on code-review instincts, all the way up to a "main volume." It looked thorough. It took real time to assemble.

Then the whole set got overturned by its own method. Not because the content was wrong — because the approach itself was self-contradictory.

Read on: spec-sonar design journal, Part 1 · Next: the Confrontation-Point Law · 繁體中文版

The problem: I used an untrustworthy method to study a doctrine about not trusting self-reports

It was Opus that pointed out the embarrassing part: Fable's own core doctrine says don't trust self-reports — look at what actually got done, not what got said about it. And the method I'd used to write those eleven books was exactly that: ask "how do you think," then treat the answer as ground truth.

I used the discredited method to document the very doctrine that discredits it. Once that was named, only four principles out of the whole set survived. Everything else needed a rebuild. I wasn't angry so much as deflated — that wasn't ten minutes of work, it was a full conversation's output, now judged to have a broken foundation.

The fix: stop asking what it thinks, start reverse-engineering what it actually did

The new method was simple: instead of asking "how do you think," we took two things Fable5 had actually produced — the router-reasoning.md and decision-table.md from an n8n cold-start generator — and reverse-engineered the decision rules baked into them. Trust the scar, not the self-report — applied to the method itself, this time.

Then we ran a two-pass adversarial interview, four rounds. Pass one: no draft shown, free-form answers. Pass two: show it the draft, let it attack. The point of this structure is to surface two different kinds of error — pass one catches what we didn't think of; pass two catches what we got wrong.

Round one already surprised us. We'd planned to frame the method as something like "five joints." Fable rewrote the whole naming itself: forge, runway, customs — forge is where you're still building, runway is where you run an already-proven pattern, customs is the border crossing where the two are forced to meet and get checked. That wasn't a wording polish. It was a correction to how we'd classified our own method.

Round two: it caught an error happening live, inside our own document

Round two got more interesting. Opus's compressed one-line summary read: "reduce cold-start to lookup then emit a goal graph." Sounds fine. Fable pointed out that compression had quietly deleted two non-happy-path exits — "ask" and "stop" — which happen to be where every safety mechanism in the method actually lives. We checked the real document. The omission was real.

Something more embarrassing happened in the same round: Opus had written, in its own appendix, "we need to update this playbook" — and never did. Fable caught that live drift and used it as proof of its own thesis on the spot: canonical and working copies drift, not as theory, but as something happening in front of us.

Round four: an exam that could actually fail

Round four was a holdout test — a problem it had never seen (e-invoice integration), to see whether it would apply the same method cold. It didn't improvise an answer. It first grepped the real repo, found an old "manual invoice era" precedent, and answered from that. We checked afterward, line by line: the file and line number it cited matched the real file, verbatim.

That was the first time I'd actually verified a model's reasoning instead of just feeling like it guessed right — and a verified-correct answer is a categorically different thing from a lucky-and-unchecked one.

The last move: it turned its own rule against itself

The whole extraction closed on one line from Fable:

"You keep saying evidence. Do you have any evidence for yourself?"

The method's own second core tenet is the evidence rule — every frozen claim needs an anchor you can check. Fable pointed out that the entire playbook had, up to that point, zero evidence for its own validity. We thought the interview was over. Instead we got judged by the very method we'd built.

So we ran a "firing-record audit" — checking every claim in the method against the real git history of the CS project. The result wasn't a flat "no evidence." It split into three tiers: battle-tested (actually stopped a real problem), armed but not fired (the mechanism exists but hasn't faced a real test yet), and structurally unfireable (there's no way this claim could ever be checked). That three-tier grading is now how we evaluate any governance mechanism we build — not a vague "does it work," but an honest label of what evidence tier it's actually at.

What it cost me

Honestly: I never deleted those eleven overturned books, but I stopped using them. That was a full conversation's worth of time, spent to arrive at "the approach itself was wrong." If there's one real scar in this piece, it's that — I picked the wrong extraction method first, and it took a whole round to find out.

What crystallized

The four rounds converged into two reusable skills: connective-system-design (forge/runway/customs, for designing cross-system connections) and root-cause-diagnosis (counterfactual climbing, for finding a systemic root cause). The seal Fable added itself: "Trust the scar, not me." — including don't fully trust this article either. Go check the anchors.

The method was immediately put to two real tests: a cross-repo ecosystem audit across seven repos, and a real client project's frontend/backend asymmetry diagnosis. Both are the next two posts in this series.

Takeaway: if you want to actually learn a method from a model, don't ask "how do you think" — look at what it actually produced, reverse-engineer the rule, test it on something new, and verify before you believe it. A self-report isn't free truth. It's a claim you still have to go check.

本文原載於我的部落格：Don't Ask a Model to Narrate Itself: How I Actually Extracted a Method From Fable5

I turned both knobs on my on-prem Chinese RAG all the way up. It didn't get a single extra question right

Dexterlung — Tue, 01 Sep 2026 13:05:15 +0000

Something you can do right now

If you're about to "increase the retrieval depth" or "swap in a newer reranker" to improve your Chinese RAG accuracy, spend ten seconds on this first:

Take one question you got wrong, and compute where the passage containing the correct answer ranks in your raw vector similarity.

No reranker. No full pipeline. One embedding call and one round of cosine similarity.

That number decides everything downstream:

If it ranks outside your candidate depth → going deeper might help (only might — see below)
If it ranks inside your candidate depth → going deeper definitely won't, and your bottleneck is somewhere else

Without that probe I'd have swept top-k = 10 / 20 / 30 / 50 / 80 / 100, six configurations, 29 questions each, tens of seconds per question — about an hour. After the ten-second probe, those six configurations collapsed to one.

Where this started: a to-do I wrote for myself

I was evaluating a "the data never leaves the building" knowledge base for a client, and wrote up my research as a handoff doc for the next session. The to-do was specific:

Step 1: top-k 10 → 50 (zero cost, zero VRAM) — a public benchmark on 3,493 traditional-Chinese questions shows a weak embedding plus a reranker reaching 98%+ at k=100, beating every single-stage retriever.
⚠️ Don't just crank it to the max (a SIGIR 2025 paper reports that over-reranking degrades results); there's a sweet spot and you have to sweep for it.

Step 2: bge-reranker-base → bge-reranker-v2-m3 (+1.3GB, same architecture, same API, no prompt changes)

It reads as completely reasonable. External benchmarks behind it, a counter-example warning, a clear execution order.

Both steps ended up as "don't." And that "zero cost" in step 1 is wrong.

First cut: the cheap adjacent question

Before touching anything I looked at the table from my previous round:

Configuration	K=1	K=3	K=5
Vector only	84%	88%	92%
Vector + rerank (candidate depth 30)	88%	96%	96%
Vector + rerank (candidate depth 10)	84%	92%	92%

96% is 24 out of 25. And the miss is always the same question.

That sentence matters, because it caps the question "how much can deeper candidates buy me?" — at most one question, i.e. 4 percentage points.

So the question changed from "which k is best?" into something far cheaper: did the reranker ever even see the correct answer for that one question?

I wrote a ten-second probe: for each of the 25 questions, where does the passage containing the correct answer rank by raw vector similarity?

Rank of the first correct passage	Questions
1st	22
3rd / 5th / 21st	1 each
83rd	1 (the usual suspect)

One table cut six configurations down to one:

Depth 10 → correct answer present in candidates for 23/25 (ceiling 92%)
Depth 30 → 24/25 (ceiling 96%)
Depth 50 and 80 → still 24/25, because that question sits at 83. Identical ceiling to 30 ⇒ no need to run them
Depth 100 → 25/25 (ceiling 100%)

And check the calibration: depth 10's ceiling is 92%, measured 92%. Depth 30's ceiling is 96%, measured 96%. Both match to the decimal.

At that moment I derived a conclusion: "whatever's in the candidate set, the reranker pulls into the top 5 ⇒ this pipeline's bottleneck is how deep we fish, not how good the reranker is."

The derivation itself is sound (the miss set must contain the out-of-candidates set, and the two have equal size, so they're the same set).

Then I ran depth 100, and it falsified that conclusion.

Depth 100: ceiling 100%, measured 96%

	Depth 30	Depth 100
Retrieval hit rate (K=5)	96% (24/25)	96% (24/25)
Miss	that question	that question
Median latency per question	6,834 ms	37,651 ms

The correct answer ranks 83rd, so it did make it into the 100 candidates. The reranker saw it. And then ranked it outside the top 5.

So the right statement isn't the one I derived — it's a more useful one:

Candidate depth is necessary, not sufficient.
For that question to land, two things must both hold: ① you fish deep enough to see it, and ② the reranker is willing to rank it near the top.
Depth 30 fails at ①. Depth 100 clears ① and fails at ②.

Incidentally: "zero cost" is wrong. A reranker is a cross-encoder, and its cost is linear in the candidate count — send it 100 candidates and it computes "how relevant is this passage to this question" 100 times. Going from depth 30 to 100 took each question from 6.8 to 37.7 seconds: 5.5× the latency, for zero extra correct answers.

"Zero VRAM" is right. "Zero cost" is not. And what it spends is precisely the one budget in this setup that had already bottomed out.

Second cut: swap in a "newer" reranker

If the bottleneck is the reranker, replace it. My to-do said bge-reranker-v2-m3.

First discovery: that model isn't available on my stack. I run ONNX through fastembed on CPU, and its cross-encoder list has six models — no v2-m3. Only two are multilingual: the bge-reranker-base I'm already on (a 2023 base-tier model), and jinaai/jina-reranker-v2-base-multilingual (1.11 GB, 2024 generation, marketed on multilingual).

Actually using v2-m3 would mean swapping the whole service from ONNX to PyTorch. That's a separate project. So I swapped in the jina model — same API, one parameter changed, no prompt edits.

It lost in three directions at once:

reranker	K=1	K=3	K=5	Pure Chinese prose questions, K=1	Latency/question
`bge-reranker-base` (2023)	88%	96%	96%	88.9% (8/9)	6,834 ms
`jina-reranker-v2-multilingual` (2024)	76%	96%	96%	66.7% (6/9)	8,219 ms

K=1 dropped 12 points (3 questions). K=1 is the column that most affects the quote — it decides how many documents the client has to feed the LLM, which decides whether they need to buy that GPU.
The worst drop is on pure Chinese prose (88.9% → 66.7%). A multilingual-first 2024 model losing to a 2023 base-tier model on traditional Chinese prose.
It's also 1.2× slower.

But the fourth reason is the fatal one, and I nearly missed it.

The fourth reason I nearly missed

My previous round had turned up a bonus finding: the reranker's score itself works as a "refuse to answer without calling the LLM" gate. Below a threshold, return "not found" — skip generation entirely. Fast, and structurally incapable of making things up.

For this client that matters more than usual, because making things up is exactly what they're afraid of.

The scoring script prints a "median top-1 score" line each round:

bge-reranker-base: answerable 5.26 / unanswerable −2.14
jina: answerable 0.84 / unanswerable −0.76

That looks like "the gap got smaller." What I wrote in the report at the time was "this doesn't let us say jina made the gate worse [unknown]."

But the statistic that finding needs isn't the median. It needs the minimum score among answerable questions against the maximum among unanswerable ones — because two distributions can have very different medians and still have overlapping tails.

The per-question scores were already in the scoring run's JSON. I computed it afterwards at zero extra cost:

reranker	Answerable min	Unanswerable max	Gap	Can one threshold separate them?
`bge-reranker-base`	−0.5237	−1.8008	+1.2771	✅ yes
`jina-v2-multilingual`	−0.9005	−0.6736	−0.2269	❌ overlap
Vector only (no rerank)	0.5895	0.6377	−0.0482	❌ overlap

Switching to jina means: K=1 down 12 points, latency up 1.2×, and then you also forfeit the entire refusal defence.

And the median made it look like it merely "got smaller." That's the most valuable lesson of this round: the statistic a report prints is not necessarily the statistic your conclusion needs. I turned it into a command (reads the stored scoring JSON, zero cost) and left a test inside it: fixtures of answerable [9, 9, 9, −1] and unanswerable [0] — medians differ by 9, looks trivially separable; the truth is min −1 < max 0, overlap. That test exists so that if anyone ever switches it back to medians, it goes red.

What's actually wrong with that one question: I tried a third road too, and it's also closed

I had one hypothesis left. Looking at the top-5 for that question across four configurations, I noticed all five slots in every configuration were filled by the same class of file — my own blog drafts. Derivative articles discussing the same incidents, not the canonical records of those incidents.

That matches a published finding: what hurts RAG isn't "irrelevant," it's "relevant but wrong." It also matches my own earlier observation that this corpus writes about the same event a dozen times over.

So I added an exclusion rule, filtered the entire blog-source directory out of the candidates, and re-ran that question.

Still missed. And the correct answer didn't even make the filtered 30 candidates — excluding the derivative material just pushed other non-answers up.

Three roads walked:

Hypothesis	Measured	Conclusion
Not deep enough	depth 100 got it into candidates, reranker still didn't rank it	❌ necessary, not sufficient
Reranker too old	swapped to a 2024 multilingual model; still missed, every other metric regressed	❌ actively worse
Corpus noise drowning the canon	excluded the biggest source of derivatives; it can't even reach candidates	❌ hypothesis falsified
Embedding not good enough	untested	⬅️ only one left

That question asks "what has to be done to a temp table containing personal data for it to count as safe, and which incident forced that?" — and the file with the answer is anchored on a database table name. The question doesn't contain that table name; only the answer does. It's a pure semantic-bridging task, and my embedding model puts it 83rd.

This round proved the embedding is the bottleneck, by elimination — every knob outside the embedding was turned to the stop with no effect. My original plan wrote the precondition for "swap the embedding" as "only swap once you've proven embedding is the bottleneck." That precondition now holds; it just got proven by exclusion.

My next cut will be query rewriting (have the LLM expand the question into a hypothetical answer first, then embed that) rather than swapping the embedding outright — much cheaper, and among the open-source projects I've read, the one that handles Chinese most seriously does its work on the query construction side (part-of-speech weighting, positional proximity weighting), not by changing models.

A red flag that goes straight into the quote

This round also turned up something I wasn't looking for.

To get a same-day control, I re-ran the depth-30 configuration. Recall was identical (96%, same question missed) — comparability confirmed.

But latency was 6,834 ms per question. A week earlier I'd measured the same configuration at 18,152 ms.

Same model, same corpus, same question bank, same laptop. A factor of 2.66.

Two consequences:

I'd been comparing a fresh depth-100 number (37,651 ms) against last week's 18,152 ms and computing "2.07×." Wrong. The correct same-day ratio is 5.51×. A latency denominator has to be produced in the same round as its numerator — the same discipline as "pin the corpus to a commit," which I'd applied to the corpus and not to latency.
Worse: the conclusion I'd written a week earlier was "18 seconds per question — unusable for interactive Q&A." That sentence cannot be measured today (6.8 seconds). 6.8 seconds is still slow, but "slow" and "unusable" are completely different things in a client quote — and that sentence was headed into the proposal.

I don't know the source of the variance yet (thermal throttling? something else running? a package version?). All I can state is that corpus, question bank, and model are ruled out, because all three were re-verified identical the same day.

So the rule now is: any latency number going into a quote must have its own control, measured in the same round, on the same machine, with no other load.

Net result

Four scoring runs, three probes, roughly an hour of machine time. What it bought:

Don't touch either knob. Stay on bge-reranker-base with candidate depth 30.

That sounds like nothing happened. In practice I got three things:

A number I can commit to: on a 25-question benchmark, 96% retrieval at K=5, 88% at K=1, plus a refusal gate that doesn't depend on the LLM.
A number I can't commit to: 100% retrieval. Measured as unbuyable — deeper candidates don't buy it, and neither does the newest reranker I can actually run. That's far more valuable than "I think we can probably get there," because it lets me write a ceiling into the proposal.
Two numbers not to quote: any local CPU latency (2.66× day-to-day variance), and any presumption that "a newer model will be better."

Plus one observation I can sell: derivative copies inside a client's corpus (drafts, meeting notes, several versions of the same policy) will systematically outrank the canon — my own blog drafts pushing my own canonical records out of the top 5 is the live example. But don't sell "curate the corpus" as a cure-all, because in the same round I measured it doing nothing for that question.

Four things you can take away

Before turning any expensive knob, find a ceiling you can compute in ten seconds. "At most how many questions can this buy?" routinely makes an entire sweep plan disappear.
"Newer, bigger, multilingual-first" does not mean "better on your corpus." Marketing labels aren't evidence. I put a 2024-generation multilingual reranker in and it lost to a 2023 base-tier model on traditional Chinese prose.
The statistic a report prints isn't necessarily the statistic your conclusion needs. A median can't answer "are these separable?" Ask which statistic your claim rests on, then go compute that one.
Don't quote a latency number without a control. The same configuration varies 2.66× day to day on my machine, and it nearly put the word "unusable" into a client proposal.

If you're doing this kind of measure-before-you-change work, I've written others in the same family: my test report printed "0/96, 0% pass rate" — the truth was my account was out of credit, on why a red light needs as much suspicion as a green one; and the A/B test where I cut 41 tool descriptions in half, which is the same discipline applied to a change I was confident about — an absolute number can't tell you whether something dropped, so you have to go build the control arm.

本文原載於我的部落格：I turned both knobs on my on-prem Chinese RAG all the way up. It didn't get a single extra question right

DEV Community: Dexterlung

I ran claude plugin eval on 89 skills. The trigger rate fell from 100% to 55.6%.

The problem with skills is that failure is silent

The mechanism is simple. One design choice isn't.

First measurement: 100% with one skill loaded

Second measurement: 55.6% with 89 loaded

Then I did something stupid

The traces: it didn't fire the skill, but it did the skill's job

A flaw in my test design, while I'm being honest

The 0.00 that never ran

Why would Anthropic make this a first-party command?

What I know and what I don't

Google crawled 270 of my articles and indexed none — the body it received was 0 characters

The numbers that started it

First move: fetch your own page as Googlebot

Fix 1: put the body in the first-hand HTML

Fix 2: 157 soft 404s, because the site answered 200 to everything

The load-bearing part isn't the 404, it's fail-open

Fix 3: one stylesheet costing 2,222 ms

Fix 4: a 531K PDF bundle preloaded on every page

Then I shipped a new defect: the title, three times

The test was an accomplice

A footnote: a byline can't be a shell

If you run an SPA content site, check in this order

Where the numbers come from

RLS says yes and Postgres still says permission denied: the 403 family I only understood on the second one

RLS says yes and Postgres still says permission denied: the 403 family I only understood on the second one

The symptom: the admin is locked out of his own system

The cause: RLS governs which rows, GRANT governs which columns

The fix: a DEFINER wrapper with the guard on the first line

The turn: fixing the second one told me it was a family

Not every member of a family gets the same fix

Three things to take away

My most important test had never failed — because what it proved was 1 = 1

Do this first — it is uncomfortable and it takes five minutes

The setup

Then I handed it to a different model

Why it could never go red

I made the same shape of mistake three times in one day

What the working version looks like

One more trap: a prove-it-goes-red run must control its input

Why having the rule written down did not save me

The one-minute version

31 articles vanished from my site. No error, no log — one hardcoded .limit(80).

31 articles vanished from my site. No error, no log — one hardcoded .limit(80).

The symptom: the data is there, the people are gone

The root cause: .limit(80) truncates in silence

Stopgap: change 80 to 200 and move on, for now

The real fix: delete the premise that the frontend loads everything

Verification: "they're back" is not a number

Three things to take away

Supabase had granted anon TRUNCATE on 96 tables. I never wrote that line, and every check I own was green.

Do this first, it takes fifteen seconds

How I found it

Where the grants came from

Why every check I own was green

The more embarrassing half: I made the same mistake again while fixing it

The three rules I took out of it

The fix, copy-pasteable

What it cost me to learn

Two identical runs scored 89 and 89. Two cases had flipped.

First I went looking for the answer I'd already paid for

The baseline

Where the noise actually lives

The bad news, which is also the interesting news

The threshold question

What I'd take from this

What this doesn't establish

Dead Code Is More Dangerous Than Broken Code — How a Non-Homologous AI Caught the Blind Spot the AI Itself Had Rubber-Stamped

Dead Code Is More Dangerous Than Broken Code

The opening: an AI confidently told me to reuse a capability that didn't exist

What "existential rot" is

The real root cause: pricing authority actually lives in the browser

Why my governance saw none of this

Why it had to be a non-homologous model

Six green lights, not one of them watching the load-bearing thing

The fix: let the zombies die all the way first, then give the living body witnesses

Turning one expensive insight into a free mechanism forever

Closing: this was never really about pricing

AI Auditing an AI's Own Toolchain: The Repair Tool That Overwrote Its Own Fix

RLS says yes and Postgres still says `permission denied`: the 403 family I only understood on the second one

31 articles vanished from my site. No error, no log — one hardcoded `.limit(80)`.

The root cause: `.limit(80)` truncates in silence