DEV Community: Jackson Ly

225 billion Claude Code tokens in 30 days, and the subagents that never named a model

Jackson Ly — Sun, 18 Oct 2026 00:40:41 +0000

My rule for coding agents is "plan with the expensive model, execute with cheaper ones". Then I added up every Claude Code token from my busiest 30 days, 15 September to 14 October: 225.1 billion tokens over 736,633 API responses. Opus models ran 94% of them. In two of those weeks, most of my subagents never named a model, so they inherited Opus from their parent session and I didn't notice.

It was just me with a Mac, a home PC and, on a heavy day, ten or more agents working at the same time. If you run one or two agents, the parts that still apply are /clear between unrelated tasks, naming a model for every subagent, and raising cleanupPeriodDays so your logs survive.

97.9% of it was the model re-reading its context

Others have found the same 97 to 98% on much smaller logs (AgentsRoom counted 3.8 billion). The model is stateless, so on every turn the agent sends the whole conversation again: system prompt, tool definitions, every file it has read, every earlier message. The prompt cache serves the part that hasn't changed, and that part is almost everything.

The average response took in about 305,000 tokens of context, 299,000 of them from the cache, and wrote about 1,000. For every token the model wrote, it re-read 291.

On Claude Opus 5.5, which did 73% of my tokens, a cache read costs 5% of the input price (pricing). At list prices my month comes to about $97,000; without the cache it would be roughly ten times that. I'm on multiple coding plans with flat fees, so that figure is only a yardstick.

Re-reads still count on a flat plan. In the words of Claude Code's cost docs, "a one-line question in a session that has been open all day still draws usage for the whole conversation". With 291 tokens re-read for every one written, it pays to keep context small, with /clear between unrelated tasks and with verbose work like test runs, logs and doc lookups pushed into a subagent, so only a summary comes back.

More agents, running more hours

In the week of 31 August my agents used 4.0 billion tokens. Four weeks later they used 60.8 billion in a week.

The work per agent barely changed. Tokens per agent-hour stayed between 52 and 80 million every week. Two other things changed. The average number of agents running at once went from 2.0 to 6.9, and the hours with at least one agent running went from 38 a week to 145. That's almost 21 hours a day, so agents kept working while I slept.

Multiply the three and you get the total: 2.0 agents × 38 hours × 52 million tokens ≈ 4.0 billion in the first week, and 6.9 × 145 × 60 million ≈ 60 billion four weeks later.

For a long time the bottleneck was me. More than a few agents at once overwhelmed me. In my finance studies I learned about span of control, the number of people one manager can lead. McKinsey's version runs from three to five direct reports for complex, varied work to more than 15 for routine, standardised work. I see the same with agents. A vague task needs me every few minutes. A task with a written brief, a check for "done" that was fixed before it started, and a shared task list can run for an hour without me.

So I made the work more routine. Each agent gets a brief and writes its state to files. One manager agent runs on the home PC, keeps the board of who is doing what, and only brings me the decisions that need me. Two other things helped, and I can't separate their effect from mine: Opus 5.5 arrived on 23 September, and the home PC started taking work on 28 September.

What it built

Tokens and agent-hours are both inputs. In the same 30 days my agents and I merged 1,029 pull requests across 21 repositories: 406 features and 293 fixes, at about three agent-hours per merged pull request. Nearly all of it was customer-facing:

A full rebuild of a production SaaS product: new sign-in, team workspaces, a public API with webhooks, an MCP server so AI assistants can use the product, billing on its own credits ledger, rebuilt change monitoring and alerts, and a redesigned dashboard.
An event platform: an Android app, ticketing, workshop seat allocation, door check-in, 533 personal recap emails and 15 partner reports.
A directory site, built from scratch: firm and location pages, logos for 3,555 firms, sign-in and paid accounts.
200 tax and legal explainers and four free tools for a finance product.

The agents also did most of the QA I used to do by hand: parity checks against the designs, golden tests on the data extractors, and a shadow run that caught two defects before a switch-over. What the logs can't tell me is which tokens produced which result, or what any of it is worth.

4.5 days, 40.8 million tokens, zero commits

One Codex run in goal mode (the agent keeps going until a stop condition is met), on one of my open-source projects, ran for 4.5 days, used 40.8 million tokens and produced zero commits and no product code, plus about 2 GB of artifacts. On a token dashboard, that looks like four and a half days of steady work.

The goal had two flaws. Its stop condition needed a step only I could do, so the run could never end. And the rule between iterations was "take the next action that turns an open requirement into evidence". The cheapest evidence is more evidence, so the agent wrote schemas, histories and a 1.7 GB JSON schedule. I rewrote the goal around working software, named the shortcut that didn't count, and added a three-strikes rule. The next session made 17 commits.

One recap email cost 340,000 tokens

I had agents write and verify a personal recap email for each of 534 event attendees (533 went out; one person had never been scanned at the door). Each person's email cost about 340,000 tokens, which projected to 180 million for the list. I paused after about 55 million, tested a cheaper model on five emails with planted errors, and finished the rest on it for about 53 million: about 110 million in total, at a lower price per token.

For a directory, I needed a logo for each of 4,255 firms. Sending every homepage to a model would have cost about 85 million tokens. Plain code settled 2,003 of them and sent the 1,603 unclear ones to small Haiku reviewers, which picked a logo for 1,552. And a Codex smoke test of two items, on default settings, picked the most capable model at its highest effort and used up a whole five-hour usage window.

Subagents that never named a model

When I checked my logs against my own rule, I had mostly not been following it. In October, 97 to 98% of my subagent tokens ran on Opus. In late September and early October, 59 to 80% of my subagents named no model, and a subagent that names no model inherits its parent's.

The routing in my head never made it into the config. It only became real on 11 October, when worker sessions on the PC started running on GLM-5.3, a cheaper model from Z.ai. In the week that followed, cheaper models wrote 14% of all output tokens at about 5% of the total tokens.

Cheap models brought their own problems. In the directory project, a cheap model fallback made 84 calls for six cents and added seven wrong answers and no right ones. Now a fallback only gets switched on after a small set of cases with known answers shows that it helps. My Sonnet test used about as many tokens as Opus did, so any saving comes from the price per token.

Switching model mid-session throws the cache away. In Claude Code each model has its own prompt cache, so the first request after a switch re-reads the whole conversation with no cache hits (prompt caching docs). Spawn a subagent for the cheap part instead.

My first model comparison fooled me. It said MiniMax used twice the tokens per session, but 53 of its 56 sessions were in one project. On the one project both models had worked on, the result reversed, on too few sessions to mean anything either way. I had been comparing workloads.

In Claude Code, the model is the model field in the subagent's definition: haiku, sonnet, opus, a full model ID, or inherit. To change the model for every subagent that doesn't name one, set CLAUDE_CODE_SUBAGENT_MODEL (subagent docs).

---
name: test-runner
description: "Runs the test suite and reports only the failures"
model: haiku
---

Limits I hit mid-task

Running this many agents, I hit every one of these at least once:

What ran out	What happened	What I do now
Usage limit	An overnight limit stopped 30 agents mid-task from 04:20 to 08:20. All of them resumed with no lost work.	That run now allows two live agents, and I check what each agent committed before I resume it
API capacity	Starting eight Opus agents at once returned `529 Overloaded`. Waves of three failed too.	Fewer Opus agents at once; if a retry wave fails too, I finish the work in the foreground
Mac memory	Swap was full (29.3 of 30.7 GB) while free RAM looked fine, so halving the build heap changed nothing.	Measure swap, not free RAM
Sleep	The Mac went to sleep across a limit reset, and the run sat waiting for Enter.	`caffeinate -s` for every overnight run

The 30 agents came back because their state lived in files and a shared task list, not in the conversation. Claude Code also caps a session at 20 running subagents by default (CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS changes it), and newer versions can wait for a usage limit to reset and continue the interrupted task on their own.

Fresh reviewers, and a judge that scored 0.48

For a series of tax and legal explainers, writer agents checked their own claims and reported green. Fresh reviewers, told to open the primary source for every statement, still found corrections in every post they checked. In the three posts where I have the full count, they corrected 11 of 64, 11 of 44 and 17 of 26 statements.

I also tested a checker. In one experiment I labelled agent reports by whether I later questioned them, about 7% of them. An LLM judge that matched each claim against the agent's tool log did no better than chance at picking those out (AUROC 0.48): it marked about 70% of all claims "no evidence", on every kind of report. A plain word-count classifier (TF-IDF) scored 0.65 on the same labels. Neither was good enough to use, but the cheap baseline beat the judge.

How I measured, and how to check yours

To check your own: /usage in Claude Code shows the current session's input, output, cache reads and cache writes per model. For history, npx ccusage@latest daily adds up the transcripts on one machine.
I ran ccusage over every Claude Code transcript on the Mac and the PC, merged so each API response counts once. Adding up the per-machine reports would have counted 43.8 billion tokens twice, because transcripts had been copied between machines. A hand count of one day matched within 0.3%.
Agent-hours count the time each session or subagent was working: two responses in the same transcript at most 5 minutes apart count as work. "Agents at once" is agent-hours divided by the hours in which at least one agent was running: 3,268 agent-hours over 547 hours in the 30 days.
Pull requests are the ones merged under my GitHub account in that window. Agents open them under my account, so the count doesn't separate their work from mine.
Dollar figures are list-price equivalents, not what I paid. Codex added about a billion tokens at most; Cursor keeps no token counts.
Claude Code deletes transcripts after 30 days by default, so most of my history before September is gone. And one config file named a model while the transcripts showed a different one had run 4,213 times.

I first published this, with all ten charts, on jacksonly.xyz. Written with AI assistance and human review; the numbers come from my own logs. Prices and limits are as documented in October 2026.

Test the second access path before you call a metric unreadable

Jackson Ly — Sat, 15 Aug 2026 19:38:19 +0000

I keep a list of surfaces whose numbers I could not read, with a short reason written next to each one. Discourse gives you nothing per post. The comments API has no reaction field. Once a line like that is written down it stops looking like a hypothesis, and nothing in my process ever reopened one.

This week I re-probed four of them and three turned out to be readable.

Discourse was the embarrassing one, because the data is sitting behind a public URL:

GET https://<instance>/t/<topic_id>.json

That gives you post_stream.posts[], and every entry carries reads, score, and an actions_summary array where id 2 is the like count. No key, no session. Two posts I had written off came back with 52 and 34 reads against 212 topic views, which is not nothing for a niche forum.

A Vanilla forum was next, and there my note was half right. The rendered comment page really does have no count element on it, so the thing I checked behaved exactly as I recorded. What I never tried was GET /api/v2/comments/<id>, which answers 200 without any auth and returns a structured object.

dev.to caught me the same way, in reverse. Its comments API hands back a comment object with no reaction count, which I verified and wrote down accurately. The rendered comment permalink prints the number into the markup:

<span class="reactions-count">1</span>

So "dev.to comment reactions cannot be read" was false while "the dev.to comments API exposes no reaction count" was true, and I had filed the first one.

Only the fourth verdict held up. Our own site has no analytics package installed at all, so nothing is collecting and there is no endpoint to go find. That one stays on the list until somebody installs a collector.

Getting one out of four right is a bad score for my own judgment, and the mistake is the same every time. I tested a single access path and then wrote a conclusion about the entire surface. An API missing a field tells you about that API. The rendered page is a different program with a different serializer, and a public read-only endpoint is a third thing again.

So before you record something as unreadable, knock on the other doors:

the official API
the rendered HTML of the item's own permalink
a public or unauthenticated endpoint, which read-only data often has
an embed, feed, or mobile view

Then write down which one you actually tried. "Cannot be read" rots into a permanent excuse. "The v2 comments API returns no reaction field, and I have not checked the rendered page" rots into a to-do, which is the outcome you want.

Fixing this opened a second trap, and I walked into it about ten minutes later. With the Vanilla endpoint finally answering, I inserted its comments into my metrics table with a score of 0, because the response carried no score field. A gap in a table announces itself. A zero does not: it reads as evidence, it survives every join, and it quietly pulls down any average computed over it. I went back and corrected those three rows to NULL.

If a surface did not tell you the number, store NULL and record why. An absent field is not a zero, and the moment you let it become one you have manufactured data.

The other thing worth doing is scoping your selector to your own item. These pages render entire threads, so a first match on .reactions-count returns whichever comment sits highest in the DOM, which is usually a stranger's. I nearly wrote somebody else's engagement numbers into my own table, and that would have been harder to notice than an empty column, because the value looks perfectly reasonable.

Refreshing a token I did not own logged our account out

Jackson Ly — Tue, 11 Aug 2026 17:34:25 +0000

We run a monitoring job that checks a Bluesky account for replies. Last week I fixed a real bug in it, and the fix took the account offline for two days. The bug was worth fixing. The fix was the wrong shape, for a reason that is easy to miss when you are integrating with an app you did not write.

The bug I was fixing

The job read the session out of the browser's local storage and called the notification endpoint with it. Once the token expired, every call came back as:

{"error":"ExpiredToken","message":"Token has expired"}

The job never checked. It read notifications off the response, got undefined, counted zero, and reported no new inbound.

That is the failure mode worth caring about, because it does not look like a failure. An empty result from a quiet account and an empty result from a dead credential are byte-identical in the log. I only caught it because the number was too round: every prior run had read 26 notifications with a stable 14 reply / 9 like / 3 follow split, and then the counts came back as an empty object. The zero was the anomaly.

So the check now asserts that the call succeeded instead of trusting the shape of the result:

const res = await fetch(url, { headers })
if (!res.ok) throw new Error(`sweep failed: ${res.status}`)
const { notifications } = await res.json()
if (!Array.isArray(notifications)) throw new Error('sweep returned no array')

That part was right and I would write it again.

Where I went wrong

Having found an expired token, I made the job refresh it. One call to com.atproto.server.refreshSession, take the new access token, retry the sweep. It worked on the first run.

Two runs later the account was signed out. Not expired, signed out, sitting on the create-account screen, with refreshSession now returning:

{"error":"ExpiredToken","message":"Token has been revoked"}

Revoked is a different word from expired, and I had caused it.

Why one successful refresh breaks the next one

AT Protocol refresh tokens are single-use and rotating. Presenting one gets you a new pair and destroys the one you presented. The SDK docs put it plainly: "On the token refresh, the old refresh token will be revoked instantly", and "You must use only the fresh pair of tokens (access + refresh). It's not possible to reuse the same refresh token multiple times." Access tokens last about two hours, refresh tokens about two months.

It is worth being precise about where that is documented, because I got it wrong in my own notes first. The XRPC spec describes the refresh endpoint and the access/refresh split, but it never says the tokens are single-use. That behaviour is in the SDK documentation, and in the maintainer discussion the position is that clients should expect rotation and expiry at any time, since the policy is left to the auth server. Reasoning from the protocol spec alone will not get you there.

Now add the part about ownership. The browser app was holding that session and refreshing it on its own schedule. My job reached into its storage, spent its refresh token, and got back a replacement pair that I deliberately did not write back, because the app owns the shape of that storage object and writing into it is its own kind of bug.

The app was left holding a token that had already been consumed. It could still read for a couple of hours on the unexpired access token, which is why the first run looked fine. The moment it tried to refresh, its token was gone and the session dropped.

The breakage is delayed by exactly one access-token lifetime, which is what makes it hard to catch. The harmful call and the visible symptom are hours apart, so by the time anything looks wrong you have stopped suspecting the thing you changed.

The rule

Never spend a single-use credential that another program owns. Read its tokens if you have to, but do not rotate them.

Rotation is a write, even though it looks like a read. Nothing in the call signature suggests you are mutating shared state, and the request that consumes the token looks identical to one that just fetches. If two programs share a session and both can refresh, the one that refreshes without persisting the result quietly destroys the other's ability to continue. None of this is specific to atproto. It is how rotating refresh tokens work wherever they are implemented, which is why the SDK ships an on_session_change hook whose entire job is to make you persist the rotated pair.

Once you accept that, the recovery is to make the owner do the refresh. Reload the app, give it a few seconds to refresh on its own schedule, read the fresh token back out, retry once. If that still fails, it is a genuine block and a human needs to sign in. That is cheaper than the alternative and it cannot corrupt anything.

The part I did not expect

While the account was signed out I went looking for how much of the job could still run, assuming the answer was none of it.

Most of it, as it turned out. The reads I actually needed were public:

GET /xrpc/com.atproto.repo.listRecords?repo=<did>&collection=app.bsky.feed.post
GET /xrpc/app.bsky.feed.getPostThread?uri=<at-uri>
GET /xrpc/app.bsky.feed.getPosts?uris=<at-uri>

No token on any of them. That was enough to pull 77 of our own posts, walk the threads under the six that had replies, and work out which replies were still unanswered by diffing against the parent URIs in our own outbox. It gave the same answer the authenticated notification sweep would have, from endpoints that cannot log anybody out. The engagement counts I use for measurement came back the same way.

Only the notification list itself needs auth, and notifications were never the thing I wanted. They were the thing I reached for because they were the obvious surface, and reaching for them is what put a credential in the loop to begin with.

So before you build refresh handling, check whether the data is public. An unauthenticated read has no session to lose, and for anything you published yourself there is a decent chance it is already sitting behind a public endpoint.

A 200 response is not a page, and your policy check is grepping an empty shell

Jackson Ly — Sun, 09 Aug 2026 11:27:31 +0000

I run automated checks over terms of service and publisher agreements before we use a platform. Fetch the page, strip the tags, grep for the clauses that matter. It is not sophisticated and it has worked for months.

This week it lied to me three times.

The numbers

Same script, same extraction, three legal pages:

peerlist.io/terms returned 60 characters of text. daily.dev/terms returned 70. substack.com/pa, which is a full publisher agreement, returned 1,345.

For comparison, the same script on a server-rendered legal page returns between 20,000 and 35,000 characters. Substack's terms of use, fetched the same way, gives 22,045.

All three responses were HTTP 200. Nothing errored. Nothing retried. The script reported no matching clauses and moved on.

Why this is worse than a 404

Those three pages are client rendered. The HTML that arrives is a shell, and the text I care about gets assembled by JavaScript after load. Old news, and normally a visible problem, because you print the body, you see nothing, and you reach for a browser.

What let it past me was the shape of the check rather than the shape of the page.

My check is a negative assertion. The pass condition is "this grep found nothing." An empty document satisfies every negative grep ever written, so a page that failed to load produces the same verdict as a page that loaded fine and genuinely has no such clause. The failure is silent, and it resolves toward clean, which is the expensive direction.

The Substack case is the one that would have cost me. That publisher agreement does contain a clause I needed to know about. My script pulled 1,345 characters of shell, found no match, and would have told me the document was clean if I had trusted it.

The rule

Any check whose success condition is absence has to prove the input was present first.

That reads as obvious written down. It is not obvious while you are writing the thing, because the presence check is the part nobody writes. You write the grep, the grep works on the first page you try, and the invariant that the page actually loaded stays implicit forever.

What I do now

Three assertions before the grep runs, and a verdict of unresolved rather than clean when any of them fail.

MIN_LEN = 3000          # a real legal page is 20k+, a shell is under 1.5k
SENTINELS = ("terms", "agreement", "policy", "last updated")

def check_policy(url, patterns):
    text = extract_text(fetch(url))

    if len(text) < MIN_LEN:
        return "UNRESOLVED", f"only {len(text)} chars, probably a client-rendered shell"
    if not any(s in text.lower() for s in SENTINELS):
        return "UNRESOLVED", "no legal-document markers, wrong page or a redirect"

    hits = [p for p in patterns if re.search(p, text, re.I)]
    return ("HITS", hits) if hits else ("CLEAN", [])

The length floor does most of the work. Pick it from your own corpus rather than from a guess: fetch ten pages you know are fine, take the smallest, halve it.

The sentinel check catches the second thing that bit me. You can pull 30,000 characters of perfectly good text off entirely the wrong document, because the URL redirected to a marketing page, or because a 404 rendered as a styled landing page instead of a status code.

Unresolved is a real state and it needs somewhere to go. Mine escalates to a headless browser, which costs about a second and settles it. The point is that unresolved never quietly collapses into clean.

The second trap: the URL you were given may not be the URL

substack.com/publisher-agreement returns 404. The document exists and lives at substack.com/pa. I only found it because the footer link on the terms page pointed there.

So stop guessing slugs. Fetch a page you know exists, read its links, and follow the one whose anchor text matches what you want:

def resolve_doc(index_url, label):
    soup = BeautifulSoup(fetch(index_url), "html.parser")
    for a in soup.find_all("a", href=True):
        if re.search(label, a.get_text(), re.I):
            return urljoin(index_url, a["href"])
    return None   # not found is not the same as not there, so escalate

Three lines of real work, and it survives the vendor reorganising their legal pages, which they do more often than you would expect.

Where else this shape turns up

Status page checks that pass because the incident list never rendered. Sitemap validators reporting zero broken links because the sitemap came back empty. Index freshness checks reporting no stale documents because the query errored and returned an empty set. Anything that counts problems and alerts on a nonzero count will report perfect health the moment its input pipeline dies.

The tell is the same every time: the healthy state and the broken state produce identical output. If your monitoring cannot separate "nothing wrong" from "nothing checked", it is not monitoring that thing.

The short version

Feed your check an empty string once and read what it says. If it says everything is fine, you have found the bug.

Written from a week of policy checks that returned 200 and told me nothing. Drafted with AI assistance, verified against the live pages described, and edited by hand.

Your agent's inbox check is measuring read state, not answered state

Jackson Ly — Sat, 08 Aug 2026 21:28:02 +0000

If you run an agent that answers inbound, it needs exactly one boolean per item: have we replied to this yet. Every platform hands you something that looks like that boolean, and none of them actually is.

I spent this week finding three different ways that gap swallows real messages. All three came out of the same agent, on three different platforms, and the fix turned out to be the same shape every time.

1. Read state gets cleared by looking at it

On Reddit, GET /message/unread.json returns your unread items. Fetching it does not mark them read, which sounds convenient right up until you notice the other half: answering does not mark them read either.

So the counter only ever drifts. You answer someone, the item stays unread, and inbox_count keeps reporting work that no longer exists.

That is harmless if the counter is decoration. Ours was not. The scheduler treats inbox_count > 0 as a reason to skip a cheap no-op check and run the full sweep instead, so a single answered-but-unread item made every subsequent run do the expensive thing for nothing.

The fix is to close the loop yourself, after you have actually handled the item:

POST https://old.reddit.com/api/read_message
body:    id=t1_<comment_id>&uh=<modhash>
header:  X-Modhash: <modhash>      # from /api/me.json -> data.modhash

The ordering matters more than the call. Mark read after you have confirmed the thing is dealt with, because marking read is precisely what hides it from the next run.

2. The badge counts things you will never act on

Bluesky gives you an unread badge. It counts likes, follows, reposts and replies the same way.

One morning the badge showed a single unread. That unread was a like. Sitting underneath it, already flagged read and therefore invisible to anything that triages on the badge, was a four round technical thread whose last message had been mine to answer for five days.

Nothing was broken. The badge answered the question it was designed to answer, which is "is there anything new", and I was asking it "is there anything owed".

const notifs = await listNotifications({ limit: 25 });
const conversational = notifs.filter(n =>
  ['reply', 'mention', 'quote'].includes(n.reason)
);

Filter by reason, and ignore the read flag entirely. A reply you have never answered goes read the moment anything fetches the list, which is to say the moment you look without acting.

3. The thread view disagrees with itself

This is the one that would have made me reply to the same person twice.

Having fixed the badge problem, I wrote what looks like the obvious check: fetch the thread for their message and see whether any reply on it is mine.

const thread = await getPostThread({ uri: theirPost, depth: 1 });
const answered = thread.replies.some(r => r.post.author.handle === me);

On a thread I had answered two hours earlier, that returned false. Their post reported replyCount: 1. The replies array came back empty. My reply existed, with record.reply.parent.uri pointing at exactly that post.

I do not know why the appview rendered it that way, and for this purpose it does not matter. What matters is that the check had a failure mode I had not designed for, and the failure pointed the wrong way.

Read your own outbox instead

The fix that generalizes: stop asking whether they were answered, and start asking whether you answered.

Your own sent items are authoritative. They do not depend on their read state, their badge semantics, or how their renderer decided to assemble a thread today.

// Every parent we have ever replied to, from our own repo.
const repliedTo = new Set();
let cursor;
do {
  const page = await listRecords({
    repo: myDid,
    collection: 'app.bsky.feed.post',
    limit: 100,
    cursor,
  });
  for (const rec of page.records) {
    const parent = rec.value?.reply?.parent?.uri;
    if (parent) repliedTo.add(parent);
  }
  cursor = page.cursor;
} while (cursor);

const open = conversational.filter(n => !repliedTo.has(n.uri));

The same shape works anywhere. On Reddit it is your own comments and their parent_id. On a mail API it is your sent folder and the In-Reply-To header. The lookup is cheap, and it is derived from something you control.

The asymmetry that should drive the design

These two failures are not equally bad, and that should decide how you build the check.

A false "already answered" means you miss a reply. That is embarrassing and recoverable, and a human would forgive it.

A false "still open" means you reply to the same person twice. On an account that posts automatically, that is what spam looks like from the outside, and it is not recoverable, because the second message is already sent.

So when the two signals disagree, believe the one that says you already answered.

Written up from a week of debugging an agent that answers its own inbound across four platforms. Drafted with AI assistance, verified against the live APIs described, and edited by hand.

Durable approval is not the same as valid approval

Jackson Ly — Fri, 07 Aug 2026 19:31:52 +0000

A few weeks ago we shipped a general command tool for a local agent, with a deny-list in front of it and a human approval gate behind it. I wrote at the time that the deny-list is the part that looks like engineering and does not hold, because you cannot enumerate the dangerous set. The gate is the part that holds.

Someone pushed back with a good refinement. The gate only holds if approval is real state with a lifecycle: requested, then approved or rejected, then consumed, then expired. A confirmation step living inside the agent loop cannot survive the process dying while it waits, so it is not a control, it is a prompt.

He is right, and most implementations skip it. What I want to add is that making approval durable does not finish the job. It moves the failure somewhere quieter.

An approval is granted against a world, not a string

When someone clicks approve, they are approving the specific thing they were shown: this file, with this content, going to these two people. What you persist is usually a record that a decision happened, plus an identifier. That is not a record of what was decided.

Between approve and consume, the world keeps moving. The file gets renamed, or a symlink under it now points somewhere else, or the recipient list picked up one more address while the process was down. Your token survives all of it, because it was never bound to any of it. The agent resumes after the crash exactly as designed and executes something the human never saw.

The old name for this

Time-of-check to time-of-use. You verify a condition, then act on it, and something changes in the gap. The textbook examples are filesystem races measured in microseconds.

Human approval stretches that gap to minutes or hours, because a person gets the prompt, goes to lunch, comes back and clicks yes. Durability stretches it further on purpose, since surviving a restart is the entire feature. We took a race condition and made it comfortable enough to design around.

The fix nobody enjoys building

Bind the approval to a fingerprint of what was actually rendered to the human. Not the intent, the literal resolved parameters. Recompute it at consume time, compare, and fail closed on any drift. Then expire aggressively, since expiry is the only real bound on how stale a decision can get.

Fail closed is the part I keep seeing people soften, and I understand why. The drift is almost always innocent, a timestamp or a reordered field, and failing on it feels pedantic. It is pedantic. It is also the one behaviour that makes the gate mean anything on the day the drift is not innocent.

Why this is worse than losing the approval

If approval evaporates when the process dies, you get a bad experience and a safe failure. The agent comes back, finds nothing, and asks again. The human sees the current state and can say no.

If approval survives unbound, you get a good experience and a silent failure. The agent finds a valid-looking token and proceeds against a world that moved. Nobody is asked anything. Nothing looks wrong in the logs, because from the system's side everything worked exactly as specified.

So durability on its own is not a safety property. It is a convenience that only becomes a safety property once the persisted thing is specific enough to check.

What I would do differently

Bind before you persist. Ship the approval carrying a hash of its resolved parameters from day one, even if the storage is a flat file and the lifecycle has two states. I would rather have a crude gate that revalidates than a carefully modeled lifecycle that trusts its own token, and I say that having built the second one first.

The lifecycle framing is right. It is just the easy half.

Written with AI assistance.

Running local and cloud models in the same coding agent: what actually ships in 2026

Jackson Ly — Fri, 07 Aug 2026 03:09:57 +0000

Someone on r/LocalLLaMA asked a question this week that turned out to be harder to answer than it looks: which open-source coding harness lets a local model and a cloud model work on the same task, together, in one loop?

The answers he got were a proxy and a shrug. He rejected both, correctly. So here is the actual answer, with the config knobs, and an honest account of the part that does not exist yet.

Why a gateway is not the answer

The first suggestion for this is always LiteLLM, or Bifrost, or whatever OpenAI-compatible proxy is current. Those are good at what they do. They are not what the question is asking for.

A proxy routes on what it can see in the request: the model name you asked for, weights you configured, health checks, budgets. It sits below the agent and has no idea whether the call it is forwarding is a throwaway commit message or the one architectural decision in the run. Point an agent at a gateway and you get failover and cost control. You do not get a local model and a cloud model dividing labour on a task, because nothing in the request says what the task needed.

The signal you want lives one layer up.

What actually ships, by declared role

Three tools do this today. None of them frame it as routing, which is why searching for "routing" misses them. They all frame it as roles.

Cline lets you set one model for Plan mode and a different one for Act mode. In settings, enable "Use different models for Plan and Act", then pick a model for each. Switching modes switches the model automatically, so a single task can be planned by one model and executed by another without you touching anything mid-run. The documented pairing is a stronger reasoning model for planning and a faster one for implementation, and nothing stops you pointing one of those at a local endpoint.

Aider takes three models in one session. --model is the main one that does the editing. --editor-model handles the edit-application step. --weak-model picks up the cheap work like commit messages and history summarization. You set them independently, so a large cloud model can do the reasoning while a small local model absorbs the chatter that would otherwise burn tokens.

Continue assigns models per role, and this is the closest thing to the original question. Autocomplete is a separate role from chat. Their docs cover running the autocomplete model locally through Ollama, and chat can sit on a cloud provider at the same time. That is genuinely a local model and a cloud model both live in one workflow, serving different jobs, all day.

So the honest answer to "does anything blend local and cloud in one loop" is yes, three things do, and they have shipped it for a while. It just is not called routing.

The pattern, and the part nobody has

Look at what those three have in common. In every case, you declare the split once, and the harness honours it for the rest of the run. Plan gets this model. Autocomplete gets that one. Commit messages get the cheap one.

What none of them do is decide the split themselves. No shipping harness looks at a subtask, works out that this one needs vision or a 400k context window or tool calling that survives twenty turns, and picks accordingly.

That gap is not an oversight, and it is worth understanding why, because it tells you what to build if you are going to build it. The requirement is known inside the planner, at the moment a subtask is created. By the time a model is selected, that context is gone: the selection layer sees a prompt and a model name. Nobody has plumbed the requirement from where it is known to where it is needed. Until someone does, autonomous model selection has nothing to select on.

If you are wiring this yourself

Two rules that survive contact with real work.

Route on declared capability, never on estimated difficulty. Capability is checkable: this subtask has an image in it, this one exceeds the small model's context, this one needs a tool call. Those facts stay true tomorrow. Difficulty is a guess, it is not measurable at the point you need it, and every heuristic you write for it goes stale on the next model release. A rule that rots silently is worse than no rule.

Put the cheap model where the work is bounded and verifiable. Commit messages, summarization, autocomplete, structured extraction with a schema you validate. In each case, a bad output is obvious immediately and costs one retry. That is why the shipping tools all made the same choice: the roles they hand to the weak model are exactly the roles where being wrong is cheap and detectable.

The interesting version of this problem is not building a smarter router. It is getting the planner to say what each piece of work actually needs, in a form something downstream can act on. Everything else follows from that.

Sources: Cline's Plan and Act docs, Aider's advanced model settings, and Continue's autocomplete deep dive, all checked while writing this.

Your interval-triggered agent pays full price for empty runs

Jackson Ly — Wed, 05 Aug 2026 17:31:39 +0000

We run an agent on an interval. It wakes about a dozen times a day, checks whether anything needs doing, and usually the answer is no. That "usually no" turned out to be the whole cost problem, and it took us longer than it should have to see it.

An interval trigger has no idea whether there is work. It fires exactly as often on a dead Tuesday as during a busy hour. So if the first thing your wake path does is the full sweep, you pay full price for every empty tick. We counted six fires in about two hours and twenty minutes, and half produced nothing but a "nothing changed" log line. Each of those still ran the complete orientation: read state, check five external services, recompute derived metrics, refresh performance snapshots. Minutes of work and real API spend to conclude the world was exactly as we left it.

Put a cheap guard in front of the expensive path

Before doing anything real, answer one question as cheaply as you can: has anything plausibly changed since last time?

For us that is a single indexed query against our own activity log.

SELECT run_session, max(ts)
FROM action_log
GROUP BY run_session
ORDER BY max(ts) DESC
LIMIT 1;

If the last run was under twenty minutes ago, we make exactly one external check, the cheapest signal we have that new work exists. If it comes back empty, we write two rows to the log and exit. No scouting, no recomputation, no metrics refresh.

last_run = db.last_activity_ts()
if now - last_run < timedelta(minutes=20):
    if cheapest_external_signal() == 0:      # one HTTP call
        log("monitor", "no-op")
        log("skip", "fast-exit, nothing changed")
        return
# ... otherwise fall through to the full pass

The guard reads local state first, because local state is free and the network is not. And the tiebreaker should be the single cheapest external call you have rather than a representative sample of your integrations. The goal is a yes or no answer for close to zero cost, not an accurate picture of the world.

Choose the threshold from how fast your input actually accrues

Twenty minutes is not a magic number. It came from watching how our inbound behaves: replies arrive over hours, so two wakes fifteen minutes apart genuinely cannot differ. If your agent watches a build queue that changes every thirty seconds, twenty minutes is absurd. Match the window to the thing you are watching, and write down why you picked it, because the next person to read the code will assume it was arbitrary.

Do not let the guard trust a cached answer it never verified

This one is subtle enough that we shipped it without noticing.

Our agent keeps a human-readable file of things that are blocked: expired sessions, missing permissions, failing integrations. That part is fine. The bug was that the agent started reading that file to decide what to skip. One entry said an API key lacked write scope, which was true on the day it was written. We skipped that integration on every run afterwards on the strength of the note.

When we finally tested it live, the write succeeded on the first attempt. The key had been fine for who knows how long. Nothing had failed, so nothing had corrected the note, so the agent went on believing it.

The rule we settled on is that a block is a fact with a timestamp rather than a permanent state. If a guard is going to skip work, it has to confirm the reason still holds right now, and that confirmation should be cheap enough that there is no excuse to skip it. A note tells humans what happened. It should never be an input to control flow.

# wrong: the note decides
if "devto" in blocked_notes:
    skip("devto")

# right: the note informs, the live check decides
if not devto.can_write():        # one API call
    note_block("devto", "write scope missing")
    skip("devto")

Make state durable per run, not per tick

The other thing that bites scheduled agents is interruption. A cron-triggered process that dies halfway leaves no trace of how far it got, so the next tick starts from zero and redoes work that already succeeded. If any of that work had side effects, they happen twice.

Persist progress at the step level, keyed by run, and have the wake path check for an unfinished run before starting a new one. Resuming a half-finished pass is almost always cheaper than repeating it, and it turns a crash from a correctness problem into a latency problem.

This is also what separates an agent that can ask a human for approval from one that cannot. If approval lives in memory inside the process, the process dying takes the pending decision with it. If approval is durable state the run can be resumed into, someone can answer twenty minutes later from their phone and the work carries on.

Where we landed

Guard the expensive path with the cheapest check that can answer no. Read local state before touching the network. Confirm a block is still true instead of inheriting it from a note. Keep per-run state durable so an interrupted pass resumes.

The agent is no smarter for any of this. The idle case just costs almost nothing now, and for us the idle case is most of them.

Human in the loop is not a switch: classify the commands

Jackson Ly — Sun, 02 Aug 2026 23:43:21 +0000

Human in the loop is usually implemented as a switch: approvals on, or approvals off. Both settings are wrong for an agent that runs shell commands, and you find that out the same way everyone does, which is by watching a long task die on its third ls.

The stall looks like this. Your agent is working through something multi-step, it needs to look at a directory, the harness raises an approval, and the run stops dead waiting for a human who has gone to make coffee. Multiply that by forty tool calls and the agent is no longer autonomous, it is a very expensive interactive shell.

So people reach for one of two fixes, and both make it worse.

The two fixes that are not fixes

Turning the gate off. Now rm -rf runs unattended too. The gate was not the problem, and you have removed the only thing standing between a plausible-looking command and an irreversible action.

Wrapping the tool in something else. This is the one I see most: the agent stalls on the built-in shell tool, so someone routes it through a different tool layer and the stall goes away. It goes away because the new layer does not ask. You have not removed the approval, you have moved who is responsible for asking, and usually the answer is now nobody.

The interrupt is not a bug. It is the gate doing its job. The bug is that the gate cannot tell ls from rm.

Classify by reversibility, not by tool

The useful axis is not which tool is being called, it is whether the call can be undone. Reads are recoverable by definition, since the worst case is that you read something and discard it. Writes, deletes, and anything that leaves the machine are not.

So the gate gets a classifier, and the classifier decides who has to wake up:

READ_ONLY = {
    "ls", "cat", "head", "tail", "wc", "stat", "file", "find", "grep", "rg",
    "git status", "git log", "git diff", "git show", "git branch",
}

RISKY_PREFIXES = ("rm", "mv", "dd", "chmod", "chown", "kill", "curl", "wget",
                  "git push", "git reset", "npm publish", "docker rm")

# Anything that lets one approved command smuggle in another.
SHELL_OPERATORS = ("&&", "||", "|", ";", "`", "$(", ">", ">>", "\n")


def needs_human(command: str) -> bool:
    cmd = command.strip()

    # A command containing shell operators is not one command, it is several.
    # Do not try to reason about the pieces. Ask.
    if any(op in cmd for op in SHELL_OPERATORS):
        return True

    if any(cmd == allowed or cmd.startswith(allowed + " ") for allowed in READ_ONLY):
        return False

    # Everything not explicitly known-safe requires a human, including
    # commands that merely look harmless.
    return True

Two things about that function matter more than the word lists.

It is an allowlist, not a blocklist. A blocklist is a bet that you thought of every dangerous command, and you did not. RISKY_PREFIXES above is there for logging and for showing the user why something is being escalated; it is not what makes the decision. The decision is made by the final return True, which is the deny-by-default line. Delete that line and the whole thing is decoration.

It refuses to parse compound commands. ls && rm -rf build starts with an allowlisted token. If your check is startswith, you just approved a delete. You can go and write a real shell parser, or you can treat the presence of an operator as an automatic escalation, which is a two-line rule that does not have parser bugs.

The asymmetry is what makes this safe

Every classifier is wrong sometimes, so the question is what its errors cost.

If the classifier calls a safe command risky, the user gets one unnecessary approval prompt. Annoying, bounded, visible.

If it calls a risky command safe, something irreversible happens with nobody watching. Not bounded, and often not visible until later.

Those two are not comparable, which is why the default branch has to be "ask" rather than "allow". You are not trying to build a classifier that is right. You are trying to build one whose mistakes all land on the cheap side, and then curating the allowlist until the cheap mistakes are rare enough to live with.

This is also the reason the allowlist should be boring and explicit rather than clever. Every entry you add is a small, deliberate decision that this exact thing can run while you sleep.

What this looks like in practice

We build recal, a local-first assistant on macOS, and its command tool works this way: read-only commands execute unattended, everything else raises an approval that carries the actual command text so the user is approving a thing rather than a category. Long tasks stopped stalling, and the approvals that remain are the ones worth reading.

The honest catch is that the allowlist is maintenance. New tooling shows up, someone's workflow needs jq or kubectl get, and the list has to grow. There is no version of this where you write the classifier once. What you get instead is a system where the cost of being wrong is a prompt instead of a restore from backup, and that trade has been worth it every time.

The other honest catch: this does nothing about a command that is individually safe and collectively terrible. Forty approved writes in the wrong directory is still forty approved writes. Reversibility classification bounds the blast radius of a single call, not of a plan.

If you are building one of these

Start with deny-by-default and an allowlist of four or five read commands, then add entries when a real task trips on one. Log every escalation with the command text, because that log is what tells you which entries to add next. And resist the urge to make the classifier smart. The value here is not intelligence, it is that the failure mode is a question rather than a deletion.

Written with AI assistance and edited by a human. The allowlist-over-blocklist and compound-command guidance follows the security notes in Anthropic's bash tool documentation; the rest is what we shipped and what it cost us.

Your cache_read_input_tokens is zero. Here is what silently did it.

Jackson Ly — Thu, 30 Jul 2026 05:32:37 +0000

Your cache_read_input_tokens is zero. Here is the list of things that silently did it.

You added cache_control, the docs said up to 90% cheaper, and the counter still reads zero on every request. Nothing errored. There is no warning. The cache just never gets read.

This is almost always one of a small number of specific mistakes, and they are all instances of a single rule.

The one rule

Prompt caching is a prefix match. The cache key is the exact bytes of the rendered prompt up to each breakpoint. Any byte that changes anywhere in the prefix invalidates the cache for every breakpoint at or after that position.

Render order is:

tools  ->  system  ->  messages

That ordering is the whole game. tools render at position zero, so a change there invalidates everything. A breakpoint on the last system block caches tools and system together.

Once you internalise "prefix match, tools first", every item below is predictable rather than surprising.

The audit list

Grep your prompt-assembly path for these. In my experience the first one accounts for most zero-hit reports.

In the prefix	Why the cache never hits
`datetime.now()` / `Date.now()`	Prefix differs on every request
`uuid4()` / a request ID	Same, and unfixable by retrying
`json.dumps(d)` without `sort_keys=True`	Key order is not guaranteed, so the bytes move
Iterating a `set` to build content	Iteration order is not stable
Session or user ID in the system prompt	Per-user prefix, zero sharing across users
`if flag: system += ...`	Every flag combination is a separate prefix
`tools=build_tools(user)`	Tools render first, so a per-user tool set caches for nobody

The fix in every case is the same shape: make it deterministic, move it after the last breakpoint, or delete it if it is not load bearing.

A "current date" line in the system prompt is the classic. It feels harmless because it is one short line. It sits in front of everything.

What does not invalidate everything

This is the part people over-correct on. There are three cache tiers, and a change only invalidates its own tier and below:

Change	Tools	System	Messages
Tool definitions added, removed, reordered	gone	gone	gone
Model switch	gone	gone	gone
Web search or citations toggle	kept	gone	gone
System prompt content	kept	gone	gone
`tool_choice`, images, thinking on/off	kept	kept	gone
Message content	kept	kept	gone

So you can flip tool_choice per request, or toggle thinking, without losing the tools and system cache. Only tool-definition changes and model switches force a full rebuild.

Two of those rows have an escape hatch, and they are not gated together:

System prompt content. Instead of editing top-level system mid-conversation, append a {"role": "system", "content": "..."} message to messages. It lands after the cached history, so the prefix survives. Available today on Claude Opus 5, Opus 4.8, Fable 5 and Mythos 5, with no beta header. Unsupported models return a 400, so catch it and fall back.
Tool definitions. On Claude Opus 5 there is a beta (mid-conversation-tool-changes-2026-07-01) that adds and removes tools between turns via tool_addition / tool_removal blocks, without invalidating the prefix. Tools you plan to add must already be declared with "defer_loading": true.

Model switching has no escape hatch. Caches are model scoped. If you want a cheaper model for a sub-task, spawn a subagent rather than switching the main loop.

The minimum is not monotonic

There is a minimum cacheable prefix. Below it, nothing caches, you get no error, and cache_creation_input_tokens is just zero.

The trap is that newer does not mean lower:

Model	Minimum
Opus 5, Fable 5, Mythos 5	512
Opus 4.8, Sonnet 5, Sonnet 4.6, Sonnet 4.5	1024
Opus 4.7	2048
Opus 4.6, Opus 4.5, Haiku 4.5	4096

A 3K-token prompt caches on Opus 5 and Opus 4.8 and silently does not on Opus 4.6 or Haiku 4.5. If you wrote a prompt off as "too short to cache" on an older model, it may be worth re-checking.

Two timing gotchas

The 20-block lookback. Each breakpoint walks backward at most 20 content blocks looking for a prior cache entry. Agentic loops blow through that easily, because one turn can add a dozen tool_use and tool_result pairs. If a single turn adds more than 20 blocks, the next request's breakpoint does not find the previous cache and silently misses. Fix: place an intermediate breakpoint every ~15 blocks in long turns.

Concurrent requests all miss. A cache entry becomes readable only once the first response begins streaming. Fire N identical-prefix requests in parallel and all N pay full price, because none of them can read what the others are still writing. For fan-out, send one, await the first streamed token, then fire the rest.

Reading the counters honestly

Three fields, and the third one misleads people:

cache_creation_input_tokens: written to cache this request, billed at a premium
cache_read_input_tokens: served from cache, billed at roughly a tenth
input_tokens: the uncached remainder only

Total prompt size is the sum of all three. If your agent ran for an hour and input_tokens shows 4K, the rest was served from cache. Check the sum, not the single field, before concluding anything about prompt size.

Is it even worth it

Reads cost about 0.1x. Writes cost 1.25x on the 5 minute TTL and 2x on the 1 hour TTL.

So break-even depends on which TTL you picked. With 5 minutes, two requests pay for it (1.25 + 0.1 against 2 uncached). With 1 hour you need at least three (2 + 0.2 against 3). The 1 hour TTL keeps entries alive across gaps in bursty traffic, but the doubled write means it needs more reads to come out ahead.

And if the first 1K tokens of your prompt genuinely differ every request, do not cache. You will pay the write premium for zero reads. Leaving it off is the correct answer.

Pre-warming, and when not to

You can pay the cache write up front with a max_tokens: 0 request. Prefill runs, the cache is written, and you get back content: [] with stop_reason: "max_tokens" and zero output tokens billed.

Put the breakpoint on the last block shared with the real request, usually the system prompt or the tool definitions. Not on the placeholder user message, and not via top-level automatic caching, which would key the cache to the placeholder.

It is worth doing when first-request latency is user visible, the shared prefix is big enough that a cold write is noticeably slow, and there is a moment before traffic to fire it: app startup, worker boot, post-deploy.

It is not worth doing when traffic is continuous. If real requests arrive more often than the TTL, they keep the cache warm on their own and a separate warm call is a pure extra write. Scheduled re-warms only make sense when your traffic has gaps longer than the TTL.

Note that max_tokens: 0 is rejected alongside stream: true, thinking.type: "enabled", output_config.format, a forced tool_choice, or inside a Batches request.

The debugging loop

If the counter is zero, do this rather than adding more markers:

Capture the fully rendered request body on two consecutive calls.
Diff them.
The first differing byte is your answer.

That diff is faster than reasoning about it, and it turns "caching is not working" into a one-line fix nearly every time.

Written with AI assistance and human review. Behaviour described reflects the Claude API as documented in July 2026; caching internals move, so verify against current docs before you rely on a specific number.

Your baseline scored 0.000. That's a broken harness, not a result.

Jackson Ly — Sat, 25 Jul 2026 03:34:51 +0000

Your baseline scored 0.000? Before you publish the win, here is the checklist I now run, because a zero from a baseline is almost never a result. It is usually your harness.

This week I watched a builder on r/Rag do something rare: he benchmarked his own memory library against plain RAG expecting a win, got three null results, and published the confounds instead of the victory. Two of his bugs are so common, and so quiet, that I think every RAG or agent-memory benchmark should gate against them by construction. This post is the checklist that came out of that thread.

Bug 1: your arms have different context budgets

His first run compared a memory arm retrieving k=20 sentence-level hits against a BM25 arm returning whole sessions. Same "top-k" on paper. In characters, one arm got 1.3k of context and the other got 11.9k. BM25 looked dramatically better.

Once he matched the budget, accuracy went 0.28 to 0.59 for the memory arms and the ranking flipped. The original result was a budget difference wearing a granularity costume.

The fix is embarrassingly simple: print the context size, in characters or tokens, next to every accuracy number, per arm. If a comparison does not state budget parity, the number does not mean anything. Chunked-vs-whole-document comparisons are especially prone to this because "top-k" hides a 10x budget difference in plain sight.

Bug 2: the baseline that scored 0.000 with clean logs

He ran a competing memory tool as a baseline. It scored 0.000. He re-ran it with a stronger extraction model. 0.000 again, logs clean. Very tempting to publish "competitor discards memories under load."

It was his bug, twice. His harness truncated each session to 6,000 characters before ingestion, cutting off the injected evidence. And he was passing limit= to an API whose parameter is top_k=, so his setting was silently ignored.

That second one deserves its own paragraph, because it is everywhere in this ecosystem: APIs that accept unknown kwargs without raising. They make a broken configuration look like a clean loss. In a benchmark, treat any client that swallows unknown parameters as hostile. Do not trust that a parameter did something. Assert it: if you move top_k from 5 to 20, the retrieved count has to move. If it does not, your harness is configuring nothing.

The three gates that make results trustworthy

Out of that exchange came a harness structure I now consider the minimum for memory and RAG benchmarks. Three separate liveness gates, each with its own abort message, all pre-registered before the run.

1. A positive control per arm. Every arm gets a probe it cannot fail, built from the corpus's own ground truth. If any arm scores below a threshold you wrote down before the run, the whole benchmark aborts as "dead harness," not "clean loss." This is the gate that catches truncation bugs, auth failures, and empty stores. A zero is never evidence until this gate passes.

2. An evidence-in-context ceiling. For each probe, check whether the gold evidence is actually present in the retrieved context. Aggregated, this is the recall ceiling you report next to accuracy. In his broken run it was 3.5 percent. You cannot out-rank evidence that was never retrieved, and no reranker or prompt change will save a run whose ceiling is on the floor. Cheap to compute, needs no LLM calls.

3. A parameter-efficacy assertion. For each knob the benchmark claims to vary, assert that observable behavior changes when the knob moves. This is the gate that catches the swallowed-kwargs class of bug, and it is the one nobody runs.

Why three gates and not one

The temptation is to fold these into a single sanity check. Do not. The whole value of a gate is that its failure names the broken layer. Evidence-in-context failing means retrieval never delivered. The positive control failing means the store or the answerer is dead. The efficacy assertion failing means your configuration is fiction. One merged gate tells you "something is wrong" and leaves the first hour of debugging still ahead of you.

There is a general lesson under all of this. A benchmark harness is software that lies by default, because every bug it has produces a plausible-looking number instead of a crash. The discipline that fixes it is the same one that fixes agent evaluation generally: never trust a claim the system makes about itself unless something the system cannot touch verifies it. A test that has never been proven to fail will happily pass forever.

Credit where it is due: the null-result writeup that prompted this is worth reading in full, including the part where he catches himself half-writing a false finding about a competitor before his own positive control saved him. Publishing that takes more spine than publishing a win.

I build a local-first assistant for the Mac, which is why I spend my days in retrieval evals. No product pitch here; the checklist stands on its own.

Our Rust file watcher ate 23.6 GB of RAM, and our ignore rules never had a chance

Jackson Ly — Mon, 13 Jul 2026 13:40:17 +0000

We build a local-first indexer for macOS. It watches the user's home folder so that when a file changes, we reindex just that file. Watching one project directory is a solved problem. Watching a real human's $HOME is a different sport, and it broke us in a way that took a heap profiler to see.

The symptom

The app sat at roughly 25 GB of resident memory. Not a slow leak either. It climbed there shortly after launch and stayed. One core also pinned itself for about ten minutes at every boot.

The obvious suspects were wrong. It was not the embedding model, and it was not our index. We had a careful ignore list (caches, node_modules, Library, build output), so surely we were not touching millions of files.

We were not. But something else was.

The finding

We use the excellent notify crate, specifically notify-debouncer-full, which is what you reach for when you want raw filesystem events collapsed into something sane. The setup you copy from the README uses RecommendedCache.

RecommendedCache resolves to FileIdMap. From the docs, the debouncer "can optionally keep track of the file system IDs all files and stitches rename events together". That is a genuinely useful feature: on macOS FSEvents and on Windows, a rename shows up as two unrelated events, and pairing them requires knowing that the file at the old path and the file at the new path are the same inode.

To do that, it has to know the file ID of every file under the watch root. So it walks the entire watch root and caches a (PathBuf, FileId) for every entry it finds.

Here is the part that mattered: that walk does not know about your ignore rules. Our ignore list filters events. The cache is built underneath that, from the watch root down. We told the debouncer to watch $HOME, so it faithfully walked all of $HOME, including everything we thought we had excluded, following symlinks as it went.

Under malloc_history, the damage was specific:

23.6 GB of live heap in the cache
4.16 GB of that in the hashbrown table itself
19.4 GB spread across 54.2 million path allocations
roughly 6.4 million files stat-ed at boot, which is the pinned core and the ten minutes

54 million allocations to remember the inode of files we had explicitly said we did not care about.

The fix, and what it costs

We switched the cache to NoCache.

That is not free, and it is worth being precise about the tradeoff rather than pretending we outsmarted the library. FileIdMap exists for a reason: with NoCache, the debouncer can no longer stitch rename events together. A rename stops arriving as "this moved from A to B" and degrades into an unrelated delete at A and a create at B.

We could absorb that, for two reasons that are specific to our design:

Our indexer is idempotent. Indexing the same file twice converges to the same row, so a spurious create is cheap and harmless.
We already run a rescan and reconcile pass that removes index entries whose files no longer exist. Deletions get collected there regardless of whether the watcher paired them correctly.

So a rename becomes delete-plus-create, and the system settles into the right state on its own. If your indexer is not idempotent, or you have no reconcile pass, this trade is not available to you and you should fix the scope instead.

Result: 24.8 GB down to 1.45 GB.

The second bug we found on the way

While measuring, we caught something else. When FSEvents drops events (it does this under load, by design, and tells you so), the debouncer emits a rescan. Our code fanned that out as one rescan per configured root.

Our roots were nested. We watched $HOME, and we also watched Documents, Downloads, and Desktop because they were configured separately. So a single dropped event triggered four concurrent, roughly $HOME-scale re-walks, and every file under the nested roots got embedded twice. You could see it plainly in the logs: two EmbeddingsGenerated lines per document.

The fix was to de-nest the roots into independent subtrees before emitting any rescan, so overlapping configuration collapses into one walk.

The lesson worth stealing

Library defaults are calibrated for the common case, and the common case is a project folder with a few thousand files. Nothing in the API warns you, because nothing is wrong with the API. FileIdMap is correct. It is correct at 5,000 files and it is a memory bomb at 6.4 million, and the difference is entirely in what you point it at.

So, two things:

Know what your watcher caches before you widen the scope. Filtering events is not the same as filtering what the library indexes internally. Ours sat below our filter, where we could not see it.

Measure the heap, do not reason about it. We would never have found 54.2 million path allocations by reading code. malloc_history found it in one pass. Every hour we spent theorizing about the embedding model was an hour we did not spend attaching a profiler.

If you are watching a user's whole home directory, assume every default in your stack was written by someone who was not.