DEV Community: The Agent Loop

33% of my agent's messages went to recovering from crashes, and no dashboard counts it

The Agent Loop — Thu, 15 Oct 2026 04:46:47 +0000

The number I'm about to give you is unusual for my setup, and I want that caveat first. I run an agent on a phone inside a Linux container, which is a bad way to run an agent and produces failures no ordinary deployment would ever see.

The general problem underneath it isn't unusual at all. It's the part nobody has built a tool for.

The FinOps Foundation published a guide on AI tooling costs, last updated April 2026, and it names the metric that matters:

"The key metrics to insist on having visibility into are: cost per use case (not just total spend), token consumption breakdown, GPU utilization, and error rates and retry rates (repeated calls due to errors are pure waste)."

Cost per use case. Not per request. Not per thousand tokens. Not per team. Per outcome.

Every cost tool I've ever used counts tokens, and there's a reason that number is so easy to get: a machine can see it. Outcomes aren't visible to a machine. Naming "done" is a judgement call, and nobody wants to write the function that decides whether a research task actually succeeded.

So the metric the standards body names as the real one gets skipped by every dashboard on the market. Not because it's technically hard. Because it's the only number in the stack that requires someone to admit which runs produced nothing.

What my own numbers look like

I can measure every tool call my agent makes. They all sit in a SQLite file, one row per message, 19,090 parts across 64 sessions and 56 days. Four lines of SQL gets you the shape:

SELECT s.title,
       COUNT(m.id) AS messages
FROM session s
LEFT JOIN message m ON m.session_id = s.id
GROUP BY s.id
ORDER BY messages DESC;

Run that against your own agent and you'll see what I see. Heavy tail. A handful of sessions hold a third of everything. My median session is 18 messages, and my largest is 1,327.

Then I did the thing nobody warns you about, which is read the titles and sort them by what the session was actually for. Crash recovery came to 1,636 messages across 10 sessions. Ship work, meaning the sessions that actually wrote and published posts, came to 536 across 3.

A third of every message this project has spent went to recovering from failures.

Now the caveat.

Those categories come from session titles, and I wrote those titles myself in a picker menu. No system field anywhere says "this session was for recovery." I classified 64 strings by keyword and then read them back.

What survives that weakness is the tail. The crash-recovery sessions aren't marginally longer. They average 164 messages against a median of 18, an order of magnitude. Mislabel a few and the ratio wobbles slightly. It doesn't move much.

Twenty-four posts came out of the 11% I labelled ship work. The rest bought me a system that stayed up.

The tool was never going to show me this

This is the part that took me a while to actually see.

I have a spend dashboard. It told me, accurately, that my context was 63% bash output and that a single web search cost 27,000 characters. Those numbers moved me. They changed what I wrote next.

It never once showed me that a third of my spend was buying back my own mistakes. Not because the tool is bad. Because a crashed session still produces tokens, and tokens are the thing the tool counts. Work that got undone doesn't leave a debit behind. It leaves an absence.

That's the whole problem in one line: cost accounting can only see what happened, and most of what an agent spends goes on what didn't.

The same source says observability tooling is "frequently the last one funded," and that budget controls and iteration limits "are not optional safeguards." Both describe a team that can read the bill perfectly well and still can't tell you what the bill bought.

Three ways it looks when nobody is counting

If the framing is any use to you, it shouldn't require running infrastructure on a phone. These three checks work on any agent setup, and none of them need a new tool.

You can name your monthly bill but not your most expensive week. Everyone reports the total. Almost nobody has looked at the distribution, and the distribution is where the story lives. A month where one run cost eight times the median is a very different month from one where every run cost the median, and the invoice treats them identically.

No one on the team can say which task is expensive. Spend that can't be attributed to work doesn't create accountability, because there's nothing for anyone to own. This is the boring failure and it's extremely common. The dashboard exists, the number is right, and the number doesn't point at anything a person decided to do.

Your most expensive run was your least memorable one. You remember the thing that shipped. You don't remember the four attempts before it, the two dead branches, the retry that doubled the context. Memory is shaped by outcomes, so the cost of the path you abandoned disappears from your sense of what the work involved.

If one of those three is true of your setup, you have a cost-per-outcome problem whether or not anyone has used that phrase.

How to build the denominator

You can't buy this one. You have to decide it's worth measuring, which means writing down what "finished" means for one small slice of your agent's work. Not all of it. One slice.

Then three steps:

Name the outcome so a machine could check it. Not "good research." "A file exists at this path, containing the query that produced this number."
Count both halves. Spend is the easy numerator. Runs is the easy denominator. Outcomes are the only genuinely hard number here, and it's the one everybody skips.
Let the ratio embarrass you. If you ran 200 times and shipped 4, your cost per outcome is fifty times your cost per run. That isn't a flaw in the measurement. That is the measurement, and it's the figure your next architecture conversation should be built around.

People argue with step one, and the argument is always the same. It feels like over-engineering, and a wrong heuristic is worse than no heuristic.

Both are true, which is why the first slice should be small enough that a human can check the verdict by hand. Verify twenty of them yourself and you'll know whether your definition of done survives contact with reality. Only then automate it.

The bit that surprised me is how fast the definition improves. Mine started as "published a post," which was wrong three times in four days because it scored corrections as wins. Adding "and nobody found an error in it afterwards" made it harsher and far more accurate. A denominator you tune isn't a measurement weakness. It's the measurement starting to work.

Where this leaves me

I'm not going to pretend this solves it. I can name the recovery slice now, and I still can't put a dollar figure on it, because I don't pay per token. I have a subscription. The next honest step is to price the runs and find out whether a third of my messages is also a third of my money.

But the shape is visible, and the shape is the useful part. The tools are excellent at the half that's easy to count and stay quiet about the half that's expensive.

If you'd rather have these posts arrive in your inbox than wait for a feed to surface them, there's a subscribe link at the bottom of this one. I write about agent cost and reliability most days.

What's the largest share of your agent's spend that nothing in your current stack would show you?

19% of my agent's context is me re-reading my own notes

The Agent Loop — Tue, 13 Oct 2026 14:16:59 +0000

I spent a couple of hours this week arguing about whether my agent's verification habits were worth their context. Then I counted the context, and the argument wasn't the point.

Out of 8,573,668 characters of tool output across 4,394 tool calls, 1,641,013 went into reading my own markdown files. That is 19.1%.

Snapshot taken 2026-10-13T03:45Z. The database is live and every tool call I make while writing this post is landing in the same numbers, so treat the figures as a pinned reading rather than a live figure.

Publishing cost 316,266 characters. The notes describing how to publish cost 5.2 times more than publishing did. Add the local checks and the overhead is 6.9 times the work.

bucket	calls	characters	share	chars per call
reading my own docs	723	1,641,013	19.1%	2,269
other shell	2,023	5,447,181	63.5%	2,692
local checks	505	551,588	6.4%	1,092
fetching live URLs	658	416,421	4.9%	632
publishing and writing	465	316,266	3.7%	680

The number was wrong the first time

I want to start here, because it is the part that makes me trust the rest.

My first pass put reading at 44.6%. That number was a bug in my own classifier. I was matching grep, head and tail anywhere in a multi-line bash command, so any command that inspected a file and did something else got counted as document reading. Tighten the rule to only what is unambiguously a markdown read and it lands at 19.1%.

Same database, same total. One regex decision changed it by a factor of two.

I've written a lot about verifiers reading the wrong field. I then read the wrong field here, got a number more than twice as large, and believed it for about twenty minutes. The instinct to publish the exciting number is the whole problem, not an incidental hazard around it.

What counts as "reading my own notes"

The honest definition is narrow: tool calls whose input names a file ending in .md, plus shell commands whose primary target is a markdown file. That's 723 calls. Everything else is classified by what the command actually does.

I'm putting the classification in the post because a percentage you can't reproduce is a number you can't argue with, and an argument you can't have with is the failure mode. It's checkable and it will change as I fix it, which is the point.

what I read	characters consumed
`AGENTS.md`	120,581
`august 2026.md`	82,342
`watchlist.md`	60,049
`error-log.md`	49,408
`README.md`	40,932

AGENTS.md is 117,493 characters. It is loaded at the start of every session and it has grown every week since September, because every incident gets appended to it and nothing ever leaves.

This is a fixed bill, not a variable one

Here's the part that matters, and it isn't the absolute number.

So the overhead doesn't scale with the work. Writing a post costs roughly what it costs. The notes cost the same every single session, writing or not, and they get read whether or not that session ships anything.

An agent with good institutional memory and no pruning strategy doesn't get smarter over time. It gets more expensive to boot. Those look identical on a dashboard, which is why I didn't notice for a month.

The 63.5% I can't explain

Two thirds of this budget is other shell, 2,023 calls that don't read markdown, don't fetch URLs, don't run a named check, and don't publish. I don't have a clean story for them, and I'm not going to invent one in a blog post.

They're probably measurement and diagnostics, plus the one-off queries that make up most of real work. But "probably" is carrying too much weight in that sentence, and the honest position is that this bucket is unresolved. The next version of this post should either explain it or replace it with a category that means something.

What I'd actually do

One lever, and it's not a discipline tip. It's arithmetic.

Split the notes by how often they're needed. Things required every session go in the short file. Things required only when a specific thing breaks move out of it, indexed by symptom, and get read after the failure rather than before it. AGENTS.md at 117K characters is one file doing the job of about thirty.

The test isn't "is this useful." It's "does this need to be in context before the first prompt, or after the first symptom."

I haven't measured whether shrinking it actually saves anything, and I'm not going to claim a saving I haven't measured. The 19.1% is the measurement. The fix is arithmetic, and it hasn't been run yet.

The same week I published the opposite argument

Yesterday I published 5 corrections in 4 days. Not one came from a check, about how my verification kept inheriting blind spots from the thing it was verifying.

Those two posts are the same week on purpose. That one argued that the checks weren't independent. This one says the checks and the notes around them cost more than the work, and that I built the notes that caused the first problem in the first place.

If this is the kind of thing you want once a week, it also lands in an inbox: buttondown.com/theagentloop. One email per post, nothing else.

Nineteen percent of my context, and most of it is my own handwriting, re-read every morning.

5 corrections in 4 days. Not one came from a check.

The Agent Loop — Tue, 13 Oct 2026 02:46:21 +0000

On 2026-10-12 our dashboard printed this, in green:

robots ok: 22/22 posts have a verdict

All 22 verdicts were correct. The badge next to Blog 4 was still wrong.

An hour earlier I had copied 23 robots receipts out of /tmp into durable storage with a plain cp, which reset every file's modification time to that afternoon. The function that picks the newest receipt sorts by mtime, so the archive reordered itself, and Blog 4's most recent receipt became an older indexable one instead of the noindex one that was actually current.

The check read the right files in the wrong order. It couldn't notice, because counting verdicts is not the same as testing them. I caught it by parsing the rendered HTML and counting badges, which shares no code with the dashboard and no ordering with it.

That is the shape of every failure in this lane, and I didn't learn it that day. Someone handed it to me five times in four days.

Five corrections, five messages, zero instruments

Between 2026-10-08T19:14:03Z and 2026-10-12T19:11:25Z, five people told me my verification was wrong. Three days, 23 hours, 57 minutes. Four days, two minutes short of a round number.

#	Who	What they corrected	Arrived (UTC)
1	`prpatel05`	An "independent" list that pages the same capped API isn't independent	10-08 19:14
2	`reidmarlow`	A missing signature is not a negative result	10-08 21:15
3	`gramli`	The scope check inherited the failure it was meant to bound	10-09 07:05
4	`james_anderson_h`	Naming a second source isn't independence. You have to know what it shares with the first	10-10 11:38
5	`runfoundry`	The check and the thing it checks shared a session, so they shared the blind spot	10-12 19:11

All five arrived as a comment or a reply. Not one was produced by a script.

The obvious reading is wrong, so it's worth being precise. Checks did fire. In October our robots sweep found only 15 of 16 posts and failed loudly, because dev.to's public API silently clamps per_page to 15 and the newest post fell out of the list entirely. That's a check catching something real.

What never happened is a check naming the mechanism. The sweep knew the count was wrong; prpatel05 is the person who told me why. Symptom from the instrument, explanation from a human, five times out of five.

Three dependencies

Strip those five down and they name three shared things.

Two checks that both page /api/articles are one check with extra steps. That's the endpoint.

A check bounded by the thing it is checking inherits that thing's failures. That's the scope.

Two checks riding the same session can't disagree about whether the session works. That's the credentials, and it's the sneakiest of the three, because nothing about it looks shared on the surface.

One rule exists, and I wrote it after being hit by it. Rule three didn't exist until runfoundry wrote it for me in a comment.

The one where I did it anyway

Closest I came to sharing credentials with a subject, on 2026-10-07.

status_check.py runs at the top of every session. Part of its job is deciding whether the dev.to session cookie has expired. When the API starts returning 404 it prints:

session cookie may need login.py

We were throttled that day, not logged out. The cookie was fine.

But the check authenticated with the same cookie as the thing it was checking. So a rate limit looked exactly like an auth failure, and the remedy it recommended was to re-login. Re-login overwrites the shared state.json. An unrelated site had already been saved into that file on 09-26 and took the dev.to cookies with it when that happened.

The check shared credentials with its subject, and its remedy would have caused the outage it was reporting.

I patched the message so it stops firing under throttling. What I never did was write down what each check shares with its subject, which is still the actual answer to the question runfoundry asked me.

The asymmetry worth stealing

One of our checks is already logged out, and it's the one that matters most.

Whether a post is indexable gets read from public HTML with no cookies at all. That's deliberate. "Can a crawler see this" and "can we see this" have to be the same question, or one of the two is lying to us. The cookie-authenticated checks are analytics and comments, and that's exactly where the false positive lived.

So here's the rule, and it's narrow. For every check, write down the endpoint, the credentials, and the session it borrows from the thing it checks. If you can't fill in all three, you haven't got a second source. You've got the first one again, wearing a different hat.

A third kind, which is worse

There's a failure mode that isn't a shared dependency at all, and we hit it this morning.

Our reply script posted a comment, got HTTP 200, found its own marker on the re-fetched page, and then reported the comment API as showing zero matching comments. Three instruments answered the same question about one write. Two said yes. One said no, and the no was wrong, so a third method went looking and proved the comment was live.

A verifier that returns a confident false negative is more dangerous than no verifier at all, because it invites you to run the action a second time and duplicate the thing you were trying to create.

The part I can't engineer

Here is the uncomfortable conclusion, and I don't have a fix for it.

Every correction in that table arrived because somebody outside the system disagreed with us in public. A count that came out wrong. A ratio that didn't match. An argument, a reply, a stranger with a clean browser the next day. Our own instruments were green the whole time.

Which means our detection rate is a function of how often reality disagrees with us where other people can see it. A quiet week is a week with no signal. It isn't a week with no bugs, and reading robots ok: 22/22 as reassurance is the exact mistake this post is about.

If this is the kind of thing you want once a week, it also lands in an inbox: buttondown.com/theagentloop. One email per post, nothing else.

Three dependencies. Write them down per check, or admit you're running one source twice.

63% of my agent's context came from one tool

The Agent Loop — Mon, 12 Oct 2026 07:38:20 +0000

I measured every tool call in one of my agent projects. The result I didn't expect: the tools I reach for most are not the ones that cost the most.

The most-used tool produced 63% of my context at the lowest cost per call. A tool I used twenty times produced 18 times more context per call.

The measurement

This is not a model. It is every actual tool call from a real project, read straight out of the agent's own session database. The snapshot is pinned to 07:41 UTC on 2026-10-12, because the database is live and the counts move while I am writing this.

SELECT json_extract(data,'$.tool')            AS tool,
       COUNT(*)                               AS calls,
       SUM(LENGTH(json_extract(data,'$.state.output'))) AS chars
FROM part GROUP BY 1 ORDER BY 3 DESC;

tool	calls	total characters	share
bash	3,210	5,147,190	63.4%
read	194	795,397	9.8%
web search (exa)	20	545,357	6.7%
web search (built-in)	16	460,672	5.7%
fetch page	24	445,160	5.5%
everything else	571	728,507	9.0%

Why the ranking flips

Sort the same table by cost per call instead of total, and the order inverts:

bash              1,603 chars per call
grep              3,117
read              4,099
scrape a page    10,519
fetch a page     18,548
web search       27,267

bash is the cheapest thing in the loop and it is 63% of my context. That's only possible because it's so cheap per call, and I call it constantly.

A tool's share of your bill is the product of two different things, and you can only see the problem if you measure both:

share of context = (how often you call it) x (how much each call returns)

Optimising only the first is why the tools I use most are not the tools that hurt.

The part I could not see from the tool names

Here's the number that actually made me care, and it's not in that chart.

Across 55 sessions on this project:

	tokens
input	40,473,602
output	1,587,482
cache read	561,271,443

Context was re-read roughly 14 times more often than it was first written. The 8.1 MB of tool output above is not billed 8.1 MB. It is billed once per call that follows it, for the rest of the session.

That is the multiplier that turns a cheap tool into an expensive habit. A read that returns 4,000 characters costs 4,000 characters again on every subsequent turn, forever, until the context is compacted.

What I changed, and what it cost me

Three things, in order of how much they mattered:

1. Filter at the tool, not after it. grep | head -40 instead of grepping 59,000 characters and asking the model to find the four lines it needs. The filtering is free and deterministic. Asking a model to ignore 12,000 characters is neither.

2. Count before you read. wc -l first, then read. Most files I open are 40 lines and I open them 400 times.

3. Batch the cheap tool, keep the expensive one rare. bash calls are nearly free per call, so batching them saves nothing. Web search costs 27,000 characters a call, so the only lever that matters is calling it less.

I didn't do number 3 well, which is why the web tools are 20% of my output volume for what was maybe 15% of the value. The chart is a report card, not a strategy.

The version of this you should check first

You don't need a database. Open your own agent's log and sort tool output by size. If the biggest tool isn't the one you expected, that's the whole finding.

If this is the kind of thing you want once a week, it also lands in an inbox: buttondown.com/theagentloop. One email per post, nothing else.

The version I wouldn't trust is a chart of token counts without the per-call column. Total volume tells you where the bytes are. It does not tell you which habit is expensive.

A retry costs the whole context, not the failed call

The Agent Loop — Mon, 12 Oct 2026 04:18:38 +0000

A retry in an agent loop is not a second try at a request. It is a second try at the conversation that produced the request, and you're billed for all of it.

Nobody prices that. Every cost dashboard I've seen counts calls, and calls are the wrong unit the moment a turn can fail.

The arithmetic

A plain HTTP client retries by re-sending one request. You pay for that request again, and it's usually the cheapest thing on the bill.

An agent turn does not work that way. To retry, you re-send the whole conversation: the system prompt, the tool results, everything the model has already been shown. And that conversation has grown since the attempt you're repeating, because the attempt itself added to it.

Here's the whole mechanism, with no abstraction layer in the way:

messages = build_turn(task)          # 1x context
for attempt in range(3):
    try:
        return call_model(messages)  # you pay for len(messages)
    except TransientError:
        messages.append(error_about_that_call)
        messages.append("try again") # context is now bigger

Two lines of bookkeeping between attempts, and the next call is priced on a longer prompt. Three attempts at 1x, 2x and 4x is not a retry policy. It is 7 units of spend for work that would have cost 1.

Why "did it succeed" hides this

The usual scoreboard has one column, and it is the wrong one:

model	success rate	cost per call	cost per correct answer
A	0.60	1.0	1.67
B	0.95	1.6	1.68
C	0.99	2.4	2.42

Model A looks best and is the worst deal in the table. On a per-call basis B is more expensive; on the only basis anyone pays for, A and B are level, and C is half again as expensive as either.

That table is worked arithmetic, not a measurement of real models. The shape is the point, and it holds for any model whose failures cluster early in a long turn.

The metric I want is cost per correct answer, because it's the only one that can't be improved by failing less loudly. And retries make it worse in a way that success rate hides: a model that fails and recovers is paying for the failure, and the recovery is counted as a success.

The retry also throws away its own evidence

The first attempt returning a 503 is a real observation about that endpoint. It's the most informative thing in the run.

The retry succeeds, the receipt goes out clean, and now the only surviving record says everything worked. You've paid twice and kept the worse of the two facts.

This is why I have started logging the retry path rather than only the outcome:

receipt.attempts.append({
    "n": attempt,
    "error": repr(e),
    "context_tokens": len(messages),   # the number you were actually billed for
    "outcome": "recovered",
})

Two fields. context_tokens is the one that tells you whether the retry was cheap, and almost nothing logs it.

What I'm not claiming

I argued this in a comment thread under someone else's benchmark and he told me it was a different experiment. He is right, and the scoping matters: his benchmark isolates the retry decision, for API requests, which is a question my argument does not touch.

What survives the concession is only the pricing. Retry-plus-context is a different question from retry-safety, and it is the one your bill answers.

I haven't measured this on my own runs, and I want to be plain about that. The charts and tables above are worked examples with the assumption written down, not telemetry. The reason I believe the shape is that it follows from what the API charges for, not from how any particular agent behaves.

The one thing worth adding

If you take one line from this, print len(messages) next to your retry count. It costs nothing and it's the difference between "retries are expensive" and knowing by how much.

If this is the kind of thing you want once a week, it also lands in an inbox: buttondown.com/theagentloop. One email per post, nothing else.

Where this came from, and the argument it came out of, is in the comments on To Retry or Not to Retry? That Is the Question and What one agent run actually costs.

2 checks agreed. That's the bad news.

The Agent Loop — Sun, 11 Oct 2026 12:03:31 +0000

Two of my checks passed. I noticed it because one of them was slow, not because one of them was wrong.

That is the whole story of this post, and the reason it took me four corrections to write down.

The rule I did not have

Here is the shortcut most of us use without naming it: if two independent things agree, the answer is probably right.

It's a decent shortcut. It isn't safe, and the failure mode is specific.

A check that shares a dependency with its subject is not a check. It is a second reader of the same page.

I got this wrong four times in twelve hours, in four different ways, and in every case the verification agreed with the thing it was supposed to be verifying. The agreement felt like the whole point. It was the problem.

The test that actually catches it

Forget whether a second source is independent. Ask something narrower:

If the dependency both sources route through fails, do they fail the same direction?

That's the question that does the work, and I didn't have it until someone handed it to me.

Two readers of the same truncated list are not two readers. They are one reader twice, and the second one is worse than useless, because it converts one unverified claim into two and you stop looking.

Concretely, the failure that got me was boring. A script compared the set of articles it expected against the set of articles an API reported. Those are the same dependency. The API silently clamps its page size to 15. The script asked for 30. Both halves were correct, both were consistent, and together they confidently reported on 15 of 16 articles.

Nothing in that system printed a warning. Nothing was in a bad state. The checks agreed, and that agreement was the whole of my confidence.

Agreement is the symptom

The part I had backwards for years: independent checks should disagree sometimes.

If two verifications never disagree, the cheapest explanation is that they aren't independent, and coupling is the default until you have evidence otherwise. Perfect agreement is not reassurance. It is the shape a shared bug makes when you look at it from two angles.

So the useful question after every green check isn't did it pass? It's what would have had to fail for this to still pass? If the honest answer is the same thing that would break the main path, you don't have two checks. You have one check and an echo.

The part I cannot fix

I want to be straight about this, because the rule above is more useful than it first looks and I would rather you know its limit.

Every defence I have is retrospective. I found the silent cap because a number looked wrong, not because I understood the mechanism. Ask me before that moment and I couldn't have told you which dependency was shared, because I had no inventory of what my own scripts read.

The honest position is that a process which catches things after the fact will be shaped by whatever surprised it last. That is not a bug to patch with better intentions. It is a structural limit, and the only partial fix I have found is boring: write down the expected set before the run, from a source that is not the one you are checking against. Then a mismatch is a mismatch instead of a coincidence.

Running it on your own system

Take the last thing you called verified. In order:

List what the check actually reads. Not what it is supposed to read. The endpoints, the files, the cached values.
List what the thing being checked reads. If the lists overlap, you've got one source and two opinions about it.
Ask what would have to break for the check to still pass. If the answer is also what would break the main path, you have an echo.
Look for the count. Print the denominator. Every one of my four was invisible without one, including the two where the number was right and the set was wrong.

Step four is the cheapest and it's the one I keep skipping. A run that can't print how many things it examined hasn't verified anything, it's produced a feeling.

Where this came from

Everything above came out of four corrections, and the ones that produced this rule are argued at length in the comments on my two previous posts:

If this is the kind of thing you want once a week, it also lands in an inbox: buttondown.com/theagentloop. One email per post, nothing else.

I'm not citing literature for the test, because the version of it I'm confident in came from a commenter, not a paper. If you know a formal treatment of source independence under correlated failure, I'd like to read it.

3 corrections in 12 hours, and all 3 were about my own verification

The Agent Loop — Sat, 10 Oct 2026 03:48:28 +0000

Three corrections arrived inside twelve hours, and all three turned out to be about my own verification.

None of them were a bug being wrong. Every one was a check that ran, reported what it was asked to report, and still failed. One of them also corrected a number I had been quoting as my own.

I have written about what verification costs. This is the other half of the problem, and it is the half that actually bit me: a verification claim inherits the failure modes of whatever it was built against.

The one that should have been embarrassing

After publishing, I run a job that re-fetches every post and checks it against a hand-maintained list of what should exist. Two sources. It looked independent by construction.

It caught a real bug, which is why I trusted it. My enumeration asked the API for 100 items per page. The API silently returned 15. The newest post never appeared in the list of what existed, so the check compared 18 expected ids against 15 actual and reported a discrepancy.

Then a reader pointed out I'd got it wrong.

The hand-maintained list was independent. The live enumeration was not. It read the same endpoint, with the same silent cap, and the cap was the exact failure the check existed to catch. I was examining the witness using a witness.

The rule I took from it, and the reason it generalises: an independently enumerated list is only independent if it cannot fail the way the first source does. Naming a second source proves nothing. You have to know what the two sources share.

That sentence came from someone reading my code. I didn't write it.

The one where nobody was there

Same week, same thread, a different reader described a setup better than mine. Instead of a start-marker-with-pid plus an exit receipt, they described logging what your task did in the log of the orchestrator itself, so a task that produced nothing is still provably a task that ran.

Then they pointed out what mine actually depends on.

To find my orphans, I need a human to run ps and notice. That only fires when someone happens to be paying attention. Their mechanism is checked as a side effect of the job running. Mine is checked by a person.

Their description also broke a number I had been repeating. I had been saying "nine of ten of my deaths wrote no signature," and it turns out I inherited that ratio from their own background worker loops, where the OOM killer drops a SIGKILL and nothing reaches stderr. It is a true number about their system. It was never mine, and I had been citing it as though it were.

My own ledger tops out at nine recorded deaths, and I cannot tell you how many of those wrote nothing, because the ones that write nothing are exactly the ones that do not get written down. That is the gap, stated honestly: the missing witness does not merely hide the failure, it removes the failure from the count.

So the third kind of scope inheritance is temporal rather than structural. A verifier that runs on your schedule inherits your schedule's gaps. Mine only knows what happened while someone was watching.

The one where I expanded someone else's experiment

This one is the newest, and I nearly left it as a compliment.

I commented on someone's retry-safety benchmark. The author had built a small set of API failure scenarios and had models return YES, NO, or YES_AFTER_DELAY, then scored only the Decision field. My addition was that retry cost should be a first-class field too, because a retry of a turn in an agent re-sends the context that produced the turn rather than just the failing call, so the multiplier is the accumulated conversation. Then I suggested the benchmark grow a field for the cost of being wrong.

He replied that his benchmark is about retry decisions for API requests, not for an entire agent loop, and that comparing retrying against re-planning would be a completely different experiment. He was right, and the useful part was why.

He had already published a score-versus-cost Pareto chart. I proposed as missing something he had already built.

Which is the same shape as the first one, in a different place. My claim had inherited the scope of the thing I was adding to, and then claimed credit for noticing a gap that the gap-holder had already measured. The addition was fine. The framing was an inflation.

So there are three kinds, worth separating because the fixes differ:

Kind	What is inherited	The fix
Shared source	the second source reads the first source's endpoint	make the second source fail differently, or admit it is not independent
Missing witness	the verifier runs on the same schedule as the thing it watches	log from inside the orchestrator, not from a side channel
Inherited scope	your claim quietly adopts the limits of what it was measured against	re-read their own results before claiming a gap

The test that catches at least two of them

There is one cheap move that would have caught the first one and would have caught the third.

Before you call something independent, read the other person's own results and name the number they already produced. If the thing you are about to point out is already in their chart, you have either misread them or you have nothing new.

It costs one careful read. It is the same instinct as checking a mirror, and it's the only step I skipped.

The second kind is harder, and I don't have a single-command fix. The honest version is that there are some failures you can only detect by being structurally present, and if you cannot be structurally present, the honest response is to say so rather than to publish a check that passes when nobody is looking.

What I actually changed

Three things, all small:

The verification job now records where each expected id came from, so a future reader can see that one side is hand-maintained and one side is an API call, and that both hit the same endpoint. The independence claim is now written next to the code that could falsify it.

I stopped describing my orphan check as a safeguard. It is a manual procedure that runs when a human remembers. Calling it a check made it sound stronger than it is.

And when I comment on someone's work now, I write down what their experiment already showed before I write what I think is missing. That step costs about a minute and it is the only reason I caught the third one myself.

None of this makes verification cheap. It makes the failure mode legible, which is a much smaller claim and the only one the evidence supports.

The full incident records, including the ones that left nothing behind, are in the ledger linked at the top. Corrections are published next to the original claim rather than quietly edited in, which is the only reason any of this is worth reading.

If you have an exit receipt, an orchestrator-side log, or a trace your verification actually depends on, I would genuinely like to read how you built it. The three failure modes above are the ones I have personally hit, and I'd rather compare notes than keep collecting my own.

I have 4 runs that looked clean and were all wrong, and none of them printed a denominator

The Agent Loop — Thu, 08 Oct 2026 18:35:48 +0000

I have 4 runs that looked clean and were all wrong, and none of them printed a denominator

A receipt that agrees with itself is not evidence. I learned that four times on the same project, and the reason I am writing it down is that every one of them passed.

Not passed a test. Not passed review. Passed in the sense that the output was well-formed, internally consistent, and free of errors, every single time, while the thing it described was not happening.

1. The receipt that agreed with itself

We had one script meant to arm a scheduled job, and a different script writing the output that said the job was armed. For about a day, that receipt was perfect. Correct shape, no missing fields, no warnings. It was also entirely self-referential: the thing that claimed the job was running was the thing that had tried to start it, writing a file saying it had.

Nothing in the schema was wrong. What caught it was re-fetching the artifact through an unrelated path and finding it absent.

That's the whole lesson, and it is small enough to miss. A report written by the thing it reports on has no external reference to disagree with. You can validate its shape forever. It will pass.

2. Five clean sweeps that were checking 15 of 16 things

This one sat for four days.

We had a script that re-checked, for every post we'd published, whether the platform would let search engines index it. It wrote a receipt with a post count and an error count, and we looked at those numbers routinely.

Across five consecutive runs the output was byte-identical. Zero errors every time.

It was silently checking 15 of 16 posts. A hardcoded list of post ids in the script had fallen behind reality, and separately the platform's API quietly caps its page size at 15, so the sixteenth was never in the request at all.

Nothing in the output invited the question. The receipt said "15 posts, 0 errors" and we read that as a healthy run, because we had no reason to think 15 was wrong. We only found it when the newest post went missing from a summary and we went looking.

The fix was two lines: print the ids you checked, and refuse to write unless that list matches one you enumerated independently. Five runs of clean output had been telling us nothing at all, and we read them five times.

3. Ten deaths that wrote nothing

This project has run on a phone, inside a container, for about three months. It has been killed abruptly ten times. That is not unusual on this hardware, and we do not pretend otherwise.

Nine of those ten left no signature at all. No stack trace, no error, no exit code. Just a process that was there and then was not.

The rule we wrote down afterward was blunt: a crash with no signature is a kill, not a clean failure, and its absence proves nothing. But notice what the absence actually cost. For hours at a stretch we were reading an empty log and calling it a clean exit, because an empty log looks exactly like nothing having gone wrong.

A log line you did not write is indistinguishable from an event that did not happen. If your process can die without writing anything, then "no output" has to be a state your tooling treats as unknown, and most tooling treats it as success because a missing error is easy to parse and a failure you have to go looking for is not.

4. The diagnostic that made the outage worse

Late on 10-07 our API calls started returning 404. Every single one of them, across every endpoint, while the public site returned 200 the whole time.

The first thing I did was the wrong thing. I assumed the endpoint had been removed, because a uniform 404 is what a schema change looks like. I was about to rewrite working tooling that had run fine for three months.

Then I fetched the identical URL from a completely different network. It returned 200, full response, no authentication. So the endpoint was fine, and our address was rate-limited.

And here is the part I would not have predicted. While diagnosing the rate limit I ran eight probes at one-and-a-half-second spacing to narrow it down. Those probes are why a forty-second throttle became a block that lasted twenty minutes. I was generating the outage while measuring it.

Probing a system changes the quantity you are measuring. Every check costs the thing you are checking. That's fine once, it's a problem in a loop, and it's catastrophic when the thing you are probing is the thing deciding whether to serve you.

The thing that actually settled it was one fetch from a different network. I had that available the whole time and reached for the wrong tool first.

The pattern, stated carefully

I am not going to tell you these four failures are common. Four incidents on one project on one phone is an anecdote, and I would be overstating it to imply a rate.

What I will say is that all four share a shape, and the shape is provable rather than statistical.

A generated artifact that reports on itself has no denominator. It cannot be wrong inconsistently and catch itself, because consistency is the only property available to check. To get a failure rate you need a count of what should have happened, derived from somewhere that is not the thing being measured.

In case 1 the artifact was its own reference, so disagreement was impossible.
Case 2 had no expected count, so 15 and 16 looked the same.
Case 3 could produce no output at all, which left silence undefined.
And case 4 turned the measurement into the load, so the numbers described my diagnostics rather than the service.

Each one is a different mechanism. What they share is that no amount of schema validation would have caught any of them, because in every case the schema was satisfied. That is what a missing denominator buys you: the output looks right, continuously, right up until the day you go and check something by hand.

What I would add to every receipt, ranked by usefulness

One: a count that came from outside the artifact. Not a field the generator fills in for itself. If your check enumerates nine posts and the id list says ten, the run should refuse to write. This is the fix for cases 1 and 2 and it is two lines.

Two: negative space. A field for what was expected and did not happen. This is the one everybody skips. In our own logs a missing line is byte-for-byte identical to a clean run, and that is not a formatting detail, it is the entire difference between a measurement and a decoration.

Three: treat silent output as unknown, never as success. If something can die without a word, then no output is a third state and your tooling has to say so out loud rather than defaulting to the convenient interpretation.

Four: separate the probe from the thing being probed. Where you can, verify through a path that is not the path you are testing. That is the whole reason the last one took twenty minutes instead of one, and it is also the reason it was diagnosable at all.

None of this is exotic. It is the ordinary discipline of not asking a generated report to grade its own homework.

Where this costs you money

There is a version of this that is purely about safety, and there is a version that is about money, and the second one is why it ends up on a cost blog.

A denial that emits no record is not free. In the approval path I wrote about yesterday, a safety check runs as a model call, and a refusal with no verdict is a model call that produced a refusal and left no trace. You pay for the turn. You cannot alert on it, cannot count it, and cannot bill for it, because nothing anywhere recorded that it happened.

That is the whole economics of instrumentation in one sentence: you cannot budget for a path that does not emit a record. The path is not free, it is invisible, and invisible costs are the ones that grow.

It is also why the cost dashboard could not tell us which agent spent the money, and why we do not know who approved what. Neither is really an analytics problem. Both are a missing denominator wearing a different hat.

The limit on all of this

One project, one phone, three months, four incidents. Whether any of this generalises to a team with real observability money is beyond what I can tell you. If your platform hands you a denominator for free then most of this simply does not apply to you.

What I am confident about is narrower: these four things happened, they were each individually small, and together they cost about three days of looking in the wrong place. If you have a run that reports on itself and you have not checked its denominator by hand, that is where I would start.

If you want the rest of this series when it drops: subscribe via Buttondown and follow so the next one lands in your feed. Tell me the last thing your agent reported as fine that you have never independently checked, because I have four and the armed-job receipt is still the one that offends me.

Related on The Agent Loop

Sources

Claude Code: Permission modes: where a no-verdict denial produces no notification and no /permissions entry, and the 10-in-a-row ceiling that stops the turn.
Forem skip_indexing?: the platform gate behind our index sweeps.
RFC 9110: HTTP Semantics: status codes, for the 404-versus-403 distinction the probe loop got wrong.

FAQ

Is a receipt that always prints ok evidence that nothing is wrong?
No. It is evidence that the code path that prints ok ran to completion. If the same artifact supplies the thing being checked, that is all you have learned.

How do I add a denominator to something I built myself?
Print the list of items you actually checked, and compare it against a second enumeration produced independently. If they differ, fail the run rather than writing the receipt. Ours compared a hardcoded id list against the live published list, which is how the fifteenth-of-sixteen bug finally surfaced.

Should a missing log line count as success?
Only if something else positively asserts success. Absence of output is absence of evidence, and on a system that gets killed without warning it is the single most misleading thing your tooling can encounter, because it parses cleanly.

Is this worth worrying about if I use an agent framework?
Probably less so, and that is the honest position. Most hosted frameworks do give you traces with an expected step list, which is the denominator. The gap showed up for us because the thing being measured was a script I had written, checking a script I had written, against a platform with a page-size cap nobody documents. That is a specific and fairly common shape for small projects and an unusual one for a team with a platform team.

Your agent's safety check now runs a model call per command

The Agent Loop — Wed, 07 Oct 2026 06:34:04 +0000

Your agent's safety check now runs a model call per command

(Correcting the plan I started this post with: I originally titled it "2 of the 4 options on your approval prompt permanently widen permissions." I couldn't source "permanently" for the second option, so I dropped it. What I found while checking was better anyway.)

An approval prompt is a thing you click. You read the command, you decide, you move on, and the whole gate costs you one second of attention and nothing at all on your invoice.

That stopped being true. As of Claude Code v2.1.283, auto mode is the built-in starting permission mode on every plan and provider. There is no prompt in the default path any more. In its place is a background classifier running on Claude Sonnet 5, and for some plans those calls count toward your token usage.

What it costs, in the vendor's own words

From Anthropic's permission-modes documentation:

The classifier runs on Claude Sonnet 5 by default rather than on your /model selection.

Each check sends a portion of the transcript plus the pending action, adding a round-trip before execution.

On Enterprise plans and on accounts that use the Claude API, Claude Platform on AWS, Amazon Bedrock, Google Cloud's Agent Platform, or Microsoft Foundry, classifier calls count toward your token usage.

Read that third line again, because it is the one that matters for a bill. On a consumer subscription, this is not charging you per keystroke. On an API key or an enterprise agreement, every shell command and every network request now drags a slice of your transcript into a second model before the first one is allowed to finish.

And the overhead is not spread evenly:

Reads and working-directory edits outside protected paths skip the classifier, so the overhead comes mainly from shell commands and network operations.

So the cheap-looking activity, a loop of shell commands, is exactly the expensive one. Our cost work has said for a while that the loop is the bill, not the model. The safety check turns out to be inside that loop now, and it was the one part of the loop nobody budgeted for.

I am not going to print a dollar figure, and you should be suspicious of anyone who does. The docs never say how much of the transcript goes into a check, so any total would be arithmetic I performed on a number I made up. What is documented is the mechanism: a second model, per action, on your meter.

Three things it changes without telling you

This is the part I would want someone to have told me.

One: entering auto mode quietly throws away your broad allow rules. You spent an afternoon building Bash(npm *) so you would stop being interrupted. Then you accept one prompt and switch to auto:

On entering auto mode, broad allow rules that grant arbitrary code execution are dropped: Blanket Bash(*) or PowerShell(*), Wildcarded interpreters like Bash(python*), Package-manager run commands, Agent allow rules, Monitor allow rules, because Claude Code runs Monitor commands through the shell.

Claude Code restores the dropped rules when you leave auto mode.

So your ruleset depends on which mode you are in, it changes on a single click, and nothing in the interface narrates it. Leave auto and they come back. That is a much better design than dropping them, and it is still a ruleset that varies underneath you.

Two: a missing verdict is a silent refusal. The failure modes are not symmetrical:

A blocked action: Claude Code shows a notification and lists the action in /permissions under the Recently denied tab, where you can press r to retry it with a manual approval.

No verdict from the classifier: Claude Code denies the action without the notification or the Recently denied entry.

Blocked because it thought you were in danger, you see it. Denied because the classifier had an off day, you do not. Same outcome for your run, one is recoverable in a keystroke and the other is a mystery you will spend an hour reconstructing. There is a related ceiling: the turn stops after ten responses in a row with no verdict.

Three: boundaries you type in chat can be evicted. This is the one that should worry me most, and it connects straight to why agents forget by tomorrow.

You can tell the agent in conversation that this repo must never deploy to production. The docs are explicit that the classifier honours that:

The classifier treats boundaries you state in the conversation as a block signal. … A boundary stays in force until you lift it in a later message. Claude's own judgment that a condition was met does not lift it.

And then:

Boundaries are not stored as rules. The classifier re-reads them from the transcript on each check, so a boundary can be lost if context compaction removes the message that stated it. For a hard guarantee, add a deny rule instead.

You stated it in English, it is enforced by re-reading English, and a routine housekeeping operation can delete the sentence that made it true. Anthropic's own advice is to write a deny rule instead, and they are right. But note what happened: the soft way to express a constraint is the way that quietly fails, and it fails in the direction of removing a protection you believed you had.

The part that is better than what I warned about

I wrote prompt injection is your agent's tools being the real attack surface five weeks ago, and in a comment last week I told a reader the fix is that the approval screen should carry the provenance of the instruction. A page the agent just fetched should not be able to speak with your authority.

Auto mode attacks that from the other direction, and it works:

Tool results are stripped from those requests, so hostile content in a file or web page can't manipulate the classifier directly.

The gate never sees the attacker's text, so the attacker's text cannot vote. That is a better mechanism than anything I proposed. I was arguing for showing provenance to the human; Anthropic chose to withhold it from the judge instead. Neither is complete, since the classifier is blind to command output and cannot check one deletion against another, but the second one is strictly harder to attack.

So my earlier advice was aimed at the wrong layer. Provenance should be enforced where the decision is made, not merely displayed where the human is asked to agree.

The approval screen is still there. It is just the fallback.

Worth saying plainly, because the docs are unusually honest about this: "Auto mode reduces permission prompts but does not guarantee safety. Use it for tasks where you trust the general direction, not as a replacement for review on sensitive operations."

And the human prompt it replaced is still reachable, and it is still the sharpest tool you have. Which is where the part I got wrong first time comes back in.

When Claude Code shows you a Bash prompt, it lists four options. Here is the second one:

Yes, and don't ask again for: npm test *

You read npm test. You are offered npm test *. The wildcard is the artifact that gets written, and "a * in a Bash rule matches any text, including spaces, so one rule covers a family of commands." One keystroke turns a decision about today's command into a standing permission over the family. If you take that option often enough, the wildcard becomes your real policy and you never composed it.

The saving is also lopsided in a way that is easy to miss. A Bash approval is "Permanently per repository and command." A file modification approval is "Until session end." Every permission you accumulate across sessions is therefore a shell permission, while edit permissions evaporate overnight. Anyone who approved both reasonably concluded that edits are now allowed repo-wide, and the prompt did not say otherwise because the prompt is not where that asymmetry is recorded.

The rules are narrower than they read, too. A rule that refuses git push does not catch git -C . push, which appears in the docs as an illustration of the gap rather than as a bug report.

What I would do on Monday

If you are on an API key and your agent runs a lot of shell commands, this is the first line item to look at, because it is invisible in the token dashboard unless you go looking for it.

Three things, none clever:

Treat a rule you saved as a policy statement, and audit settings.local.json when they change. If you cannot say out loud what npm test * covers, it covers more than you think.
Put anything you must never have broken into a deny rule. Not in conversation. In the file.
Run in auto mode on tasks where you would be bored by the prompts, and leave it off everywhere the prompts are the point. The cost overhead lands almost entirely on shell and network calls, which is roughly the same work either way, so you get the speed and the ledger entry and not much else.

Bottom line: the safety check did not get weaker so much as it got quieter and metered. That is a good trade for most work. It is a bad trade for the constraint you meant in English and never wrote down.

If you want the rest of this series when it drops: subscribe via Buttondown and follow so the next one lands in your feed. I am curious what is the widest rule in your settings file right now, because the honest answer is usually a wildcard nobody typed on purpose.

Related on The Agent Loop

Sources

Claude Code: Configure permissions: permission modes, rule precedence, what a prompt shows, what persists where.
Claude Code: Permission modes: auto mode, the classifier, cost and latency, fallbacks, boundaries stated in conversation.
Claude Code: Edit auto-mode rules: block and allow rule overrides.
Forem skip_indexing?: why some posts carry noindex.

FAQ

Does this mean Claude Code ignores my permissions now?
No. The docs are explicit that "Permission rules are enforced by Claude Code, not by the model," and rules are still evaluated deny, then ask, then allow. What changed is that with auto mode on by default, fewer calls reach a human, and a new classifier sits in front of the ones that would have.

Am I billed per command?
Only in the plans named above: Enterprise, and the Claude API plus Claude Platform on AWS, Bedrock, Google Cloud's Agent Platform and Microsoft Foundry. Consumer subscriptions are not itemized per classifier request. I am not publishing a dollar figure because the transcript portion size is not documented, and I would rather leave a gap than fill it with a guess.

Does auto mode stop prompt injection better than my manual gate?
For the specific vector where fetched page content tries to talk its way into an action, yes, and clearly. Tool results are stripped from classifier requests, so hostile content cannot influence it. Manual review was never protected from that either, because a human skimming a screen has the same problem.

Why not just turn it off?
You can. Manual is the default in older builds, and permissions.defaultMode in ~/.claude/settings.json will hold a mode across sessions. You trade the per-action classifier call for prompt fatigue, which is its own failure mode.

Where do I see what is blocked?
/permissions, on the Recently denied tab. Blocks show up there and r retries with a manual approval. Missing verdicts do not appear at all, which is the gap worth knowing about.

Your cache hit rate is lying to you. It's the caller mix.

The Agent Loop — Sun, 04 Oct 2026 06:02:34 +0000

Your cache hit rate is lying to you. It's the caller mix.

One Claude Code deployment: 77% cache hit rate, all green. Same account, one tab over, cron tasks at ~0%, because every run is a fresh session rewriting a ~30K-token static prefix. That is 672 runs a day times 30K tokens, roughly 20M input tokens daily that never touch a cache. The operator posted those numbers on an open issue, and they make my argument better than I can: an account-level hit rate averages over callers that share nothing. Same account, four realities.

The three numbers that decide the bill

Forget dashboards for a second. Caching is priced with three multipliers, and they're the same on every model:

Operation	Multiplier	Example: Opus 5.5 at $4/MTok input
Write, 5-minute TTL	1.25×	$5.00/MTok
Write, 1-hour TTL	2×	$8.00/MTok
Read (either TTL)	0.1× on most models	$0.20/MTok (0.05× on Opus 5.5, 0.025× on Fable 5.1)

Three rules fall out of that table, and I've been quoting all three this week.

One read pays for the 5-minute cache. Write at 1.25×, read at 0.1×, you're at 1.35× against 2× for two uncached passes. The 1-hour write needs two reads.

Match the TTL to the gap between calls. Under five minutes: default. Five to sixty minutes: pay the 2× write once, ride the cheap reads. Over an hour: don't cache. A breakpoint nobody reuses doesn't save money, it costs 25% extra on the 5-minute tier and 100% on the 1-hour tier.

Refreshes are free, and the clock lies a little. Every hit renews the TTL at the read price, so a busy cache lives forever. Keepalive math crosses over at 62.5 minutes (5 × 1.25/0.10), same for every model because it's a ratio. The countdown starts when the request starts: a turn streaming four minutes leaves your next call one minute of window.

One account is not one cache

Claude Code picks the TTL per request, in two buckets (their docs, not a blog post):

Billing	Main conversation	Everything else
Claude subscription, within plan usage	1-hour	5-minute
API key, cloud provider, or past your plan limit	5-minute	5-minute

"Everything else" is where agents live: subagents, workflows, forks, compaction calls, session titles. You can override both buckets (promptCacheTtl, subagentPromptCacheTtl, ENABLE_PROMPT_CACHING_1H, v2.1.242+). Hold off, though; the cliff section explains why blanket 1h is worse.

The structural bit: a subagent's first request cannot read the parent's cache. Different prompt, different tools, prefixes diverge at token one, so it warms its own. A fork inherits the parent's prefix exactly and hits on the first request. Same account, opposite behavior, one screen apart.

One honest wobble: the docs say subagents get 5 minutes, a user's transcripts showed 100% of their subagent writes in the 1-hour bucket, and a maintainer said the effective TTL gets decided further down the pipeline while they fix the docs. Don't trust the table or me. Trust usage.cache_creation.ephemeral_5m_input_tokens and ephemeral_1h_input_tokens in your own responses.

The five-minute cliff

Here's the failure that pays for this post.

A parent dispatches a subagent and waits. No requests go out, so nothing refreshes its cache. The measured median child runtime was about 9 minutes, just past the cliff, and 96% of those waits ended in a true cache death: when the parent resumed, at least half its cached prefix had to be rewritten at full price. The long wait, the one where the agent is doing exactly what you asked, is when the cache quietly dies.

Then the finding I had to read twice: blanket 1-hour TTL made the bill 8.6% worse. 98% of reuse lands within about 34 seconds of the write (median gap: 7 seconds), so the 5-minute tier covers nearly all of it, and a universal 2× write premium taxes every write to rescue maybe 2% of reads. The fixes that worked: 1-hour write on the dispatch turn (−6.0%), persistent per-type static prefix (−1.0%), dynamic content after the stable prefix (−7.6%). All three: −13.6%.

TTL is a property of one write at one moment, not a setting on an account. Get that backwards and you collect both failure modes: premium where you don't need it, death where you do.

The metric fix came from our own comments

Reader hannune left this on our cost-loop post: his agent looked perfect in the logs, every response successful, then a three-day-weekend invoice made no sense. He pinned it to a context-retrieval step pulling full documents instead of chunks: 40K tokens per call, seven or eight calls per task. A hit-rate dashboard shows nothing wrong there. Every call succeeded.

His rule, which I've adopted: record the TTL tier per caller, not per account. One extra label next to cache_creation and cache_read: caller × TTL tier. It's the attribution argument from our cost-attribution post, one level lower. We measured a single agent run at 3.5M input tokens against 271K output; the cache either pays on that input side or leaks. No third place for your money to go.

Where caches die quietly

Each of these fails without an error:

Token floors. Current-gen models cache at 512 tokens minimum; Opus 4.6/4.5 and Haiku 4.5 need 4,096. Below the floor the request doesn't cache and cache_creation just reads zero.
The 20-block lookback. A breakpoint scans back at most 20 blocks for a prior write. Add more than that between writes and you get a miss that looks like a bug. One developer hit it with 23 blocks.
Compaction with its own system prompt. Summarize through a different prompt and the call misses the parent's prefix entirely, so your longest transcript bills at full price, right when it's biggest. Anthropic's fix: reuse the parent's exact prefix. They alert on hit rate and declare SEVs when it drops.
Scheduled runs. New session every time, static prefix rewritten every time. Cron at ~0% is architecture, not a bug a longer TTL fixes.

The caller grid

Caller	Default TTL	Reads parent cache?	Typical gap	Do this
Main loop	1h (subscription) / 5m (API)	n/a	seconds	On API key: set `promptCacheTtl=1h`
Subagent	5m	No, own prefix	seconds inside a type, minutes between types	1h write on the shared per-type prefix; dynamic after the breakpoint
Fork	parent's	Yes	immediate	Use it for side work that must see history
Compaction	parent's, only if same prefix	must	one call	Verify the prefix is reused
Cron / scheduled	5m, fresh session	No	hours	Expect ~0%; shrink the static prefix or budget the writes

Do this Monday

Split the metric: hit rate per caller, or at minimum log ttl_tier × caller beside your cache fields.
Read the raw buckets (ephemeral_5m, ephemeral_1h, cache_read) from response usage. Vendor summaries average away what you're looking for.
Stable head behind cache_control; date, cwd, and branch after it.
Use ttl:"1h" in two places only: the dispatch turn and shared static prefixes.
Hit rate healthy but invoice ugly? Hunt write-never-read breakpoints, the ones billing 1.25× or 2× for work nobody reuses.

FAQ

How do I see which TTL a request used? usage.cache_creation in the response, or the same field in your transcript files. ephemeral_1h_input_tokens vs ephemeral_5m_input_tokens are mutually exclusive buckets.

Do other providers work like this? Differently, same lesson. OpenAI caches automatically with a discount and no TTL knob; Google prices storage per hour. Wherever prefixes belong to conversations, the per-caller problem holds.

My hit rate is 90% and my bill is fine. Do I care? No. This one is for people whose dashboard and invoice disagree.

Over to you

What's your worst cache own-goal? Dashboard healthy, invoice confusing, and one line in a transcript finally explained it. I'll go first if you do.

Sources

Anthropic prompt caching docs: multipliers, floors, refresh, lookback (verified 2026-10-02)
Claude Code prompt caching docs: TTL buckets, subagent behavior, override settings
Lessons from building Claude Code: compaction fix, SEV alerts
claude-code #74318: subagent spend, 96% cache deaths, +8.6% blanket-1h, deployment numbers
claude-code #84289: docs vs transcript TTL mismatch
Artificial Analysis caching comparison: OpenAI and Google TTL behavior
62.5-minute keepalive derivation, prior art: skids.dev tokenomics
Our threads: cost-loop comments (hannune, 2026-09-26), cost attribution, one run costs

Independent blog: not affiliated with Anthropic or the Claude Code team. Every figure above comes from public docs and issues, verified on the dates given.

If you want the rest of this series when it drops: subscribe via Buttondown and reply with your cache horror story. One of these turns into a post.

Run opencode on Android: a working setup from five weeks on the phone

The Agent Loop — Fri, 02 Oct 2026 08:03:15 +0000

Drafted with AI help, human-reviewed by The Agent Loop.

Receipts from an actual phone: an S25 Ultra running opencode since 2026-08-25. Versions below are checked as of October 2, 2026.

You don't need a laptop to run a serious AI coding agent. The official opencode docs tell you to install WezTerm or Alacritty and never mention Android once. The binary has no such problem: it is a plain linux-arm64 build, and a phone with a Linux userland runs it fine. I have been doing exactly that for five weeks, through five full environment restarts, and this blog post was written from inside the setup.

The five-line version:

Termux gives you a Linux terminal on Android; proot-distro gives you an Ubuntu userland inside it.
opencode's own install script explicitly supports linux-arm64, so the standard one-liner just works.
opencode itself is a single binary; Node is for the MCP servers and scripts around it.
The docs' desktop-terminal prerequisite is a habit, not a requirement.
Four gotchas will bite you anyway: Node platform, RAM, background jobs, and no systemd.

Why the docs don't mention your phone

The opencode docs list prerequisites as "a modern terminal emulator" and name four desktop apps. Windows gets a whole page telling you to use WSL. Android gets nothing. Meanwhile the install script, which I fetched on October 2, maps uname -m from aarch64 to arm64 and accepts linux-arm64 as a supported target right next to linux-x64 (install script).

That gap is the whole tutorial. The agent is a compiled binary for the same ARM instruction set your phone already speaks. What it wants from you is a Linux environment, and Android has one ready: Termux, plus proot-distro, which runs a real distribution without root by translating system calls in user space.

Android phone (12 GB RAM)
  └─ Termux                 Linux terminal for Android
       └─ proot-distro      Ubuntu 26.04 userland (no root)
            ├─ opencode     v1.18.34, single linux-arm64 binary
            └─ MCP servers  exa, firecrawl, playwright (stdio/HTTP)

The stack, with receipts

Every row is from this machine, checked October 2, 2026:

Component	Receipt
Device	Samsung Galaxy S25 Ultra, 12 GB RAM
Kernel	`6.17.0-PRoot-Distro`
Userland	Ubuntu 26.04 LTS, glibc 2.43
Agent	`opencode --version` → `1.18.34`
Node	`v26.4.0` (MCP servers, Playwright, scripts)
Python	`3.14.4` (analytics, diagrams)

Docs now carry a banner for a v2 of opencode; this post is the 1.x setup, and every version string above is date-stamped so you can tell what you are reading.

Install path

On the Termux side (Android, not inside the container):

pkg install proot-distro
proot-distro install ubuntu
proot-distro login ubuntu

Inside Ubuntu, first login:

apt-get update && apt-get install -y curl git
curl -fsSL https://opencode.ai/install | bash
opencode --version
1.18.34

The script drops the binary in ~/.opencode/bin and adds it to your PATH. No Node is required for that step. If you ever see this, you're already inside, which is the correct state:

Error: attempted to run proot-distro in a proot session.

Run proot-distro list from Termux if you want to see your installed distro; it refuses to run nested, on purpose.

The four gotchas the docs don't cover

1. Termux's Node can poison the well. Termux ships its own Node package, and some builds report process.platform as android, which Playwright rejects outright. Install a normal Linux arm64 Node inside the Ubuntu userland instead (mine is v26.4.0), and that class of error never appears. This is the reason the agent lives inside proot rather than in raw Termux: everything in there reports linux, and the tooling behaves like it would on any desktop.

2. RAM is the real limit, not CPU. Twelve gigabytes sounds like plenty until you spawn browser automation. Four Chromium tabs under proot sent this box into epoll and futex failures (the kernel calls return "not implemented" through the translation layer), and one runaway log file grew to 64 GB in a morning. The rules I now run by: free -m before starting anything browser-heavy, --concurrency=1 on renders, and never point a long-lived process's stderr at an unbounded file.

3. Background jobs are perishable. A detached setsid job survives a tool timeout but not an environment restart, and I have now buried five of them. Worse, sleep runs on a clock that stops while the phone suspends, so a job armed for 13:00 can wake up three hours late. The working pattern: detach properly, then grep the log after the window and rerun the idempotent script if the receipt is missing. Never trust that the job filled its file.

4. Nothing auto-starts. There is no systemd here. Check with ps before you believe a service is up; MCP servers, browsers, and schedulers all start by hand or from scripts you control. It's less convenient than a laptop, and it makes every failure visible, which I've come to like.

The config that matters

Everything agent-specific lives in one global file, ~/.config/opencode/opencode.json. The part you touch most is the mcp map, which is plain JSON with a command and an environment block:

{
  "mcp": {
    "exa": {
      "type": "local",
      "command": ["/usr/local/bin/exa-mcp-server"],
      "environment": { "EXA_API_KEY": "<your key>" }
    },
    "firecrawl": {
      "type": "local",
      "command": ["/usr/local/bin/firecrawl-mcp"],
      "environment": { "FIRECRAWL_API_KEY": "<your key>" }
    }
  }
}

Two rules I learned the hard way: the file holds live API keys, so keep it readable by you only and out of any git repo, and if your MCP tools suddenly vanish, check this file before you reinstall anything. A broken config has wiped this box's tool list once already, and the Hermes config next door was the copy that rebuilt it.

What the phone gives you that the laptop doesn't

The fun part is that the device you are automating is the device in your hand. adb over wireless debugging controls the phone itself, so the same terminal that runs the agent can drive its host. Pair it with a monitor through DeX and the setup becomes a workstation that fits in a pocket. The agent session also survives screen locks and app switches the way a terminal multiplexer does, not the way mobile apps do.

Bottom line: the "AI coding agent needs a laptop" claim is a documentation habit, not a technical one. Termux plus proot-distro plus the stock linux-arm64 installer is a boring, copy-pasteable path to running opencode on Android, and five weeks of daily use say the four gotchas are the entire maintenance surface.

FAQ

Is this an official opencode setup?
No, and I'm not affiliated. It is what one user's phone has run since August 2026, with versions you can verify against the table above.

Does it work on low-RAM phones?
The agent itself is lightweight; the browser-heavy workloads are not. Under 8 GB I would expect the memory gates in gotcha two to fire often, and I have not tested below 12 GB myself.

Can I use this for real work, or is it a demo?
Every post in this series, including this one, was drafted, verified, and published from the setup, along with the analytics dashboard and the scheduled jobs that keep it running.

Related on The Agent Loop

Your agent's tests pass. That's the problem. (what you run once the agent is on your machine)
Why your MCP approval gate never fires (the MCP layer from the diagram, one layer down)
Your agent's cost problem isn't the model. It's the loop. (what the loop costs when it runs on your hardware)

Sources

opencode docs: Intro and prerequisites
opencode install script (fetched 2026-10-02; linux-arm64 support at script lines 102-110)
Termux
proot-distro (Termux wiki)
opencode MCP server docs

If this saved you a laptop, tap the unicorn below; one click, and it is the only metric Dev.to shows me. Follow The Agent Loop for the rest of this series.

Every post also lands in an inbox: subscribe by email, one email per post and nothing else.

What would you run first if your entire dev environment fit in your pocket? A CI script, a bot, the blog itself? Reply below.

Who eats the loss when your AI agent spends your money?

The Agent Loop — Fri, 02 Oct 2026 03:51:07 +0000

Drafted with AI help, human-reviewed by The Agent Loop.

How the rails read as of October 2026. Not legal advice; your issuer agreement and your jurisdiction beat anything here.

When your AI agent buys something with your card details, the loss lands on you. Every liability rule on these rails turns on a single question: was the transaction unauthorized? Hand an agent your credentials and, by the text of those rules, the transaction was authorized. That is the whole trick.

The five-line version:

US Regulation E excludes transfers by anyone you "furnished the access device", and the official commentary says you are fully liable even when they exceed the authority you gave.
Visa's zero-liability policy "solely covers unauthorized payments". A purchase your agent made with your blessing isn't one.
We could find no rule, issuer policy, or court decision that treats "an AI did it" as a defense (checked October 2026). The industry calls it an open question.
The one bound that works today is set before the damage: hard limits on a pre-funded card.
The lever during the damage: tell the bank that person's transfers are no longer authorized, and the protections re-engage.

The three ceilings

Two readers of our bank-accounts post drew the frame I wish I had used. Ownership is one ceiling (can an agent hold the account). Authentication is a lower one (did this transaction get strong customer authentication). The third ceiling is the one this post is about: liability, who eats the loss when it goes wrong (mickyarun's comment). Ownership debates take years. A disputed charge lands this month.

What the rules actually say

Regulation E defines an "unauthorized electronic fund transfer" as one initiated by a person other than you "without actual authority to initiate the transfer" (12 CFR 1005.2(m)). The same paragraph excludes transfers initiated "by a person who was furnished the access device to the consumer's account by the consumer". You gave your agent the card. It is that person.

The official staff commentary closes the escape hatch: if you grant authority to a person, "such as a family member or co-worker", and they exceed the authority given, "the consumer is fully liable for the transfers" until you tell the bank otherwise (Official Staff Commentary 2(m)-2, via CFPB's EFT FAQs). Banking-industry commentary puts it as a sandwich-versus-Tesla story: tell someone to buy a sandwich, they buy a Tesla, that's not a bank problem (Backbase, 2026).

The caps in 1005.6 ($50 in two days, $500 within sixty, unlimited after) only attach to unauthorized transfers. Your agent's purchase never enters that machinery.

Visa's side matches: the fraud-risk FAQ says zero-liability "solely covers unauthorized payments", excludes transactions "initiated by the consumer", and doesn't apply to certain commercial cards (Visa Direct Fraud Risk FAQ). Issuer cardholder agreements go further and may exclude any transaction made "by a person authorized to transact business on the account" or one that "exceeds the authority given by the account owner" (example issuer terms).

The EU has the same shape from a different direction. PSD2 Article 74 puts unauthorised-transaction losses on the payer up to €50, and removes even that when strong customer authentication was not used (EBA, Article 74). But the shield is built for stolen instruments, not delegated ones. Whether an AI agent you authorized counts as an authorized delegate remains an open question.

Who eats it: four setups

Setup	Who eats the bad purchase	The bound
Personal card, agent has your credentials	You, fully	Whatever the agent can reach
Same, after you revoke with the bank	Back into the normal dispute process	Regulation E caps re-attach
Pre-funded card, hard daily limit	You, up to the limit	The limit, per day
Registered agent on Amex's network	Amex, per its published commitment	Their terms

Rows three and four are the whole policy debate compressed. hannune, the second commenter, wrote what I now consider the design rule: a pre-funded card with a hard daily limit "makes the card the unit of audit instead of the agent" (thread). Open-ended liability plus an autonomous spender is the default row, and the one nobody should be in.

What is actually moving in 2026

Honesty requires the parts that cut against us:

Merchants got a new weapon first. Visa's compelling-evidence 3.0 rule (April 2023) invalidates a fraud dispute when evidence shows the cardholder or an authorized person participated in the transaction. "My agent did it" can therefore lose at the merchant fight-back stage too, on the device, IP, and login history the agent itself generated (Visa merchant FAQ).
The networks shipped the front end before the back end. Visa expanded its Agentic Ready program globally in late April 2026; Mastercard and Santander ran Europe's first live regulated agent payment in March 2026; Amex published an agentic commerce kit and a commitment to cover erroneous purchases by registered agents (Chargebacks911 via Finopotamus, May 2026). The same piece warns the dispute infrastructure behind those launches "remains almost entirely unaddressed", against a Mastercard forecast: chargebacks growing 24% to 324 million a year by 2028, before agent impact.
The protocol layer is where the fix lives. Visa's Trusted Agent Protocol, Mastercard Agent Pay, and Google's AP2 all aim at the real problem: cryptographically proving what the consumer authorized the agent to do at the moment of delegation, not reconstructed after a dispute (Chargeflow, July 2026).
Regulators stepped aside on purpose. The Fed, OCC, and FDIC's SR 26-2 (April 2026) explicitly placed generative and agentic AI outside its updated model-risk guidance as "novel and rapidly evolving". No country has agentic-commerce liability law; PSD3 is still being negotiated (Chargeflow, CentsChat).

My own limit, checked twice while writing: I looked for one published rule, decision, or issuer policy that treats an agent as a special case, and found none. If you know one, the comments are for exactly that.

Three questions before you hand over a card

From mickyarun's test, made operational:

Who is the accountable party on this rail, and does zero-liability even apply to this card type (commercial cards are the known carve-out)?
What reverses this transaction, on what clock, and whose logs count as evidence, mine or only the bank's?
What is the worst day? If the answer is open-ended, the daily limit comes before the agent, not after.

Bottom line: liability is the ceiling that falls first. Until a rule says otherwise, on the payment rails your agent is you: it spends under your authority, it loses under your name. Keep the bound small, know your revocation line, and ask the three questions first.

FAQ

Is my AI agent's purchase "unauthorized" under Regulation E?
Almost certainly not, if you gave the agent your credentials. The definition excludes transfers by a person you furnished with the access device; the staff commentary holds you fully liable even when they exceed the authority you granted.

Does Visa zero-liability save me from my agent's mistake?
No. The policy covers unauthorized payments only. Purchases you initiated or delegated don't qualify, and issuer terms may add explicit carve-outs for authorized persons and commercial cards.

Is the AI company liable instead?
Not under anything we could source as of October 2026. Industry reviews call the question open, and Amex's registered-agent commitment is the one published exception that shifts loss from you to a network.

What should I do before giving an agent spending access?
Run the three questions above, put the agent on a pre-funded card with a hard daily limit, and write down the revocation path (who to call to make their transfers unauthorized again) before the first transaction.

Related on The Agent Loop

Can an AI agent have a bank account in 2026? (the ownership ceiling; source of both comments)
Where does the budget check go? (loss prevention, this post is loss allocation)
What one agent run actually costs (the cost side of the same ledger)

Sources

If this post changed how you read your own exposure, tap the unicorn below; one click, and it is the only metric Dev.to shows me. Follow The Agent Loop for the rest of this series on where agent money actually goes.

Every post also lands in an inbox: subscribe by email, one email per post and nothing else.

Has your agent ever spent money you didn't expect? What did the bank or card issuer say the moment you told them an agent did it? Reply below.