DEV Community: Harrison Guo

Same Model, 13.3% to 38.3%

Harrison Guo — Tue, 08 Sep 2026 08:05:35 +0000

Two API settings. Same model. Same benchmark. Same task set.

13.3% to 38.3%, using one sixth the output tokens.

OpenAI published that result about GPT-5.6 Sol on ARC-AGI-3, and it is the cleanest natural experiment the field has produced on a question I have been arguing from first principles for a year. Nothing about the model changed. Everything that changed was around it.

The score is Relative Human Action Efficiency, not a pass rate, and OpenAI estimates the average human tester at 48% on the same set. So the model went from 34.7 RHAE points behind a human to 9.7 behind, and the entire move came from configuration. By my arithmetic that is about 72% of a gap people had been attributing to the model.

There is a blunter version of the same fact. On the public leaderboard for one of these games, no frontier model gets past the first level. With the reconfigured harness, GPT-5.6 Sol solves all six.

Everything that moved is in the right-hand column, and none of it is the model:

	official harness	retained reasoning + compaction
model	GPT-5.6 Sol	GPT-5.6 Sol
score (RHAE, public set)	13.3%	38.3%
output tokens per game	6x	1x
reasoning between turns	discarded after every action	retained
at the context limit	rolling truncation, 175,000 chars	compaction
human tester average	48%	48%

What the official harness was doing

This is the part worth sitting with, because it is not exotic. It is the most natural way to write an agent loop.

The benchmark's harness discarded the model's private reasoning after every move, and dropped earlier actions as the context filled up.

The truncation had a specific number attached: once the conversation exceeded 175,000 characters, the oldest messages were dropped.

Both of those are the default behaviour of almost every agent framework I have read. Reasoning tokens are expensive, they are not the answer, and the API returns them as a separate thing you have to deliberately keep. Dropping them feels like hygiene. Truncating the oldest turns when context fills is the two-line version of context management and it is what you write first.

The consequence, in OpenAI's framing, is that the model had to work out the game from scratch on every turn.

Think about what that does to a task where the whole point is learning rules from interaction. The model plays a move, forms a theory about the rules, and the theory is deleted. Next turn it sees the board and its own past actions, but not why it took them. So it re-derives. It burns output tokens rebuilding a theory it already had, arrives somewhere slightly different, and acts on that.

That is not a model that cannot learn rules. That is a model with anterograde amnesia, being scored on rule learning.

The two loops differ by one arrow. The expensive one is the loop that has to rebuild
its own premise on every pass.

ARC had a reason for the generic harness, and it is a good one. A simple harness makes model shortcomings more visible, and it makes comparisons between models fairer. Commercial developers tune a harness to each model's features and quirks, which is exactly the thing a benchmark is trying to factor out.

That reasoning is sound and it still produced a broken measurement. Neutrality is not the absence of assumptions. A harness that discards reasoning has taken a position on whether reasoning should persist, and it happens to be the position that penalises models trained to think across turns.

The two settings

Retain the reasoning across turns. Compact the history instead of truncating it.

Neither is clever. Neither required a new model, a new prompt, or a new scaffold. They are settings, and OpenAI's point in publishing was that they are the same settings already running in ChatGPT and Codex. The production harness had them on. The benchmark harness did not.

The token result is the one I keep returning to. Six times fewer output tokens, while nearly tripling the score.

We have all internalised a trade here: more careful reasoning costs more tokens, and quality is something you buy. This inverts it. The expensive configuration was expensive because it was worse. Every discarded chain of reasoning had to be regenerated, and regeneration is output tokens at output prices. The waste and the failure were the same event.

OpenAI describes the mechanism plainly: with reasoning retained, the model spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Less thinking, better play. That only sounds paradoxical if you were counting the re-derivation as thinking.

I made the general version of this argument in Your AI Bill Is a Distributed Systems Problem: cost in agent systems is dominated by repeated work, not by unit price. This is that thesis with a controlled experiment attached, run by the model vendor, on a public benchmark.

Why the score was never about the model

OpenAI said the quiet part in their own writeup: an evaluation does not measure the model in isolation. It also measures the API configuration, the harness design, and how the prompt was rendered.

Their exact words: benchmarks rarely measure AI models in isolation, they also measure less visible choices about API settings, harness design, and prompting.

Then the line I keep coming back to:

This isn't the first time we've been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.

Not the first time. So there are published numbers, still in circulation, still being cited, that measure a harness defect and are read as a fact about a model. Nobody is going back to correct those.

That is an unusual thing for a lab to publish, and it is correct.

Every agentic benchmark number you have read is a measurement of a triple. Model, harness, configuration. The number is real. The attribution is where it goes wrong, and the attribution is the entire reason anyone reads the number.

This is the exact failure I wrote about in The Log Printed Exactly What I Wanted. A true signal, correctly produced, borrowed for a claim it does not support. 13.3% was a true fact about a system. It was read as a fact about a model. Those are different objects, and the difference here is 25 points.

It is also the wrong ruler at scale. The benchmark ran. It produced a number. The number was reproducible. None of that made it a measurement of the thing everyone thought was being measured, and the fact that it ran cleanly is exactly why nobody checked.

The uncomfortable follow-on

If a generic harness cost one model 25 points, then every cross-model comparison run through a single fixed harness is suspect in a specific way.

A harness is not neutral. It makes assumptions: about whether reasoning persists, about how history is trimmed, about tool-call formatting, about how much of the transcript survives. Those assumptions suit some models and penalise others. Holding the harness constant across models feels like the rigorous choice. It is actually a choice to measure each model's fit to one particular set of assumptions.

So when a leaderboard says model A beats model B on an agentic task, the defensible reading is that the pair (A, harness) beat the pair (B, harness). Sometimes that is what you want to know, because you are going to run that harness. Usually it is presented as something else.

I do not think this makes benchmarks useless. I think it makes the harness a required disclosure, the way a benchmark suite already discloses temperature and prompt.

What this says about the three harnesses

I have been reading Codex and Pi against Claude Code in this series, and this result reframes the thing they disagree about most.

Retained reasoning is a state ownership question. Between two turns, somebody has to hold the model's intermediate work. If the harness drops it, the model regenerates it. If the harness keeps it, the harness now owns a piece of state it has to version, serialise, compact and restore.

That is not a small commitment, which is why the cheap harness drops it. It is also, on this evidence, 25 points.

Compaction versus truncation is the same shape. Truncation is free and lossy in the worst possible way, since it deletes the oldest context, which is usually where the task was defined. Compaction costs an extra model call and a summarisation strategy, and it keeps the definition.

Both settings are cases of the harness paying a real engineering cost to avoid making the model redo work. That is the whole thesis of The 90% Problem, and it is why the three harnesses all spend between 794 and 1,729 lines on a five-step loop. The lines are where the redoing gets prevented.

It also lands squarely on agent memory as a cache coherence problem. Discarded reasoning is a cache with a hit rate of zero. The model recomputes on every access, correctly, and pays for it every time. Nothing is broken. It is just cold, forever.

What to check in your own stack

Two questions, and both have answers you can find today.

Does your framework persist the model's reasoning between turns, or drop it? Many drop it, few say so, and it is usually one field. If you are running a multi-turn task where the model builds understanding over time, this is the first thing to look at.

Does your context management compact or truncate? If the answer is that you slice off the oldest messages when you approach the limit, you are running the configuration that scored 13.3%.

Neither question is about your model. That is the point. The most expensive variable in your agent system is one nobody is benchmarking, and it is sitting in your config file.

Figures and quotations are from OpenAI's own writeup, "How enabling two settings tripled our scores on the ARC-AGI-3 benchmark": GPT-5.6 Sol on the public task set, 13.3% under the official harness against 38.3% with retained reasoning and compaction, output tokens cut 6x, rolling truncation at 175,000 characters, estimated human tester average 48%, scores measured as Relative Human Action Efficiency. The 72% figure is my own arithmetic on the gap to that human baseline and is not OpenAI's claim.

eBPF on Linux — kprobe vs fentry: Hooking Internals & What Production Observability Misses

Harrison Guo — Mon, 07 Sep 2026 16:50:48 +0000

There are two common ways to attach an eBPF program to a kernel function: kprobe and fentry. Most tutorials treat them as interchangeable — pick one, attach, read your data. They are not interchangeable. They install into the kernel by different mechanisms, they hand you the function's arguments in different forms, and on a hot path they cost different amounts of CPU per call. Get the choice wrong on a function that fires a million times a second and you have added measurable overhead to every request on the box.

This post walks the difference at the source level. The companion video disassembles both attachment paths so you can see exactly where the hook lands in machine code; the code here is from SentinelEdge, an eBPF project of mine that hooks the kernel with kprobe across 13 attach points, which makes it a useful place to talk honestly about when I'd reach for fentry instead.

The short version

kprobe attaches by patching the target instruction. Classically that is a breakpoint (int3 on x86) that traps into the kprobe machinery, runs your program, then resumes the displaced instruction. It works almost anywhere and needs nothing special from the kernel build. You receive a raw struct pt_regs * and dig the arguments out of registers yourself.
fentry attaches to the function's __fentry__ site — the call slot the compiler already left at the top of every traceable function for ftrace — through a generated BPF trampoline. No exception, closer to the cost of a plain call. You receive the arguments already typed, resolved from BTF. It needs a modern kernel (≥ 5.5) built with BTF, and the function has to be ftrace-attachable.

Same observability. Different install, different per-call cost, different failure modes.

The kprobe way, in real code

Here is a live kprobe from SentinelEdge, hooking do_mmap to watch memory-mapping activity per process:

SEC("kprobe/do_mmap")
int trace_mmap_detailed(struct pt_regs *ctx) {
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    __u64 addr  = PT_REGS_PARM1(ctx);
    __u64 len   = PT_REGS_PARM2(ctx);
    __u32 prot  = PT_REGS_PARM3(ctx);
    __u32 flags = PT_REGS_PARM4(ctx);

    struct fs_event *event = bpf_ringbuf_reserve(&fs_events, sizeof(*event), 0);
    if (!event)
        return 0;

    event->timestamp = bpf_ktime_get_ns();
    event->pid       = pid;
    event->operation = 10;  // MMAP
    event->size      = len;
    event->mode      = prot;
    bpf_ringbuf_submit(event, 0);
    return 0;
}

Look at what the context gives you: struct pt_regs *ctx, the raw register file at the moment of the trap. There is no do_mmap prototype in sight. You recover the arguments with PT_REGS_PARM1..N, which expand to the architecture's calling-convention registers (rdi, rsi, rdx, rcx, … on x86-64). Nothing checks that PARM3 is really prot. If the kernel's do_mmap signature shifts between versions — and it has — your offsets keep compiling and keep returning a number, just the wrong one. The breakage is silent.

That rawness is also the strength. A kprobe does not care about the function's type information and, in its classic form, does not even need the function to be specially compiled. It patches bytes. That is why it attaches to almost anything and runs on kernels that predate the typed tracing infrastructure.

The fentry way

The same hook as an fentry program looks like this:

SEC("fentry/do_mmap")
int BPF_PROG(trace_mmap, struct file *file, unsigned long addr,
             unsigned long len, unsigned long prot /* … per the kernel prototype */) {
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    // addr, len, prot arrive already typed — no PT_REGS_PARMn
    ...
}

BPF_PROG is a libbpf macro that unpacks the trampoline's argument array into named, typed parameters that mirror the kernel's real prototype, with the types resolved from BTF. You stop reading registers by position and start reading arguments by name. If the signature you wrote does not match the kernel's, it fails at load time against BTF, instead of silently handing you a wrong register at run time. That single property — a mismatch becomes a load error rather than corrupt data — is most of why fentry is the safer default on any kernel new enough to offer it.

fexit is the same idea for function return: it fires on the way out and gives you the arguments and the return value in one program, which a kretprobe cannot do without pairing it to an entry probe and stashing state.

Where the cost comes from

The overhead difference is not a tuning constant; it falls out of the two mechanisms.

A classic kprobe fires through an exception. The CPU hits the patched breakpoint, traps, the kernel walks its int3 path into the kprobe handler, runs your program, single-steps or emulates the instruction it displaced, and returns. An exception round trip is one of the more expensive things you can put on a hot path.

An fentry program fires through a call. The trampoline saves the registers it needs, calls your program, restores, and continues into the real function. There is no trap, no single-step. It is close to the cost of an ordinary indirect call.

One honest nuance the tutorials skip: on a modern kernel, a kprobe placed at a function's entry can be promoted onto the same ftrace call site fentry uses (KPROBES_ON_FTRACE) or jump-optimized, which narrows the gap considerably. So "kprobe always means an int3 trap" is not strictly true anymore. But the ceiling still favors fentry: it is purpose-built for the entry hook, it is typed, and it never falls back to an exception. When you need the lowest, most predictable per-call cost, it is the one to reach for. The video takes both attachment paths apart in a disassembler if you want to see the instruction-level difference rather than take my word for the shape of it.

What you need for fentry (and what breaks)

fentry buys you type safety and low overhead by leaning on infrastructure that older or stripped-down kernels do not have:

Kernel ≥ 5.5 for the BPF tracing (fentry/fexit) program types.
BTF in the kernel (CONFIG_DEBUG_INFO_BTF=y, surfaced at /sys/kernel/btf/vmlinux). No BTF, no typed attach — this is the usual reason fentry "mysteriously" refuses to load on a machine where kprobe is fine.
A function the tracer can reach. Anything inlined, marked notrace, or otherwise missing its __fentry__ site is not attachable. Static/inlined helpers fall into this gap; kprobe on an address can sometimes still reach them.

That list is the real decision tree. kprobe is the tool that works when the target or the kernel does not meet those conditions. fentry is the tool that wins when they do.

What production observability tutorials leave out

The standard eBPF walkthrough shows you the attach and the read. It almost never shows the three things that decide whether your tracer survives contact with production:

How the probe installs, because that is what sets the per-call cost.
What each call actually costs, because you are about to multiply it by the call frequency.
The BTF and kernel-version constraints, because they decide whether your program loads at all on the fleet you are targeting.

The multiplication is the part that bites. Hook something cold — module load, process exit — and the install mechanism barely matters; SentinelEdge uses kprobe on exactly those kinds of events (do_exit, init_module, do_mount) and the overhead is noise. Hook something on the request path — recvmsg on a service doing 100K requests a second — and the same per-call gap you could ignore becomes a percentage of the machine. That is the whole game: not "is eBPF fast," but "is this hook cheap enough at this frequency."

It shows up sharply in AI infrastructure, where eBPF is often the only way to see real syscall latency during inference without changing the serving code — network and file I/O, driver entry points, the per-token streaming path, container-boundary cost in multi-tenant serving. Those are hot paths by definition. Put a trap-based hook on one and your tail latency tells the story at 3am; put a trampoline-based one and you keep the visibility for a cost you can round off.

How I actually hook it, and when I'd switch

I'll be straight about the code you'll find in SentinelEdge: it hooks with kprobe and tracepoints, not fentry. Thirteen kprobe attach points — vfs_open, do_mmap, tcp_connect, do_exit, kmem_cache_alloc, and so on — plus tracepoints for the syscall entries. That was the right call for what the project is: a breadth-first map of kernel activity across a wide set of functions, much of it cold or medium-frequency, aimed at running on a range of kernels without assuming BTF. kprobe attaches everywhere and asks nothing of the build.

The moment I'd move a specific hook to fentry is when it lands on a genuine hot path and I control the kernel it runs on — a modern BTF-enabled build. At that point the typed arguments remove a class of silent version-drift bugs, and the trampoline removes the per-call tax that a request-path frequency would otherwise amplify. It is a per-hook decision, not a project-wide religion.

(The eBPF programs and the ring-buffer data path in SentinelEdge are real and runnable; the distributed pieces are design sketches. The kernel-hooking code is the part worth reading.)

Choosing, quickly

	kprobe	fentry
Installs by	patching the instruction (classically `int3`)	BPF trampoline on the `__fentry__` site
Per-call cost	higher (exception path; less on optimized kernels)	lower (call, no trap)
Arguments	raw `pt_regs`, `PT_REGS_PARMn`, untyped	typed via BTF (`BPF_PROG`)
Needs BTF	no	yes
Min kernel	old	5.5+
Reaches inlined/`notrace`	sometimes	no
Return values	`kretprobe` (paired)	`fexit` (args + retval in one)
Best for	old kernels, broad coverage, cold hooks	hot paths, typed access, controlled modern kernels

The one-line version: fentry is the default on any kernel new enough to give it to you, and kprobe is what you use when the kernel or the target won't. The mistake isn't picking one — it's picking without knowing that a hook on a hot function just made the choice a line item on your CPU budget.

Code above is from SentinelEdge. The companion video disassembles both attachment paths at the instruction level.

The Approval Prompt Is Not the Sandbox

Harrison Guo — Fri, 04 Sep 2026 01:33:10 +0000

Codex ships 43,591 lines of sandboxing code across four platform backends.

Pi ships zero, and puts it in the README:

Pi does not include a built-in permission system for restricting filesystem, process, network, or credential access. By default, it runs with the permissions of the user and process that launched it.

Both of those are reasonable engineering positions. I am not going to argue that one team is careless. What I want to look at is the space between them, because that is where most people are actually operating, and it is the one position that is not reasonable: running an agent that shows you a prompt, and concluding from the prompt that something is enforcing a boundary.

Two axes, not one

The single most useful thing in the Codex source is that it refuses to collapse permission into one setting. There are two enums, and they are independent.

Enforcement, from codex-rs/protocol/src/config_types.rs:

pub enum SandboxMode {
    ReadOnly,          // default
    WorkspaceWrite,
    DangerFullAccess,
}

Escalation, from codex-rs/protocol/src/protocol.rs:

pub enum AskForApproval {
    UnlessTrusted,
    OnRequest,         // default
    Granular(GranularApprovalConfig),
    Never,
}

The default pairing is ReadOnly plus OnRequest. The agent cannot write anything until something changes, and the model may ask when it wants more.

The Granular variant is worth reading closely, because it is a permission system that has been through contact with reality. It is five separate booleans: sandbox_approval, rules, skill_approval, request_permissions, mcp_elicitations. Five different things can ask a human for permission, and you can switch each of them off independently. When a field is false the request is auto-rejected rather than shown, which is the correct default for an unattended run and the wrong one for an interactive session.

Now hold those two axes apart. Four combinations, and they are four different systems:

	approvals on	approvals off (`Never`)
sandbox on	the normal case. The OS refuses; a human can grant an exception.	CI. Nothing is asked, anything outside the box fails, and the failure goes back to the model instead of to a person. Codex's comment on `Never` says exactly this.
sandbox off	where most people actually are. Every boundary in the system is a human reading a command string.	`DangerFullAccess` plus `Never`. Honest, at least. This is what a container is for.

The bottom-left cell is the one I care about, and it is not a bug in anyone's code. It is a reading error.

What the approval prompt actually measures

An approval dialog shows you a command. It does not show you what the command reaches.

Approving npm test approves the project's test script, plus whatever that script shells out to, plus any lifecycle hook in the dependency tree. Approving make approves a Makefile you did not write. Approving a Python script approves its imports. The string in the dialog is a handle on an arbitrarily large graph, and the graph is what executes.

This is the wrong ruler with a UI. The prompt is not lying. It truthfully reports the command the model proposed. The problem is what people read off it, which is "this operation has been reviewed and bounded," a claim the prompt never made and cannot support.

An approval is a statement of intent. A sandbox is a statement about capability. Intent without capability limits gives you a system whose entire safety property is that a tired human parses shell correctly at the end of a long session.

Codex's design makes the distinction structural. Two enums, and you have to set both. That alone is worth stealing even if you never run Codex.

What 43,591 lines of enforcement looks like

Here is what enforcement costs when a team actually means it. The sandbox surface of the Codex repository, tests included, measured on 2026-09-01:

crate	lines
`windows-sandbox-rs`	21,399
`linux-sandbox`	9,846
`sandboxing`	9,220
`execpolicy`	2,975
`bwrap`	151
total	43,591

Four platform paths: Apple Seatbelt, Linux Landlock, bubblewrap, and a Windows implementation that is by itself the largest of the four. The Seatbelt policies are not generated from some abstraction, they are checked-in .sbpl files: a base policy, a network policy, a preferences policy, and a set of read-only platform defaults.

Set against the agent core, with tests excluded on both sides so the two numbers are the same kind of number, that is 32,774 lines of sandboxing against 125,574. About a quarter of the core, spent entirely on making the operating system say no.

I had that ratio wrong in a draft of this piece, and the way I got it wrong is the subject of the article. I compared the sandbox crates counted with tests against a core counted without them, and got "roughly a third." Both figures were real. Neither was wrong on its own. They just were not measuring the same thing, and the sentence built on them read as authoritative anyway.

Which is exactly why Pi's position is defensible rather than lazy. That is a large, permanently expensive, four-platform commitment. A small team can reasonably decline it and tell users to bring their own container. Pi's README points at three concrete patterns for doing so: a micro-VM extension, plain Docker, and a policy-controlled sandbox.

The prompt that is not a permission

Pi does have a trust prompt, and it is the reason this piece exists. packages/coding-agent/src/core/project-trust.ts asks:

Trust project folder?
This allows pi to load .pi settings and resources, install missing project packages, and execute project extensions.

If you have used any modern editor you have seen this dialog, and you know what it feels like it means. It feels like the thing that decides whether the agent is allowed to touch your machine.

It is not. Read the sentence again. It gates loading project configuration, installing project packages, and executing project extensions. It says nothing about the agent's own tools, because those were never gated. bash is available either way. Answering "no" to that prompt does not put the agent in a box, because there is no box. The README already told us so.

So a user meets a trust dialog, answers it carefully, and walks away with a belief about their security posture that the dialog did not create and does not support. Pi documented the truth in two places. The dialog is still the thing people will remember, because a dialog interrupts you and a README does not.

This is evidence that is not evidence in its purest form. A true signal, about a real decision, borrowed for a claim it was never making.

Why this is worse than it was last year

Because the input got hostile.

I have been making the argument that a downloaded artifact is untrusted input, most recently about GGUF model files, where sixty-five bytes crash a loader. That is a parser problem with a parser fix.

Agent context is not a parser problem. Pi's SECURITY.md is admirably blunt about it:

Pi relies on users installing trustworthy extensions and loading trustworthy skills and only to use pi within trusted repositories. This is because files like AGENTS.md or instructions in comments can be used to prompt inject the coding agent trivially and this cannot be protected against.

That is correct and it applies to every harness in the field, not just Pi. There is no parse step where you can validate an instruction, because instructions are the payload. A comment in a vendored dependency is in the context window with the same standing as your own words.

So the trust boundary sits somewhere unfamiliar. The agent will at some point act on text written by someone else. That is not a hypothetical, it is the design. The only question left is what the process is capable of when it does.

Which is precisely the question a sandbox answers and an approval prompt does not.

What to actually do

Decide which of the four quadrants you are in, and say it out loud. Most people have never made this an explicit choice, and defaults chose for them.

If you run an agent unattended, on a repository you did not write, with network access and your real credentials in the environment, then approvals are not protecting you and never were. Containerize. Pi's own docs tell you how, and Codex will do it for you.

If you run Codex, set both enums deliberately. ReadOnly plus OnRequest is a genuinely good default and it will annoy you into weakening it. When you weaken it, weaken the axis you meant to.

If you run Pi, run it the way its authors say to run it: inside something. The four tools are read, bash, edit and write, and bash is not a smaller capability because there are only four of them.

And whatever you run, stop counting approval prompts as a control. A prompt records that you agreed. Only the operating system records what was possible.

Line counts measured 2026-09-01 against github.com/openai/codex and github.com/earendil-works/pi at that day's HEAD, including test files. Enum definitions quoted from codex-rs/protocol/src/config_types.rs and codex-rs/protocol/src/protocol.rs. Pi quotations are verbatim from its README.md and SECURITY.md.

Three Agent Harnesses, One Loop

Harrison Guo — Tue, 01 Sep 2026 08:01:21 +0000

Claude Code's agent loop is 1,729 lines of TypeScript. Codex's is 983 lines of Rust. Pi's is 794 lines of TypeScript.

Three teams. Three languages. No shared code, no shared lineage, and in Pi's case an explicit design goal of being the small one. The loop lands in the same place anyway.

That convergence is the least interesting thing about these three systems, and almost every comparison I have read stops there. The interesting part is what does not converge.

The reading is legal now

When I took apart Claude Code's query.ts in Part 2 of the deep dive, the only reason it was possible was a leak. That was a real limitation. Line numbers from a leaked bundle are a snapshot of a build nobody can check, and I could not tell you whether any of it survived the next release.

That constraint is gone.

OpenAI published the Codex harness under Apache-2.0 in August 2026, including codex exec, the SDK, and the app-server that hosts the agent core. Pi is MIT. Both are on GitHub with permanent history. Every number in this piece comes from a clone I made on 2026-09-01, and you can check all of them.

It moved the other way for Claude Code, which is worth saying because it cuts against my own series. I checked the install on this machine while writing: version 2.1.252 ships as a 197 MB compiled Mach-O binary. The JavaScript is in there, minified into single lines of 300,000 characters, and its internal codename appears three thousand times in the strings. It is not source. The numbers I published from the leaked bundle cannot be re-derived from what Anthropic actually ships today, by me or anyone else.

So in the same twelve months, two harnesses became readable and one became less so. Every Claude Code figure below is a historical measurement of a build that is no longer inspectable, and I am going to keep flagging that rather than let it pass as current.

With that stated, the useful question stopped being "what does a production agent loop look like inside." We can mostly see. The question now is why three teams solving the same problem produced three architectures that agree about the middle and disagree about the edges.

The middle: everyone wrote the same loop

Strip each implementation to its control flow and you get the same five steps.

Assemble context. Call the model. If the response contains tool calls, execute them. Append the results to the conversation. Decide whether to loop again or stop.

That is it. That is the whole agentic pattern, and it fits in a paragraph.

Here is what it costs each team to actually ship it:

	file	lines	language
Claude Code	`query.ts`	1,729	TypeScript
Codex	`codex-rs/core/src/codex_thread.rs`	983	Rust
Pi	`packages/agent/src/agent-loop.ts`	794	TypeScript

Claude Code's while(true) runs from line 307 to line 1728, so the loop body alone is 1,421 lines. Codex and Pi spread more of the same work across neighbouring modules, so file-to-file is the honest comparison, and file-to-file the spread is 794 to 1,729. Slightly more than a factor of two, across three independent teams.

I want to flag something here, because I got it wrong in print.

That Part 2 article carried a comparison table with a row claiming most open-source agents implement the loop in "~50-200 lines." I believed it when I wrote it. It is false, and Pi is the proof: the most deliberately minimal harness in the field spends 794 lines on its loop. The 50-line version exists in tutorials. It does not exist in anything people run all day. The gap I described as "Claude Code versus open source" was really "production versus demo," and open source has since closed it.

Here is the shape in Pi's own source, trimmed to the control flow. The full function is
runAgentLoopContinue in packages/agent/src/agent-loop.ts, starting at line 121:

while (true) {
  let hasMoreToolCalls = true;

  while (hasMoreToolCalls || pendingMessages.length > 0) {
    // inject any steering messages the user typed mid-turn
    for (const message of pendingMessages) { currentContext.messages.push(message); }
    pendingMessages = [];

    const message = await streamAssistantResponse(currentContext, config, signal, emit, streamFunction);

    if (message.stopReason === "error" || message.stopReason === "aborted") { return; }

    const toolCalls = message.content.filter((c) => c.type === "toolCall");
    hasMoreToolCalls = false;

    if (toolCalls.length > 0) {
      const executedToolBatch =
        message.stopReason === "length"
          ? await failToolCallsFromTruncatedMessage(toolCalls, emit)
          : await executeToolCalls(currentContext, message, config, signal, emit);
      hasMoreToolCalls = !executedToolBatch.terminate;
      for (const result of executedToolBatch.messages) { currentContext.messages.push(result); }
    }
  }
}

Assemble, call, execute, append, repeat. The five steps are all there and they take about
twenty lines.

Now look at the ternary in the middle, because it is the entire subject of this section.

What the extra lines buy

That branch on stopReason === "length" is the loop noticing that the model's output was
cut off by the token limit. When that happens, every tool call in the message may carry
truncated arguments, so Pi fails all of them rather than executing calls whose parameters
might be half a JSON object. The source comment is blunter than I would have been: execute
them and you are running "potentially borked calls."

That is one branch. It is not intelligence, no user will ever see it, and leaving it out
would produce a loop that works in every demo and corrupts a file once a week.

The five-step summary omits every hard part, and each team's loop is mostly the omissions.

A response can stream, so tool calls arrive before the message is finished and you have to decide whether to start executing. Tool calls can run in parallel or must not, depending on what they touch. The user can interrupt mid-turn, which means an abort has to unwind cleanly without corrupting the transcript. Context fills up, so compaction has to fire without losing the thread. Providers fail in ways that are worth retrying and ways that are not. Approvals can suspend the loop indefinitely while a human decides.

None of that is intelligence. All of it is the difference between a demo and a tool, which is the argument I made in The 90% Problem before I could check it against three codebases. Now I can. The loop is 5 steps and 800 to 1,700 lines, and the ratio between those two numbers is the whole point.

The edges: three different boundaries

Here is where they stop agreeing.

Every harness has to answer one question: what is outside the loop, and how does the loop talk to it. Each of these three gave a different answer, and the answer determines nearly everything else about the system.

Claude Code puts the boundary inside the process

The loop, the tools, the permission checks, the compaction and the UI live together. Coordination is function calls and shared state. That buys tight control over things that are painful to coordinate across a wire, and the five-level compression pipeline is the clearest example: it can make decisions using information the loop has not committed to anything yet.

The cost is that everything must be in-process. Another program cannot drive the agent without going through the surface Anthropic chose to expose.

Codex puts the boundary on a protocol

Codex pulled the agent core into app-server and put a documented, bidirectional JSON-RPC 2.0 interface in front of it. The README is explicit about the three primitives:

Thread: A conversation between a user and the Codex agent. Each thread contains multiple turns.
Turn: One turn of the conversation, typically starting with a user message and finishing with an agent message.
Item: Represents user inputs and agent outputs as part of the turn, persisted and used as the context for future conversations.

That is not an implementation detail written down after the fact. It is a contract. thread/start, thread/resume, thread/fork to branch with copied history, turn/start, turn/interrupt, streaming item/started and item/completed notifications, and turn/completed carrying final token usage.

Transports are stdio by default, with a unix socket and an experimental websocket listener. There is even backpressure in the protocol: saturate request ingress and you get JSON-RPC error -32001, "Server overloaded; retry later." A harness that has an overload error code has stopped thinking of itself as a CLI.

The consequence is that the VS Code extension, the terminal, and any third-party client are all peers. None of them is the real one. The agent is a service, and the loop is what runs inside it.

The cost is versioning. Once thread and turn are wire types, changing them is a compatibility event.

Pi puts the boundary at the extension host

Pi ships four tools: read, bash, edit, write. Everything else is a TypeScript extension, loaded from the project, hot-reloadable, and able to register tools, persist state into the session, and render its own terminal components.

That is a genuinely different bet. Codex says the interesting extensibility is other processes driving the agent. Pi says it is the agent growing new tools inside your repository, at runtime, in the same language as the harness. Pi deliberately ships no MCP support at all, which I will come back to in a later piece, because the reasoning is better than the summaries suggest.

The cost is that the boundary is a language boundary. Extensions are TypeScript because the host is TypeScript.

The minimalism that is not

This is the part that surprised me most, and it is the part the coverage keeps getting wrong.

Pi's reputation is radical minimalism. Four tools. A system prompt small enough that people quote its token count. That reputation is earned. I measured the default prompt at 550 tokens, which is genuinely tiny, and I will take that number apart properly in a later piece because the interesting half is what it leaves out.

Now count the codebase.

	source lines
Pi, `packages/*/src`, tests excluded	121,240
Codex, `codex-rs/core/src`, tests excluded	125,574

The minimal agent and the OpenAI agent core are within four percent of each other.

Inside Pi that is 60,960 lines in the coding agent, 23,668 in the provider layer, 17,000 in the terminal UI, 12,640 in the agent package. This is not bloat and it is not a gotcha. It is what shipping takes.

Both things are true at once, and holding them together is the actual lesson. Pi minimizes what the model has to reason about. It does not minimize what the team has to maintain, and it never claimed to. Those are two different surfaces, and almost every writeup I have seen collapses them into one sentence about how small Pi is.

The same confusion runs the other way with Claude Code. A 510,000-line bundle sounds like the opposite of minimal, until you notice the model still only sees a loop and a tool list.

Model-facing surface and engineering surface are independent axes. You can be small on one and large on the other, and every one of these three is.

Why the boundary is the decision

Once you see the boundary as the design, the rest of each system reads as consequence rather than taste.

Codex has 43,591 lines of sandboxing crates and four platform backends because a service that anyone can drive cannot assume the caller is a trusted human at a terminal. Pi has none of that, and says so in its README, because it assumes exactly that. Neither is careless. They are answers to different questions, and I will take that specific difference apart next, because it is the one with security consequences that people are getting wrong right now.

Codex's thread/fork and Pi's parent-linked session tree exist for the same reason and were reached independently: once a session is a real object rather than a transcript, branching is cheap and re-running from a known point stops costing you the whole context.

This is the same move as the technique boundary. Deciding where determinism ends and judgement begins is the design. Deciding where your process ends and the outside begins is the same kind of decision, one level down.

How to choose

Not by benchmark. Benchmarks of harnesses measure the harness plus the model plus the configuration, which is a trap I will spend a whole piece on shortly.

Choose by boundary.

If other processes need to drive the agent, take Codex. The protocol exists, it is documented, it generates its own TypeScript and JSON Schema, and rebuilding that on top of a library is months of work that ends worse.

If the agent needs to grow tools inside your codebase and you can live in TypeScript, take Pi. The extension host is the shortest path from "the agent should be able to do X here" to the agent doing X, and hot reload means the agent can write and test its own tools.

If neither is true, you may not need a harness. A loop you understand, with the four tools everyone converged on anyway, is 800 lines. That is not a small number, but it is a knowable one, and now you can read three implementations before you write yours.

The loop was never the moat. It was the part that was always going to converge.

Pi and Codex measurements taken 2026-09-01 against github.com/earendil-works/pi and github.com/openai/codex at that day's HEAD, and they are reproducible. Claude Code figures come from the leaked bundle analysed in Part 2. I tried to re-verify them against the shipping artifact and could not: Claude Code 2.1.252 is a compiled binary with minified JavaScript embedded, so line counts from the leaked source have no counterpart in it. Treat those three numbers as a snapshot of an older build, not a current measurement.

Shamir Secret Sharing, Explained With Pictures

Harrison Guo — Tue, 01 Sep 2026 01:59:56 +0000

There is a claim in the seed phrase write-up that deserves more than an assertion:

Two shares don't get you "most of the way there". They get you nowhere.

That sounds like marketing. It is not — it is a property you can draw on paper in about a minute, and once you have seen it drawn you will not need to trust anyone about it again. So here is the drawing.

Start with two points and a line

Pick any two points on a page. Exactly one straight line passes through both of them. Not "roughly one" — one. Take away either point and infinitely many lines fit the survivor.

That asymmetry is the entire scheme. Adi Shamir published it in 1979, and the trick is to hide a secret inside a curve rather than inside a container:

Put the secret at one specific spot on the curve — where it crosses the axis, at x = 0.
Draw the rest of the curve however you like, at random.
Hand out other points on that curve. Those are the shares.

Nobody holds the secret. Everybody holds a point that the secret's curve happens to pass through.

Look at the left panel. One point, and five different lines drawn through it, each hitting the axis somewhere different. There is nothing special about those five; you could draw a line through that point to any intercept you like. So a person holding one share cannot narrow the secret down at all. Not to a range, not to a likely region. Every value is still exactly as plausible as it was before they had the share.

Now the right panel. A second point fixes the slope, one line fits both, and following it back to the axis gives the secret. Two shares, done.

Three of five is the same trick, one degree up

A line is fixed by two points. A parabola takes three. Curve upward through degree and you get any threshold you want:

2-of-N: put the secret on a line, hand out points on it. Any two shares reconstruct the line.
3-of-N: put the secret on a parabola. Any three reconstruct it; two leave infinitely many parabolas fitting, one for each candidate secret.
k-of-N: a curve of degree k−1.

The N is free. Five shares from a parabola, or fifty, and the threshold stays at three either way, because it is a property of the curve's degree rather than of how many points you gave away. That is why 3-of-5 and 3-of-50 are equally secure against two attackers, and why generosity with shares costs you nothing in threshold terms.

The honest footnote about the picture

The drawing uses ordinary school-graph arithmetic, and real implementations do not. Points on a real graph leak information: a share sitting at an extreme height narrows what the intercept could plausibly be, and floating-point rounding would corrupt the reconstruction anyway.

So implementations work in a finite field — arithmetic on a fixed set of values that wraps around instead of running off to infinity, usually GF(256), which is to say bytes. Addition, multiplication and division all still work, the "one curve through k points" theorem still holds exactly, and there is no rounding because there are no fractions. What changes is that "extreme value" stops meaning anything: every share is a byte, uniformly distributed, and the geometry that made the picture intuitive stops leaking the hints that made it intuitive.

The picture is true about the structure. The field is what makes it true about the security.

Where SLIP39 fits, and what it is not competing with

If you have looked into split backups for a wallet you will have met SLIP39, and probably seen it framed as "SLIP39 vs BIP39". That framing is misleading.

BIP39 is not a splitting scheme. It turns one seed into one phrase of English words. There is nothing to compare — it solves encoding, not distribution.

SLIP39 applies Shamir's scheme and then encodes each resulting share as its own word list, so the thing you write down looks familiar. It adds grouping on top, so you can express rules like "two of the three family shares, plus the lawyer's".

So the real comparison is one backup versus a threshold of several, and it is a question about failure modes rather than about formats. One phrase in one place fails when that place fails, in either direction: lost, or found.

Why a threshold beats a safe

A safe is a single point of failure wearing armour. It is very good at the thing it does — resisting force — and it concentrates every other risk into one location. Whoever opens it has everything. If it burns, everything is gone.

A 3-of-5 split inverts that. There is no single object to steal, and no single place to lose. An attacker needs to find and take three separate things, and you can lose two of them to fire, flood, a house move or a forgotten hiding place and still be fine.

What it does not do is survive time. Which brings us to the part that gets skipped.

What splitting does not fix

It does not make anything durable. Five copies that all degrade are five copies of nothing. Splitting solves the single point of failure problem, not the the medium died problem. A steel plate is still better than any digital medium at surviving a house fire, and the two are complements rather than alternatives.

Recovery reassembles the whole secret in one place. For the moment reconstruction happens, on whatever device does it, the complete secret exists — and that device is now the single point of failure the scheme was built to eliminate. The protection is real in storage and absent in use. Anyone telling you a threshold scheme means the secret "never exists in one place" is describing something else.

It does not choose your locations for you. Three shares in three drawers of the same desk is a 3-of-5 split with the security of one drawer. The mathematics is only as good as the independence of the places.

A real run, and the one thing the diagram leaves out

Everything above is the scheme. Here is our own CLI actually doing it, because a
drawing of a threshold and a threshold are different claims.

Five ordinary photos, one sentence hidden across them:

$ stegosafe_cli.py embed -i in -s "the mitochondria is the powerhouse of the cell" -o out
Key length: 32 bytes, Ciphertext length: 74 bytes
Key split into 5 shares
Share 1: x=1, y length=32
Total data length to embed: 107 bytes
Saved: out/stego_1.png
...
Encryption and embedding complete. Created 5 stego images

Read the second line again, because it is the part the diagram does not show.
What gets split five ways is not the secret. It is the 32-byte AES key. Every
image carries the whole ciphertext plus one share of the key — 74 + 32 bytes,
107 with framing. You need three images because that is how many it takes to
rebuild the key, not because the message is in pieces.

That is a better design than splitting the message would be, and it is worth
saying why: a share of a key is a fixed 32 bytes whatever the secret's length, so
the scheme costs the same whether you hide a twelve-word phrase or a paragraph.
It also means every image is independently useless in the same way — ciphertext
with no key is ciphertext.

Now delete two of the five and recover from what is left:

$ rm out/stego_2.png out/stego_4.png
$ stegosafe_cli.py recover -i out
Collected 3 shares, starting recovery
Recovered key length: 32 bytes

Recovered secret:
the mitochondria is the powerhouse of the cell

And with only two:

$ stegosafe_cli.py recover -i two
Error: At least 3 stego images are required to recover the secret
Found 2 stego images

It refuses. It does not return a partial phrase, and it does not return a wrong
one — which matters more than it sounds, because a recovery tool that guesses is
worse than one that stops.

The cost to the photographs, measured across the five: between 0.0067% and
0.0117% of pixels changed, and no colour channel moved by more than 1 out of
255. The percentage is higher on the smallest image for the obvious reason —
the payload is a fixed number of bytes, so it occupies a larger share of a
smaller picture.

Seeing it rather than believing it

The claim at the top of this page is checkable in ten minutes and does not require trusting us. Split a sentence you do not care about into five shares, delete two, recover from three. Then take a single share and try to get anything out of it at all.

The command-line tool is open source if you would rather read the implementation than the explanation. What StegoSafe adds on top of the scheme described here is where the shares live — each one inside an ordinary photograph rather than in a file that announces itself as a share — which is the subject of the seed phrase write-up, and of what actually happens when you hide a secret in a photo.

Your Model File Is Untrusted Input

Harrison Guo — Wed, 26 Aug 2026 15:51:53 +0000

Sixty-five bytes is not a model. It is not even a header with something plausible after it. It is a GGUF magic number, a few counts, and a tensor whose second dimension is zero.

Load it and llama.cpp dies.

Not with a memory error, and not because anything was corrupted. It dies dividing by a number that the file format is supposed to allow.

The loader is the trust boundary

Nobody thinks of a model file as input. You download a .gguf from a hub the way you download a JPEG, and then a C++ binary parses it into heap allocations, tensor shapes and metadata keys. Everything downstream reaches the same parser: llama-server, the bindings, the desktop apps that wrap them.

gguf_init_from_buffer is where a byte array becomes a model. That makes it the boundary, and boundaries are worth fuzzing.

The harness is unremarkable: libFuzzer and AddressSanitizer pointed at that one function. Binary formats with length fields are where parsers go wrong, and a model file is a binary format that is nothing but length fields.

Sixty-five bytes, field by field

The file that kills it, laid out:

offset	size	field	value
0	4	magic	`GGUF`
4	4	version	3
8	8	tensor count	1
16	8	metadata kv count	0
24	8	name length	1
32	1	tensor name	`t`
33	4	`n_dims`	2
37	8	`ne[0]`	1
45	8	`ne[1]`	0
53	4	type	0 (`F32`)
57	8	offset	0

A version the loader supports. One tensor, no metadata, a one-character name, two dimensions, the first of them 1. Fifty-seven of those sixty-five bytes are the least remarkable file you could construct.

The other eight are at offset 45, and what makes them fatal is that they are zero, not that they are wrong. Nothing here is malformed. There is no length lying about a buffer, no count that overruns an allocation, no string missing its terminator, none of the things a parser is written to be suspicious of. Every field is in range, including that one.

The first crash, and the check that was already there

Tensor dimensions are validated. The loader checks that each one is non-negative, which reads like exactly the right thing to do, and it has been there a long time.

Then the next check tests whether the element count can be represented at all:

if (ok && ((INT64_MAX/info.t.ne[1] <= info.t.ne[0]) || ...

That divides by ne[1]. A dimension of zero passes the first check, because zero is not negative, and arrives at the second, where INT64_MAX / 0 raises SIGFPE and the process is gone.

AddressSanitizer: FPE ... gguf.cpp in gguf_init_from_reader(...)

What makes this one worth writing down is not the arithmetic. It is that a validation check sat directly above the crash, and it passed review, and it is correct as far as it goes. ne[j] >= 0 is a true statement about what a dimension may be. It just is not the statement the line below it needed. A check that half-covers a value is worse than no check, because the reader below it stops asking. That is the wrong ruler again, a check that reads as complete and quietly is not.

Zero is legal, so rejecting it was never the fix

The obvious patch is to demand a positive dimension. That patch is wrong.

Zero-sized tensors are allowed on purpose. They exist in real files, where some diffusion models carry them as feature tags, marking a capability rather than holding data, and safetensors permits them too. A loader that began refusing them would reject models that are not malformed at all.

So the fix is not about the input. It is about the arithmetic: skip the representability check when a dimension is zero, because an element count of zero is trivially representable. Nothing is rejected that was not already rejected. The division simply stops happening on a value that was always going to be there.

This came up in review. The maintainer's recollection was that zero-sized tensors were tolerated but pointless; a collaborator corrected it with the diffusion case. The correction is the reason the patch stayed narrow.

The second crash: a type nobody checked

The other one is smaller and more ordinary.

GGUF metadata is typed. Each key declares what it holds, and the reader is supposed to honour that declaration. general.alignment is read as a uint32, without asking what the file said it was.

A file that declares general.alignment with any other type reaches a template whose whole job is to notice that mismatch:

GGML_ASSERT(type_to_gguf_type<T>::value == type) failed

An assertion is not a safety mechanism against hostile input. It is a statement that something impossible has happened, and its response is to abort. Here it was reachable from a 62-byte file, which means the impossible thing was merely undeclared.

The fix is a type check before the read, and a clean failure instead of an abort.

What review changed

I opened the pull request with the fix and the two reproducers. Three things came back, and each one made it better than what I sent.

Tests. The first comment asked for regression tests, which was the right call and the part I had skipped. Two handcrafted files went into tests/test-gguf.cpp: a metadata key with the wrong type, and a tensor with a zero-size dimension. Both crash the old build and load cleanly against the fix. The suite went 164/164.

The fix itself. My patch tested the dimensions one at a time. The review asked for ggml_nelements(info.t) > 0 instead. Identical behaviour, but it says what it means: skip the overflow check when there are no elements to overflow. Checking each dimension is how you think while you are staring at a division. Checking the element count is what the condition was always about, and it is one statement instead of several, which matters for a check whose whole failure mode was that a reader trusted it without enumerating the cases.

The error message. Rejecting a wrongly-typed key is not much help if it won't say what it wanted. It now reports both sides: key 'general.alignment' must be of type u32 but is i32.

That exchange is worth more than the patch. A fix without a regression test lasts until the next refactor, and a fix phrased as three separate conditions is an invitation for someone to eventually need a fourth.

Someone else found it the same week

Nine days after the pull request went up, a comment appeared on it from someone running an unrelated campaign against gguf_init_from_buffer, libFuzzer with ASan and UBSan at around 400,000 iterations, who had landed on the same SIGFPE by the same route.

That part is only mildly interesting. Two fuzzers aimed at the same function find the same shallow bug; that is what shallow means.

What was worth reading is what they did next. They went through the surrounding overflow check by hand to see whether the patch was complete:

the ne[0]/ne[2]/ne[3] multiplications can't independently overflow due to the || short-circuit, so the zero-check in this PR looks like it fully covers the defect rather than being a partial fix

That is the right question, and it is the same question this bug was made of. ne[j] >= 0 was a partial guarantee that read as a complete one, and it went unexamined precisely because it was there. Somebody checking whether its replacement had the same shape is the review this class of defect needs and almost never gets.

The part that generalises

Two things, neither of them about GGUF.

A downloaded model deserves the suspicion you give a downloaded binary. It is parsed by native code before anything else happens to it. These two bugs stop at denial of service, which is survivable when you are crashing your own process and much less so when a service loads models on someone else's behalf. The category is the point, not the severity.

Validation that half-covers a value is a liability. ne[j] >= 0 is not wrong. It is simply not the guarantee the line below it depended on, and its presence is what made the line below it look safe. When you find a bounds check, the useful question is not whether it is correct but what the code underneath it now assumes. That is the same reflex as reading a signal that is present but not diagnostic: the check ran, and running is not the claim you needed it to make.

The fix is in llama.cpp#25596, merged August 2026, with both reproducers in the description. If you run models locally, you already have the patched loader.

This is a Parser Field Note. It shares its spine with A Wrong Ruler Is Worse Than No Ruler on a check that reads as complete and is not, and The Log Printed Exactly What I Wanted on a true signal that supports a smaller claim than you lean on it for.

Merged Is Not Shipped: Why a Year-Old Fix Never Reached You

Harrison Guo — Sat, 22 Aug 2026 18:52:49 +0000

There is a version of a bug that is more frustrating than an unfixed one. It is a bug that was fixed, correctly, and then sat behind glass for a year while the people it was hurting kept reporting it, because the fix and the users were in two different repositories and nothing moved the one pointer between them.

I ran into exactly this. The connectivity failure I wrote up separately, where an IPv6-only cellular network quietly rewrites a VPN address and one code path dies, has a clean upstream fix. It was merged in July 2025. It is small, it is correct, and its commit message describes the exact situation I spent a night diagnosing. And more than a year later, the users hitting the bug still could not get it, including if they built the app from source. Not because the fix was hard, or hidden, or controversial. Because one git submodule pointer was never advanced.

Merging changes the source of truth. It does not move the pointers between that source and a user's install. A fix is shipped only when every pinned reference on the path from the commit to the running binary has been advanced, and in a dependency graph that last move is owned by no one.

This is the delivery half of a two-part field note. The first part was about recognizing a failure that logs no error. This part is about a fix that shows every sign of being done and reaches nobody, which is the same illusion one layer up. It is also, underneath, the completion-ownership problem again: merged is the software-supply-chain version of a 202 Accepted. It says the request was received, not that the effect happened. Somebody has to own the claim that the fix actually reached users, and in an open dependency graph, nobody does.

The shape of the gap

The project here is structured the way a great many projects are. There is a core library that does the real work, and there are several client applications that each embed that library as a git submodule pinned to a specific commit. The pin is deliberate and correct engineering. It makes every client's build reproducible and keeps an upstream change from silently altering a released app. It is also, for the same reason, a freeze: the client is stuck at whatever point in the library's history the pointer names, no matter what has been fixed since.

The fix landed in the library. Then look at where the three clients had their submodule pinned:

client	pinned library commit	has the fix
desktop (Qt)	mid-2026	yes
Android	early 2024	no
iOS	early 2024	no

The iOS client's pointer was two dozen commits behind the fix. The App Store build was older still. So an iOS user could not get the fix by updating the app, and could not get it by building from source either, because building from source faithfully builds the pinned commit, which does not contain it.

Then the detail that makes the whole thing land. The fix's own commit message says it is a workaround for iOS. It was written for iOS. And the only one of the three clients carrying it was the desktop client, which does not run on iOS. The fix reached every platform except the one it was written to help.

Meanwhile the issue tracker for the client had the bug open for a year and a half, collecting me too comments and workaround folklore, with no one having connected it to a commit that had been sitting in a sibling repository the entire time. The answer existed. It was one git log away. The distance between the fix and the people who needed it was not technical. It was organizational, and it was invisible from both ends.

Why the gap is invisible from both ends

The delivery gap survives because everyone who could close it is looking at a view where it does not appear.

The library maintainer merges the fix, the tests pass, the issue in their tracker closes, and by every signal available to them the work is done. Their instrument does not show downstream pins. Nothing tells them that a dependent client is twenty-four commits behind and that real users are stranded on the far side of a pointer only that client's maintainer can move.

The client maintainer is not watching the library's commit log commit by commit. Bumping a submodule is a deliberate, manual, and slightly risky act, because moving the pointer can drag in unrelated changes and API drift along with the one fix you wanted. It is unglamorous, it is easy to defer, and nothing schedules it. There is no notification that says a dependency you pin has fixed a bug your users are reporting.

The user sees only the bug, and an issue tracker full of other people seeing the same bug, which reads as strong evidence that the problem is simply unsolved. The one place the answer actually lived, the commit history of a dependency, is not a place most users think to look, and the tracker gives them no reason to.

So the fix sits in the seam. Merging felt like the finish line to the one person positioned to know the fix existed, and moving the pointer was nobody's defined job, and the people motivated to close the gap were the only ones who could not see where it was.

This is every lockfile you have

It would be comforting to file this under git submodules, which have a reputation for exactly this kind of pain, and move on. But the submodule is only the most visible instance of a mechanism that is everywhere in modern software, and that mechanism is the pin.

Every reproducibility guarantee you rely on is a frozen pointer. package-lock.json, go.sum, Cargo.lock, a pinned requirements.txt, a Docker FROM image:tag, a vendored module, a Helm chart's image digest. Each one exists to protect you from unwanted change, and each one, by protecting you from unwanted change, also holds you at a known-good point that can quietly become a known-stale point the moment an upstream fix lands above it. The property that makes your build trustworthy is the same property that makes it a place fixes go to wait. You do not get deterministic builds and automatic fix delivery from the same pin. You get one, and you buy the other separately, with deliberate effort, or you do not get it at all.

Which means the question is a fix I need already merged upstream is the wrong question, and answering it yes is how you end up stranded with false comfort. The right question is has that fix crossed every pin between its commit and the thing I actually run. Those are very different questions, and the gap between them is measured in the months a user spends believing a bug is unsolved.

What to actually do

For the person chasing a bug, the discipline is small and it saves real time. When you find that a bug is fixed upstream, do not stop at merged. Compare the fix commit against the exact revision your project pins, not against the upstream tip, and read the number of commits you are behind. Read commit history rather than the issue tracker, because the tracker is where the duplicate confusion pools and the commit log is where the resolution actually sits. And if you can, cherry-pick the fix onto your pinned revision and test it, which is what turns a guess into a verified claim and, incidentally, gives a maintainer a concrete reason to bump the pointer. That last step is the same move as everywhere else on this site: do not assert that it works, prove it in a loop closed against reality.

For the person maintaining a project that pins dependencies, the fix is structural, because relying on a frustrated user to eventually investigate is not a delivery process. Give the pointer an owner. Make advancing pins a tracked, routine task rather than an act of archaeology, and let automation open the update pull requests so that a dependency's fixes become visible events in your own repository instead of silence. The goal is that the gap between merged and shipped becomes something a system watches, rather than something only a stranded user ever notices.

What was scarce, again

The fix was never the scarce thing. It existed, correct and complete, for over a year. What was missing was the single unglamorous act of moving a pointer, an act that no dashboard flagged, no notification raised, and no person had been assigned. Everyone involved was behaving reasonably inside their own view. The maintainer had merged. The user had reported. The pointer sat between them, load-bearing and unowned, and a working fix helped no one for a year because of it.

Merged is not shipped. The commit is only the beginning of the fix's journey to a user, and most of that journey runs through pins that someone has to advance on purpose. Until the last of them moves, the fix is real, and it is doing nothing.

This is the delivery companion to Your VPN Is Fine. iOS Swapped the Address., the diagnosis that turned up the year-old fix. It shares its spine with Tool Calls Need Completion Ownership, where merged plays the role of accepted and shipped plays the role of observed, and with Validation Is a Loop, Not an Assertion on proving delivery rather than assuming it.

The Log Printed Exactly What I Wanted. The Fix Was Still Wrong.

Harrison Guo — Wed, 19 Aug 2026 15:31:14 +0000

I spent two days on a bug where a screen-capture stream never came back after the display went to sleep. When I finally had a patch I believed in, I ran it, put the machine's display to sleep, woke it, and watched the log print the exact line I had been chasing for two days:

Display [1] woke from sleep, reinitializing capture

That is the line. Detection working, capture reinitializing, precisely as designed. I had the confirmation I wanted. About ten seconds later the entire host process died.

The detection was right. The log line was true. And the patch it belonged to was worse than the bug it fixed, because the original bug only failed to recover, while my version took the whole process down with it. The message I had been hoping to see is exactly what nearly convinced me the thing was ready to ship.

Every piece of evidence supports exactly one claim. Trouble starts when you borrow it for a bigger claim than it can carry. A log line proves the line ran, not that the fix is right. Before you trust a signal, name the claim it actually supports, and check that it is the claim you are making.

This belongs to the same first-principles line as a wrong ruler is worse than no ruler and validation is a loop, not an assertion. A wrong ruler is about a measurement that reports the wrong number. Validation-is-a-loop is about proving an outcome rather than asserting it. This piece is about the layer underneath both: the raw signals you reason from, and the quiet ways a true signal gets promoted into a false conclusion. It came out of the same investigation as the NAT64 address bug, and it happened four separate times in one day.

The claim a signal actually supports

The frame that would have saved me each time is small. Every signal, a log line, a captured packet, a returned value, an assistant's answer, is true about one specific thing. It supports one claim. The failure is never that the signal lied. It is that I read one claim off it and used a larger one.

The log line above supports the claim the code detected the wake and started reinitializing. It does not support the reinitialization succeeded, and it certainly does not support the process is healthy. I used it for the third claim. Here it is written out, and then the other three from that day, because together they map the distinct ways a true signal turns into a wrong conclusion.

Case one: a signal that is present but not diagnostic

The wake line printed and the process died ten seconds later because the patch detected the event correctly and then tore down the capture in the wrong order. It released the video outputs while the capture session still held them, and the runtime hit a fatal over-release a moment later. The correct teardown, the one the normal stop path already used, removes the outputs from the session and its bookkeeping first, so that by the time anything is released nothing else is still pointing at it. My version skipped that and went straight to stopping the session, which felt equivalent and was not.

The point is not the ordering bug. The point is that the log line and the crash were both true, and the log line was upstream of the crash in time, so it printed before anything went wrong and told me nothing about what came after. A signal that fires at the start of an operation cannot testify about the end of it. It was present, it was accurate, and it was not diagnostic of the thing I was using it to judge. Had I closed the loop the way validation is a loop argues, checking that the process was still alive and streaming ten seconds later rather than trusting the announcement that it had started, the patch would never have looked ready. This is also the completion-ownership gap in miniature: woke from sleep, reinitializing is the accepted signal, and the process is still up and delivering frames is the observed effect, and I reported the first as if it were the second.

Case two: a signal that is correct but incomplete

Earlier that day I needed to prove a test had actually run over cellular rather than accidentally over Wi-Fi. The server logged the connecting client at an address on my VPN, I saw it, and I took it as proof the phone was on the cellular path. The address was genuinely the client's address. It was completely correct. It also proved nothing, because that same VPN address is what the phone presents on Wi-Fi too. The signal was accurate and it was insufficient, and the gap between accurate and sufficient is exactly where the wrong inference lived.

The evidence that would actually have supported the claim was different: the peer endpoint the VPN was using at that moment, which on cellular was a global address on the carrier's network and on Wi-Fi would have been a local one. That is the signal that distinguishes the two cases. The one I had could not, no matter how true it was. Reaching for a real but insufficient signal because it points the way you want is the same failure a wrong ruler describes, one measurement standing in for another it only resembles.

Case three: a fix that is plausible but wrongly scoped

When I first understood the address bug, I had a fix ready that was clean and obvious. Force the resolver to keep IP literals verbatim, so the address I typed is the address that gets dialed. It reads well, it is one flag, and it would have quietly broken every user who legitimately needs address synthesis to reach a public host from an IPv6-only network. The plausibility of the fix was doing the work that evidence of its correctness should have been doing.

The fix that actually shipped upstream scopes the fallback narrowly, keeping the raw literal only for private and carrier-grade-NAT ranges, because those addresses do not exist on the public internet and never needed synthesizing, while leaving synthesis intact for everyone else. The difference between the two is not cleverness. It is that the narrow one asked who does this break and the broad one only asked does this fix me. A solution feeling tidy and complete is not evidence that it is correct, and the tidier it feels the more it invites you to skip the question of who falls outside its scope.

Case four: no signal at all where one was expected

The fourth one was the emptiest. A build configuration step hung for ten minutes and produced no output whatsoever. The log simply stopped, mid-sentence, on an ordinary line, with no error and no progress. It is tempting to read a long quiet stretch as slow work, and I started to.

What had happened was that the build's search for a Git executable matched a graphical application on the machine that happened to be named like Git, launched it, asked it for its version, and waited forever for a GUI app to answer a question it would never answer. There was no error because nothing had failed. A process was simply blocked on something that would never return, and a blocked process emits nothing. The absence of output was the signal, and I only read it by looking at what the process was actually doing, its children, one of which was a window-less app sitting open, rather than waiting for a log line that was never going to come. Silence past the expected time is not the absence of information. It is information, and it usually means look at the state directly.

Four ways a true signal lies

Lined up, the four cases are a small taxonomy of how evidence misleads without ever being false:

Present but not diagnostic. The wake line was true and fired before the failure, so it could not speak to it.
Correct but incomplete. The client address was really the client's and still could not distinguish the two networks.
Plausible but wrongly scoped. The verbatim-literal fix was clean and would have broken a class of users the narrow fix protects.
Absent where expected. The silent hang produced no error because nothing failed, only blocked.

None of these is a fabrication. Every signal was real. Each one supported a claim, and in each case I attached it to a larger claim it could not hold. The discipline is not to distrust evidence. It is to know exactly what each piece proves and refuse to let it prove one inch more.

The assistant is another signal

All four happened while I was pairing with an AI assistant, and it was genuinely fast and genuinely useful. It read both codebases in minutes. It decoded a synthesized address to its embedded IPv4 by hand. It was also the source of the wrongly scoped fix in case three, a fix that was fluent, confident, and worse than the one already merged upstream.

The lesson people usually reach for here is that AI produces confident wrong answers. That is true, and it is tired, and it turns the observation into a complaint. The sharper version is that the assistant's output is one more signal, subject to the same single question as every other signal in this piece: what claim does this actually support? A fast, well-read, fluent answer is still just an answer, and fluency is not evidence of correctness any more than a hoped-for log line is. Building with these tools does not change the discipline. It raises the stakes on it, because the wrong signals now arrive faster and better argued. Naming the claim before trusting the signal is exactly what keeps a persuasive wrong answer from becoming a shipped one, which is the whole reason judgement matters more in this workflow rather than less.

What was real, and what it proved

In every one of these the evidence was real. The log line printed. The address appeared. The fix compiled and read cleanly. The build genuinely had no error. Not one of them was noise or fabrication, and every one of them was borrowed for a conclusion it could not support. That is the failure mode worth internalizing, because it survives more experience and better tools than the obvious kind. You get better at reading logs, and the misleading log line still reads as confirmation. You get faster with an assistant, and the plausible wrong fix still arrives plausible.

The habit that holds up is boring and it is cheap. For each piece of evidence, before it becomes load-bearing, say out loud the one claim it actually supports, and then check that against the claim you are about to make. Most of the time they match. The day they do not is the day the log line you were hoping for talks you into shipping the thing that breaks.

This piece sits in the Generative Systems, First Principles line, alongside A Wrong Ruler Is Worse Than No Ruler, Validation Is a Loop, Not an Assertion, and Tool Calls Need Completion Ownership. The four cases here came out of the investigation in Your VPN Is Fine. iOS Swapped the Address.

Originally published at harrisonsec.com.

Your VPN Is Fine. iOS Swapped the Address.

Harrison Guo — Sat, 15 Aug 2026 22:34:59 +0000

Here is a failure that survives every reasonable thing you can check.

One app, on an iPhone, on cellular data, talking to a machine on your own VPN. The app pairs. It fetches the host's info. It loads the list of things you can launch. Then the one connection that actually matters, the streaming session, hangs for ten seconds and times out. Switch the phone to Wi-Fi and everything works. Switch back to cellular and it dies again, in the same place, every time.

So you check the obvious things, and they are all fine. The server's ports are bound and listening. The firewall is off. The VPN is up. From a browser on the same phone, over the same cellular connection, you can open the exact host and port the app is failing to reach. It loads instantly. You push a few hundred kilobytes through the tunnel in both directions and it arrives without a stall. Every network path you can test is clean. And one specific connection, in one specific app, still refuses to complete, with a timeout that names nothing.

This is not a story about finding a bug. The bug was found and fixed upstream a year before I ran into it, and I will get to that. It is a story about a failure mode that is nearly impossible to recognize from the inside, and about the one observation that makes it obvious.

On an IPv6-only network, getaddrinfo() on an IPv4 literal does not hand you back that IPv4. It hands you a synthesized IPv6 address that the carrier will translate on the way out. If the literal was a VPN address, its synthesized form is outside the VPN's routes, so the packet leaves the tunnel and dies at the carrier's gateway. Nothing is misconfigured, and nothing logs an error. The bug lives in an address you were never shown.

The unit is unfamiliar, the shape is old

This is a networking field note rather than a first-principles essay, but the reason it is worth writing is the same reason a wrong ruler is worse than no ruler: the evidence you have is not the evidence you need, and it points confidently in the wrong direction. Every signal here reads as the network is fine or the connection timed out, and both are true and neither helps. The connection did time out. The network was fine. The address changed under you, silently, in one of the two code paths, and no instrument in your normal kit is pointed at that.

The symptom fingerprint

Before any capture, the shape of the failure already rules out most of what you would guess. Write it down as a fingerprint, because recognizing it is the entire skill here:

It fails on cellular and works on Wi-Fi.
Inside a single app, one protocol works and another does not.
The server is provably reachable at the same address and port from a different client on the same phone.

Here is what I verified was not the problem, all on cellular, before I stopped guessing and started capturing:

Check	Result
Server ports bound, both address families	listening on all interfaces
macOS firewall and VPN shields	off
Same host and port in Safari, over cellular	opens instantly
ICMP through the tunnel	fine
TCP both directions through the tunnel	fine, a few hundred KB in a quarter second
The streaming handshake reaching the host	never arrives

The last row is the tell. Not arrives and is rejected. Not arrives late. Never arrives. The server never logs the incoming connection at all, while HTTP requests from the same app keep landing normally throughout. One app is reaching the host on some connections and, on one connection, sending packets somewhere the host never sees.

Why one app splits into two destinations

Everything turns on a detail that is invisible unless you already know to look for it: the two kinds of connection are built two different ways.

The HTTP calls, the ones that pair and fetch info and load the launch list, go through the platform's high-level networking, NSURLSession and CFNetwork. You hand that stack an IPv4 literal and it hands the literal to the system, and the VPN routes it the way it routes everything else. It works.

The streaming connection does not use that stack. It resolves the address itself, in portable C, with getaddrinfo(). And on an IPv6-only network with NAT64/DNS64, getaddrinfo() on an IPv4 literal does something most people have never had a reason to notice. It does not return the IPv4 you passed. It returns a synthesized IPv6 address: the carrier's NAT64 prefix with your IPv4 embedded in the low bytes, so that IPv6-only clients can still reach IPv4-only hosts. That is a genuinely useful feature, and for a public IPv4 host it is exactly what you want.

But your VPN address is not a public host. When the IPv4 literal is a VPN range address and it gets synthesized into a NAT64 IPv6 address, the result is no longer inside the VPN's routes. It matches the carrier's prefix, so it leaves the tunnel entirely, goes to the carrier's NAT64 gateway, and the gateway tries to find a CGNAT address on the public internet that does not exist there. The SYN retransmits into silence until the ten second timer fires.

Same app. Same host. Same second. Two code paths. Two destinations.

Same app, one host: an IPv4 VPN literal
   |
   |-- HTTP (pair, info, launch)
   |       -> CFNetwork / NSURLSession passes the literal through
   |       -> VPN routes it inside the tunnel
   |       -> reaches the host                                  [OK]
   |
   \-- Streaming (RTSP)
           -> getaddrinfo() on the IPv4 literal
           -> iOS synthesizes a NAT64 IPv6
           -> synthesized address is outside the VPN routes
           -> leaves the tunnel, dies at the carrier gateway    [FAIL]

This is the part readers will not have seen before, and it is why the failure is so disorienting. The two connections do not disagree about the address. They were both given the same address. They disagree about what resolving it means, and only one of them resolves it at a layer where the carrier can rewrite it.

Reading a synthesized address by eye

Once you have a capture, you can confirm all of this without any tooling beyond arithmetic, because NAT64 does not hide the IPv4. It embeds it. The synthesized address is the carrier's prefix followed by the original IPv4 written straight into the low 32 bits, one byte per octet, in hex.

Take the well-known NAT64 prefix 64:ff9b::/96 and a documentation address 192.0.2.33. The synthesized form is 64:ff9b::c000:221, and those trailing hex digits are the address itself: c0 00 02 21 is 192 0 2 33. Any address you see that ends in the hex of a familiar IPv4 of yours, sitting inside a /96 prefix that is not one you configured, is a synthesized address. In my capture the streaming SYN went to a prefix I had never set, and its low four bytes were my host's VPN IPv4 in hex. The client never dialed the address I gave it.

Here is the masked shape of what the capture showed, host address written as 100.a.b.c, carrier prefix as <nat64>, phone as <phone>. Same capture, same host, same few seconds:

# Streaming (RTSP): synthesized IPv6, retransmits into silence, then ETIMEDOUT
IP6 <phone> > <nat64>::<hex(100.a.b.c)>.48010: Flags [S]      # SYN
IP6 <phone> > <nat64>::<hex(100.a.b.c)>.48010: Flags [S]      # +1s, same seq
IP6 <phone> > <nat64>::<hex(100.a.b.c)>.48010: Flags [S]      # +2s, same seq
... 7 retransmits, no reply ...

# HTTPS to the SAME host, same capture: plain IPv4, inside the tunnel, works
IP  <phone> > 100.a.b.c.47984: Flags [S]  ->  SYN-ACK, session proceeds

I posted the full masked capture and analysis to the upstream issue this failure belongs to, moonlight-ios #669, where it had sat open since 2025 with plenty of me too and no root cause. The addresses there are masked the same way, structure kept verbatim, values replaced. Never publish the raw capture. Thirty seconds of a phone's traffic is thirty seconds of everything else the phone was doing.

The capture recipe

You cannot run a normal packet capture on an iPhone, but you can mirror its traffic to a Mac over USB with a Remote Virtual Interface, which ships with Xcode. Two commands:

rvictl -s <device-UDID>          # creates rvi0, mirroring the phone
sudo tcpdump -n -i rvi0          # capture; -n so it does not rewrite what you are trying to read

One trap worth stating, because it cost me time. Capture full frames. If you truncate with a small snapshot length, the PKTAP encapsulation that carries these packets does not survive the truncation and your capture becomes unreadable. Let it capture whole packets and filter later.

And validate the instrument before you trust its silence. When I first saw no packets at all on the failing path, that was almost a conclusion, and it would have been the wrong one, because a capture on the wrong interface also shows no packets. The rule is the same one that runs through everything I write about verification: confirm the check can see a case it should see before you believe it about a case it should not. Capture a working connection first. If the tool shows you the packets you know are there, then its silence on the broken path means something.

A note on the error code, because it lies too

The streaming failure surfaced as error 60, ETIMEDOUT. It is tempting to read that as connected, but got no reply, and I did read it that way for a while, and it sent me looking in the wrong place. On Apple platforms ETIMEDOUT is produced by two entirely different paths: the receive timeout after a connection is established, and the connect timeout itself, because a TCP connect that never completes also ends in ETIMEDOUT. The same number, 60, means both never connected and connected and heard nothing. You cannot tell them apart from the code. Here it was the first one, and the code implied the second. It is a small, sharp example of a signal that is present, precise, and not diagnostic.

The fix, and the general escape

The direct fix is to hand the client an address it cannot synthesize. On an IPv6-only network the host's own IPv6 literal is passed through untouched, so pointing the client at the IPv6 address keeps the traffic inside the tunnel where it belongs. In this particular app that meant two things together: put the server in dual-stack mode so it actually listens on IPv6, and delete every existing IPv4 host entry on the client before adding the IPv6 one, because as long as a synthesizable IPv4 entry is still around the client will fall back to it and get synthesized again. I watched exactly that happen, an IPv6 entry added next to the old IPv4 one, capture still showing the synthesized destination.

The library-level fix, the one that belongs in the client rather than in your config, is more interesting because of what it gets right. The naive version is to notice the address is an IPv4 literal and force the resolver to keep it verbatim. That works for your VPN and quietly breaks every legitimate public IPv4 host that genuinely needs synthesizing to be reachable from an IPv6-only network. The correct version scopes the fallback: keep the raw IPv4 only when the literal is in a private or CGNAT range, because those addresses do not exist on the public internet and there was never any point synthesizing them. That is what moonlight-common-c shipped in July 2025, and it is a small lesson worth keeping: the fix that looks more thorough, the one that turns the feature off, is often the one that drops an entire class of users on the floor.

What was actually scarce

Nothing in this failure was misconfigured. The VPN was routing correctly. The server was listening correctly. The app was, in a sense, behaving correctly on both paths. Every log said either fine or timed out, and both were honest. The defect was a silent address substitution that happened in one of two code paths, at a layer none of the normal instruments were pointed at, and it left no error anywhere because from each layer's own point of view nothing had gone wrong.

The value here was never the fix. The fix existed, upstream, a year before I hit the wall, in a commit whose message describes this exact situation. What was scarce was recognition: seeing the fingerprint, distrusting the timeout, distrusting the error code, and getting an eye on the one address the software never printed. If you self-host anything behind a VPN and reach it from a phone on cellular, this failure is waiting for you, and now it has a shape you can name.

There is a second, stranger half to this story. That upstream fix has existed for over a year and still has not reached the users hitting this bug, because of a single unbumped pointer in a dependency graph. That one is about delivery rather than diagnosis, and I will take it up separately.

This is a Network Field Note. It shares its spine with the correctness pieces on this site: A Wrong Ruler Is Worse Than No Ruler on trusting the wrong signal, and Validation Is a Loop, Not an Assertion on proving your instrument can see before you believe its silence.

Originally published at harrisonsec.com.

Tool Calls Need Completion Ownership: Why Your Agent's 200 OK Lies

Harrison Guo — Thu, 13 Aug 2026 16:45:35 +0000

An agent is asked to send a payment confirmation email. It calls the send_email tool. The tool returns {"status": "queued", "message_id": "msg_abc123"}. The agent writes back to the user: Done. Confirmation email sent. Two hours later a support ticket arrives. The customer never got the email. The provider's queue had dropped the message during a regional failover, and no one on the path between the agent and the recipient's inbox ever held ownership of the claim this email reached the customer. Every layer received a token, forwarded it, and declared its own part finished.

The instinct in the incident review is to blame the model. It reported something false with total confidence, so the model must have hallucinated. It did not. The model read a success signal and faithfully relayed it. The signal was a lie before the model ever saw it, and it was a lie for a reason that has nothing to do with language models.

A tool call's 200 means the request was accepted, not that the effect happened. Put completion ownership in the wrapper: verify the side effect before the word success ever reaches the model. The model cannot infer completion from a status code, and it will report exactly the confidence the token implies.

This sits in the same first-principles line as validation is a loop, not an assertion and determinism where you can, judgement where you must. Validation-is-a-loop argues that in a generative system you have to check the agent's work rather than assume it. This piece is about a narrower and sneakier place the check goes missing: not the model's reasoning, but the tool boundary underneath it, where a status code gets mistaken for a fact. It is also the direct descendant of an old distributed-systems question I wrote about in RPC vs NATS, who owns completion. The agent stack rediscovered that question and, for the most part, got it wrong.

The 200 that means accepted, not done

Trace the token upward through a typical stack and you find three separate places where an honest but incomplete signal gets promoted into a false one.

The tool API layer. The tool returns 200 because the request was accepted, not because the effect was observed. Email providers return 200 on enqueue. Async job endpoints return 202 with a job id. Eventual-consistency writes return 200 the moment the write is durable on one node, before it has propagated anywhere a subsequent read would find it. In every one of these, 200 is a true statement about receipt and says nothing about outcome.

The agent framework layer. The framework sees a 2xx, marks the tool call as succeeded, and appends a success entry to the conversation. This is the layer that does the real damage, because it collapses two genuinely different states, request accepted and effect observed, into one word. The framework had the status code and threw away everything about what it meant.

The model layer. The model reads success in the tool result and generates Done. Email sent, with high confidence. The confidence is not a defect. It is the correct response to the token it was given. Ask a person to relay a message stamped SUCCESS and they will relay it as success too. The model is the last honest link in a chain that lied to it three steps earlier.

The customer, at the end of all this, gets a confident report that contradicts observable reality. And the postmortem points at the one component that behaved correctly given its inputs.

Where completion ownership went

The underlying problem is older than agents, older than HTTP. Any time an operation crosses a boundary, someone has to own the answer to did it actually happen.

RPC's classic wound is the caller who gets a network error partway through a call. Did the operation run or not? A network error is not a no. It is an unknown. The operation may have committed on the server and had its acknowledgement lost on the way back. Without an idempotency key to retry safely and a read-after-write check to confirm the effect, the caller genuinely cannot tell, and any recovery it attempts is a guess.

The way out, in every reliable system, is to name an owner. Either the caller polls until it sees the effect with its own eyes, or the receiver commits and only then reports back. As I argued in the RPC vs NATS piece, fire-and-forget messaging pushes ownership onto the caller, and request-response pushes it onto the receiver. Neither placement is wrong. What is always wrong is ambiguity about which layer holds it, because ambiguity means the answer is nobody, and nobody is exactly what produces the phantom confirmation email.

AI tool calls inherited this problem and, in the rush to wire models to real actions, mostly skipped the part where you decide who owns completion. The wrapper trusts the API's 200. The API trusts its dispatcher. The dispatcher trusts the worker. The worker logs success on enqueue. The token travels all the way up to the model and out to the user, and at no point did any layer commit to having observed the effect.

Declare the mode in the wrapper

The fix starts by refusing to let a status code stand in for an outcome. Every tool wrapper has to declare which of three modes it operates in, and that declaration lives in the wrapper, not in the model's head.

Mode	Returns when	Caller responsibility
`committed`	The effect has been observed	May report success directly
`accepted`	The request was enqueued, effect not yet observed	Must verify the effect before reporting success
`optimistic`	Best-effort send, no guarantee available	Must surface uncertainty to the user

This is a wrapper-level discipline for a concrete reason: the model cannot derive the mode from a 200. Two tools can return byte-identical success payloads while one has durably applied its effect and the other has merely queued it. The distinction exists only in knowledge the wrapper has and the status code does not carry. If the wrapper does not encode the mode, the information is gone by the time the model sees the result, and no amount of prompting recovers it. This is the technique boundary in miniature: the deterministic layer knows something the model cannot infer, so the deterministic layer has to state it.

The closed loop for accepted-mode tools

For anything in accepted mode, success is not a value the tool returns. It is a state a verifier confirms. The wrapper returns a handle, and a verification step stands between the tool and the model's next turn.

The wrapper returns {"status": "accepted", "polling_token": "..."}. It does not return success.
A verifier step polls the token until it reaches a terminal state, delivered or failed, reading the actual effect rather than re-reading the queue that accepted it.
Only the terminal state is surfaced to the model.
If polling exhausts its budget, the wrapper surfaces uncertain, the operation may or may not have completed, and explicitly not success.

That verifier is middleware sitting between the tool and the next model turn, the same shape as the loop in validation is a loop, not an assertion: the agent acts, the system verifies, scores the result, and routes on it. And the verification has to observe the effect itself, not a proxy for it. Re-reading the queue that already said accepted will happily confirm accepted forever. That is a wrong ruler: a check that looks like verification but measures the wrong thing, which is worse than no check because it manufactures false confidence. A read-after-write worthy of the name reads the recipient's mailbox state, the inserted row, the written file, not the acknowledgement that a request to produce them was received.

Agent action
   |
   v
Tool wrapper
   |
   |-- committed --> Effect observed --------------> Model context: "success"
   |
   |-- accepted --> Verifier (poll the real effect)
                        |-- terminal: delivered ---> Model context: "success"
                        |-- terminal: failed ------> Model context: "failed"
                        |-- budget exhausted ------> Model context: "uncertain"
                        \-- not yet terminal ------> keep polling (loop back)

Every path in that graph ends at an honest token. The only way the model reports success is if some layer below it actually watched the effect land.

Five questions for every tool in the box

This is ordinary reliability hygiene, applied one tool at a time. For each tool your agent can call, you should be able to answer yes to all five:

Does the return value distinguish queued from applied?
If the tool is asynchronous, is there a polling token or a callback to confirm the terminal state?
Does the wrapper enforce verification before success can reach the model?
On verification timeout, does the wrapper surface uncertain, rather than defaulting to success or failure?
Is there a retry budget owned in one place, so a burst of retries cannot compound the ambiguity?

That last question is where this connects to cost as well as correctness. Uncoordinated retries stacked at three layers are exactly the retry storm that inflates an AI bill, and they are also a completion-ownership failure: three layers each hoping the operation happened, none of them owning the answer. One retry budget in one place fixes both faces of the same bug.

Any tool that fails one of these five carries completion-ownership debt, and that debt is paid in user-visible incorrectness, the agent asserting things that did not happen.

It is 2PC versus eventual consistency, wearing a tool schema

For anyone who has built distributed systems, the whole thing has a familiar shape. A two-phase commit says I hold ownership and will not report done until the effect is durable. Eventual consistency says I forward my part and trust the next layer to converge. Both are legitimate designs. The trouble is that most agent tool stacks are built like eventual-consistency systems but report like two-phase commits: they emit a crisp, immediate success for an effect that is still only propagating, or still only queued, or already quietly dropped.

The fix is not to force every tool into two-phase commit. Plenty of effects are genuinely eventual, and that is fine. The fix is honesty in the report. If the system is eventual, the token that reaches the model has to say so, accepted and then a verified terminal state, never a premature success. If you have shipped idempotency keys and read-after-write checks before, you already have every piece of this. The polling token is a request UUID with a different label, and the verifier is a read-after-write check you already know how to write.

The bill for skipping it does not arrive as a stack trace. It arrives as a support ticket, hours later, from a customer the agent told with total confidence that something was done. Read the completion one layer earlier, in the wrapper, where you can still tell accepted from observed, and the model stops lying, because for the first time nothing below it is lying to the model.

This piece sits in the Generative Systems, First Principles line. Its correctness siblings: Validation Is a Loop, Not an Assertion, A Wrong Ruler Is Worse Than No Ruler, and Determinism Where You Can, Judgement Where You Must. The distributed-systems ancestor: RPC vs NATS, Who Owns Completion. The cost companion: Your AI Bill Is a Distributed Systems Problem.

Originally published at harrisonsec.com.

Your AI Bill Is a Distributed Systems Problem, Not a Model-Pricing Problem

Harrison Guo — Tue, 04 Aug 2026 16:01:31 +0000

A team I was helping watched their model bill jump to several times its usual size in a single month. The token meter had not predicted it. The first question in the room was the one almost everyone asks: did the provider raise prices, or should we move to a cheaper model?

Both were the wrong question, and they were wrong in an instructive way. The same engineers, if their AWS egress bill had tripled, would never have opened with did AWS raise prices. They would have asked what is calling more, pulled up the request graph, and started looking for the amplification. The instinct is correct and automatic when the unit is HTTP calls. It deserts people the moment the unit becomes tokens.

That gap is the whole subject of this piece.

A surprising AI bill is a symptom. The disease is almost always a distributed systems bug, a retry storm, a hidden fanout, a cache that stopped hitting, a conversation that grows without bound, that you would recognize instantly if it were denominated in network calls instead of tokens. Debug the call graph, not the model price.

This sits in the same first-principles line as determinism where you can, judgement where you must and validation is a loop, not an assertion. Where those are about correctness, this one is about cost, and the move is the same: take a problem that looks new because it wears an AI costume, and recognize the old shape underneath. On the architecture side it pairs with Observability and Billing for AI API Calls, which is about how to build the telemetry. This piece is about how to read it when the number jumps.

The diagnostic flip

When a backend engineer's egress bill spikes, the reflex is to ask what changed in the call graph. Nobody's first move is to email the vendor about pricing. When an AI engineer's model bill spikes, the reflex too often inverts: the first suspects are the price sheet and the model choice, and the call graph is the last thing anyone looks at.

These are the same question wearing different clothes. What is driving the spend, and is the driver behaving the way it should? We get it right for HTTP because we have years of muscle memory in that unit. We get it wrong for tokens because the unit is unfamiliar and, crucially, because the bill arrives pre-aggregated into a single dollar figure that hides the graph that produced it. The dollar figure is the least useful view of the problem, and it is usually the only view people start from.

So before anything else, refuse the pricing question. Assume the spend is a downstream-service spend that misbehaved, and go find the misbehavior. It will almost always be one of five shapes.

Five failure modes, denominated in tokens

1. Retry storms. A transient provider error triggers a retry at the SDK layer, and again at the agent layer, and again at the workflow orchestrator, because all three were written to be resilient and none of them knew about the others. One user request becomes several billed calls. This is the exact retry storm you have debugged on a degraded downstream HTTP service, where nested retry policies multiply instead of coordinate. The fix is the same: one retry budget, owned in one place, not three hopeful ones stacked.

2. Fanout amplification. An agent spawns several parallel tool calls, and each tool, somewhere inside its implementation, makes its own model call to parse or summarize its result, and each of those results feeds back into the main conversation. One user request quietly becomes many model calls, and most of them are hidden inside tool code where no one thinks to look. This is request fanout amplification, the same reason a single page load can trigger forty backend calls when each component fetches for itself.

3. Cache misses where caches should hit. The provider supports caching a stable prompt prefix, and you are not using it, so the same system prompt and the same long preamble get retokenized and rebilled on every single call. This is a CDN accidentally configured to pass every request through to origin. The content never changed, and you paid to recompute it anyway, every time.

4. Unbounded conversation growth. Conversation history is appended turn after turn with no compaction, so the tokens billed per turn grow with the length of the session, and the total cost of a session grows quadratically with its length. This is an unbounded buffer, the same class of leak that treating a channel like message passing produces, except the buffer is a transcript and the leak shows up on an invoice instead of in a heap profile.

5. Wrong model for the job. A frontier model is doing work a smaller model handles perfectly well, on every call, by default. This is running your ad-hoc analytical queries against the transactional cluster because it was the connection string you had, instead of against the warehouse built for them. It is also, in the language of the technique boundary, using the most expensive technique where a cheaper one was sufficient, which is the cost face of the same mistake that piece describes for correctness.

Four of these five are amplification bugs: something is producing more calls, or more tokens per call, than the work requires. The fifth is a tiering mistake. None of them is a price change, and none of them is fixed by switching models, which is exactly why switching models first tends to disappoint.

The forensics, in order

The diagnosis runs in the same order every time, and each step reads a distributed systems metric off the call graph, not a number off the token meter.

1. Calls per user request
   near your workflow's theoretical minimum? -> healthy
   several times the minimum?                -> retry storm or hidden fanout (modes 1, 2)

2. Tokens-per-call distribution
   stable week over week?                    -> healthy
   the tail is climbing?                     -> unbounded conversation or uncached prefix (modes 3, 4)

3. Model mix over time
   right tier per task type?                 -> healthy
   a routing change shifted traffic up?      -> wrong model / tiering (mode 5)

4. Provider reject rate
   negligible?                               -> healthy
   climbing?                                 -> the trigger that sets off mode 1

The order matters. Provider rejects sit at the bottom of the list but at the root of the causal chain, because a rise in transient rejects is what ignites a retry storm at the top of it. Read the graph top to bottom to localize the symptom, then bottom to top to find the cause.

The point of laying it out this way is that none of it is novel. Your observability stack already knows how to compute calls per request, tail distributions, traffic mix, and error rates. You compute them for every other downstream service you depend on. The only new thing is pointing that same machinery at the model provider.

  User request
       |
       v
     Agent  --fanout xN-->  Tool calls (each hides a model call)
       |                         |
       | retries xR              | retries xR
       v                         v
              Provider  (cache layer)
                     |
                     v
                 The bill

Every multiplier on an edge in that graph is a place the bill can balloon while the price per token never moves. The invoice only shows you the node at the far right. The leak is always on an edge.

Billing is not cost forensics

Here is the line that decides whether you can do any of the above. If your AI observability emits tokens multiplied by price for each call and stops there, you have billing. You know what you paid. You cannot see why, because the why lives in the edges of the call graph, and a per-call token count has thrown the graph away.

Cost forensics needs the graph kept intact: calls attributed back to the originating user request, fanout factors per tool, cache hit rates per prompt prefix, retry distributions per layer, model mix over time. That is the same telemetry shape you already maintain for HTTP dependencies, and building it deliberately is the subject of the T-shaped architecture piece. The relevant point here is diagnostic: if that data does not exist, no one, however senior, can tell you where your bill is leaking, because the evidence was discarded at collection time. Add the graph, and an engineer who has never touched an LLM but is good at HTTP cost forensics can find the leak, because a token is just a byte with cognitive content, and a model provider is just a downstream service that bills by the payload.

When a model switch is actually the answer

Not never. After you have ruled out retries, fanout, cache misses, and conversation growth, mode five is a real and common diagnosis, and moving a class of calls to a smaller model is the right fix. The discipline is only about order. Switch models before ruling out the amplification modes and you can land on a cheaper model that still has a retry storm sitting on top of it, which presents as the smaller model being mysteriously expensive too, and now you have two confusing bills instead of one. Rule out the graph problems first. Then a model switch is a measurement-backed decision rather than a hopeful guess, and it tends to hold.

The old hygiene, new invoice

If your team has someone with good instincts for HTTP retry budgets, request fanout, cache hit ratios, and payload-size distributions, that person can already debug your model bill. Everything transfers. Retry budgets are retry budgets. Fanout is fanout. An uncached prefix is an uncached prefix. The provider is one more downstream service that responds to the same hygiene you already apply to object storage, managed databases, and third-party APIs.

The bill is the last place the bug appears and the most expensive place to read it, because by the time it shows up there it has already been paid. Read it one step earlier, in the call graph, where every one of these five modes is visible as a multiplier on an edge, and where you have known how to fix each of them for years. Your AI bill is not a new kind of problem. It is an old kind of problem with an unfamiliar unit, and the moment you convert the unit back, you already know what to do.

This piece sits in the Generative Systems, First Principles line. The correctness companions: Determinism Where You Can, Judgement Where You Must and Validation Is a Loop, Not an Assertion. The architecture companion: Observability and Billing for AI API Calls.

Validation Is a Loop, Not an Assertion: Why Your AI Agent Reports Success It Never Achieved

Harrison Guo — Tue, 28 Jul 2026 16:27:16 +0000

An agent I was reviewing was asked to rename a function across a repository. It ran for a while, made its edits, and returned a clean summary: Done. Renamed parseUser to parseAccount everywhere. The summary was confident, well formatted, and wrong. Six of eight call sites were updated. Two were not, one buried in a string that built the name dynamically and one in a test helper the search had skipped.

The agent did not lie. It returned what its model believed was true at the moment it returned. The problem is that nothing sat between I believe I succeeded and I report success. There was no step that walked back out into the world, looked at what had actually changed, and compared it to what the task asked for. The agent's confidence and the agent's correctness were two completely unrelated numbers, and only one of them made it into the summary.

This is the single most common way I see production agents fail, and it is almost never a model problem. It is a validation problem, and specifically it is validation built in the wrong shape.

In a deterministic system, validation is an assertion: check once, pass or fail. In a system driven by non-deterministic model output, validation has to be a loop: observe the real outcome, score how close it is, and decide whether to accept, retry, route, or stop. The most common production agent failure is treating the second case like the first.

This piece sits alongside two companions. Determinism where you can, judgement where you must is about deciding which technique owns each part of a system. A wrong ruler is worse than no ruler is about verifying the checks you trust before you let them hold authority. This piece is about the shape of the check itself, once you have decided the output is non-deterministic enough to need one.

What an assertion assumes

assert(result == expected) is one of the most useful lines in deterministic software, and it is useful because of a set of assumptions that are almost always true in that world. The operation you just ran is deterministic, so it either did the thing or it did not. The check is far cheaper than the operation, so running it once at the end costs nothing worth counting. And the answer is binary, because in a deterministic system there is no meaningful state between success and failure. You parsed the integer or you threw. You wrote the row or you got an error.

Every one of those assumptions breaks the moment the operation is a model call.

The operation is no longer deterministic, so it did the thing is now a probability, not a fact. The check is no longer trivially cheaper than the operation, because verifying whether a model actually accomplished a fuzzy goal can be almost as hard as the goal. And the answer is no longer binary, because the whole texture of model failure is partial. The rename that got six of eight. The summary that captured four of the five key points. The extraction that found every date except the one written in words. Partial success is not an edge case in these systems. It is the main case, and an assertion has no vocabulary for it. 6 of 8 and 8 of 8 collapse to the same false, which throws away exactly the information you needed.

The shape of the loop

A loop keeps the information an assertion discards. It does not ask did it pass. It asks how close did it get, and what should happen because of that. Concretely it is four moves and a stopping rule.

Validate by observation. After the action, look at the real, observable outcome, not the model's report of it. For the rename, that means searching the codebase for the old and new names and seeing which call sites actually changed. The validator's input is the world, not the transcript.
Score with partial credit. Turn the observation into a number or a structured verdict that can express most of the way there. Six of eight is 0.75, and that number carries the two misses that a boolean would have erased.
Decide against thresholds. Is the score high enough to call the task done? Low enough to abandon and report failure honestly? Or in the band between, where more work might close the gap?
Route the in-between case. When the score is neither clearly done nor clearly hopeless, choose the next action deliberately. Retry only the two missed call sites. Ask the user whether the dynamic-string case should even be touched. Escalate to a stronger model. The router is where the loop earns its keep, because it is the part an assertion cannot do at all.
Iterate to a terminal condition. Repeat until one of three things is true: the score clears the accept threshold, a retry budget is exhausted, or a human steps in. The budget matters as much as the score. A loop with no budget does not converge, it hangs.

Drawn out, the control flow is small but it is a cycle, and the cycle is the point.

        +---------------- route: retry / clarify / escalate ----------------+
        |                                                                   |
        v                                                                   |
  Agent action --> Validate --> Score --> Decide --+-- clears accept  --> Report success
                  (observe)   (partial)            |
                                                   +-- below abandon  --> Halt, surface partial progress
                                                   |
                                                   +-- in between ------> Route ----------------------+

Notice where control lives. The model reasons inside this loop and it can even do the scoring when the judgement is subjective. But it does not own the loop. The decision of what counts as done, and what happens when the task is not done, stays with the orchestrator. This is the same boundary the technique piece draws: the model may reason under the control, it may not direct the control itself. An agent that gets to declare its own task finished and exit on its own authority has no loop. It has an assertion it wrote for itself and then graded.

The scoring function is the hard half

If you have designed retry-with-backoff for a flaky network call, you have already built most of this loop. Observe, decide, retry with a budget, give up gracefully. That machinery ports over almost unchanged. The part that does not port, the part that makes this genuinely a production AI problem rather than a distributed-systems problem you already solved, is the scoring function.

For the rename, scoring is easy, because the outcome is observable by deterministic means. You can enumerate call sites and count. Whenever you can score by observation like that, do, because a count does not hallucinate. The hard cases are the ones where the goal is subjective: is this summary faithful, is this explanation coherent, does this answer satisfy the brief. There the score has to come from judgement, often a model judging another model's output, and now you have a new problem stacked on the first one.

That scorer is itself a check, and a check that can be wrong. If you let a broken scorer drive the loop, you do not merely fail to catch errors. You manufacture confident wrong verdicts and feed them straight back into the agent's next action. A loop built on a bad ruler is worse than no loop, for exactly the reasons the companion piece lays out: a wrong check with authority does not leave you uncertain, it leaves you confidently wrong, and here it does so in a cycle that compounds. So the discipline from that piece applies in full to the scorer: verify it at least as hard as the claims it will act on, score by comparison against anchors rather than emitting a bare number, and know the ceiling of human agreement before you trust a model to beat it.

Two places the loop lives

In a real agent the loop shows up at two scales, and mature systems run both.

The first is per-step, as middleware. A validator runs after every tool call, before the model sees the result. What gets fed back into the next prompt is not the raw API response. It is the validated, scored, sometimes rewritten version of it. This is what stops a single bad step from silently poisoning everything downstream, because the model never gets to reason on top of an unchecked result. The loop here is tight and cheap and runs constantly.

The second is end-to-end, as a harness. At the boundary of the whole task, you check the full resulting state against the intended outcome and treat the score as a circuit breaker. If the end-to-end score is below threshold, the agent does not report success. It halts and surfaces the partial progress honestly: here is what changed, here is what did not, here is where it stopped. This is the layer that would have caught the rename. Per-step middleware might have passed each edit individually while the aggregate still missed two sites, and only a check against the whole goal sees that gap.

You want both because they catch different failures. Middleware catches the step that went wrong. The harness catches the task that came out wrong even though every step looked fine. Skip the middleware and errors compound before you notice. Skip the harness and you are back to an agent that reports success it never achieved, which is where we started.

The distributed-systems reframe

If all of this feels familiar, it should. It is eventual consistency wearing different clothes.

An assertion-style check on a freshly written row in a distributed store fails not because the write was wrong but because it has not propagated yet. The correct pattern was never a single assertion. It was a loop: poll until the read converges with the write, or until a timeout says stop waiting. Model output validation is the same pattern with cognitive uncertainty standing in for replication lag. The output is not yet known-good, so you observe, you score, you decide whether to wait for a better one or accept what you have, and you bound the whole thing with a budget so it terminates.

Which is why teams from a distributed-systems background tend to get this right faster once it is named. You already know that in any system where the truth arrives late and imperfectly, a one-shot check is the wrong instrument. You built loops for that years ago. An LLM is just another source where the truth arrives late and imperfectly. The retry-with-backoff instinct is half the answer. The scoring function is the half that is new.

Where an assertion is still right

The loop is not a universal upgrade, and reaching for it everywhere is its own mistake. The whole reason the loop exists is that the output is non-deterministic. Where the output is deterministic, an assertion is not just adequate, it is correct, and wrapping a deterministic step in a scoring loop adds cost and latency to buy nothing.

The schema either validates or it does not. The parser either succeeds or it throws. The row count either matches or it does not. Those are assertions, and they should stay assertions, sitting at the boundaries of your system doing exactly the one-shot job they are good at. The technique boundary is the tool for deciding which is which: put as much of the system as you can onto deterministic ground where a plain assertion holds, and reserve the loop for the parts that are genuinely non-deterministic and genuinely need judgement. An agent that scores and retries its way through a JSON schema check is making the opposite mistake to the one this piece is about, and it is just as wasteful.

The one honest question

The next time an agent hands you a confident done, ask it the question the loop asks and the assertion cannot: not did it pass, but how do you know, and what did you observe to know it. If the answer is that the model believed it, you have an assertion the model wrote for itself. If the answer is a real observation of the world, scored against the goal, with a decision that followed from the score, you have a loop. Only one of those two agents can be trusted to tell you when it failed, and it is the one that was built to look.

This piece extends two companions. The boundary it assumes: Determinism Where You Can, Judgement Where You Must. The check-verification discipline its scorer depends on: A Wrong Ruler Is Worse Than No Ruler.