DEV Community: Harrison Guo

Agent Retrieval Above the Crossover: A First-Principles Read of CodeGraph

Harrison Guo — Mon, 08 Jun 2026 19:49:41 +0000

The prior post in this series, Agent Retrieval Is a Cost Curve Problem, argued that a viable LLM-symbol-graph would need to satisfy six specific conditions — and that no existing tool had hit all six. The post went live on 2026-05-25; seven days earlier, CodeGraph had hit GitHub trending with exactly those six properties satisfied.

That's the easy version of the update: framework predicted it, someone shipped it, here's the existence proof. The companion piece (I Tested CodeGraph on Hono. The Tool-Call Savings Reproduce — the Cost Savings Don't.) handles the empirical half — 40 verified-connected runs, a decision matrix, the install-or-not call. Short version of that post: the tool-call savings reproduce on an independent repo (−55%), the cost savings from the vendor benchmark don't (+7% at Hono's size). Fewer steps, not fewer dollars, until your repo is big enough.

This post is the harder version of the update.

The interesting question isn't whether CodeGraph works. The interesting question is why are its specific architectural choices right, and where does the abstraction inevitably leak? Answering it gives you the lens for evaluating the next CodeGraph-class tool that ships — and there will be many — without redoing the benchmark each time.

To answer it concretely rather than abstractly, I read CodeGraph against its own artifact: the SQLite database it writes to .codegraph/codegraph.db. Every structural claim below is checked against the index it actually built for Hono (CodeGraph v0.9.7: 362 files, 4,128 nodes, 8,225 edges, a 7.4 MB database). The schema turns out to be the clearest statement of the architecture the tool's README never makes.

tl;dr — CodeGraph's architecture is right for three reasons that aren't obvious from the feature list, and all three are visible in its SQLite schema. (1) The AST extraction boundary: tree-sitter takes what syntax tells you (4,128 nodes across 13 kinds, 8,225 edges across 7 kinds) and leaves the rest to the LLM. The boundary is literal — references syntax can't resolve go into an unresolved_refs table instead of becoming fake edges. (2) SQLite + FTS5, not a vector DB: the index is plain relational tables plus a full-text table over symbol names. Zero embedding columns. The queries are exact lookups that B-tree indexes answer in log time; vector search would be solving a harder problem the workload never asks. This is the prior post's cost curve, recursed onto the index tool itself. (3) The abstraction leaks where syntax diverges from runtime semantics — macros, metaprogramming, codegen, JIT binding. CodeGraph tags its few guessed edges with a heuristic provenance flag (7 of 8,225 on Hono), which is honest; but what tree-sitter can't see at all gets no edge and no flag. Knowing that boundary is what separates a tool you trust from one you cargo-cult.

Why this is a first-principles question, not a tool review

Most coverage of CodeGraph reads like "19k stars in a week, here's the install script." That's news; it isn't analysis. The same coverage will get written for every CodeGraph-class tool that ships in the next 18 months, because the pattern — tree-sitter + local index + MCP server + an instruction snippet that routes the agent to it — is now demonstrated and the ingredients are well known.

The durable question isn't "is CodeGraph good?" It's "what makes this class of tool architecturally correct, and how do I evaluate the next one?" That's what a first-principles read produces. The benchmark in the companion post is one data point; this post is the lens for reading all future data points in the same space.

If you're deciding on CodeGraph specifically, read the companion. If you're thinking about LLM retrieval as a discipline — or about to bet on, or build, a similar tool — read this.

Recap: the six conditions, in 30 seconds

The prior post argued any viable LLM-symbol-graph needed:

No-compile parsing — cold start in seconds, not minutes
Language portability — one binary for many languages, not one server per stack
LLM-shaped API — flat, recordy output the model can digest, not nested LSP hierarchies
Broad enough coverage — code-as-structure plus a text-search fallback for everything else
Live update without reindex — file-watcher-driven, no manual rebuild
Zero-config install — single binary, configures the agent automatically

CodeGraph hits all six (the field-by-field mapping is near the end of this post). Taking the mapping as established, the interesting move is to ask: of the design choices CodeGraph made to hit those six, which were forced and which could have gone the other way? The forced ones are good engineering. The ones that weren't forced — where CodeGraph picked something specific over a live alternative — are where the architecture is making a claim, and where the first-principles content lives.

Three of those choices repay a deep read. The other three (file-watcher update, single-binary distribution, instruction-snippet routing) are well-understood in their own fields — OS notifications, package distribution, prompt engineering — and amount to "do the obvious thing well." The three that don't are the three this post takes apart, each against the actual index.

Section 1 — The AST extraction boundary: an information-theoretic case

CodeGraph parses source with tree-sitter and extracts a specific subset of the syntax into its graph. You don't have to take the README's word for what that subset is — it's enumerable straight out of the nodes and edges tables. On Hono, the 4,128 nodes break down like this:

Node kind	Count	Node kind	Count
import	1,033	method	240
route	873	interface	187
function	569	property	169
file	362	class	50
type_alias	358	enum_member	24
constant	247	variable / enum	16

And the 8,225 edges, which are the actually interesting part:

Edge kind	Count	What it encodes
contains	2,874	structural nesting (file → class → method)
calls	2,230	the call graph
references	1,955	symbol used here, defined there
imports	1,033	module dependency edges
instantiates	124	`new X()` sites
extends	7	class/interface inheritance
implements	2	interface implementation

Now look at what is not there. No "type" nodes. No generic-instantiation edges. No data-flow edges. No "this dynamic dispatch resolves to that concrete method" edges. CodeGraph extracts calls, references, extends, implements — relationships that are locally apparent in the syntax — and stops. The first-order reading of this is "because tree-sitter doesn't resolve types." True, but circular. The deeper reading is why this division of labor is correct for an LLM consumer.

The information-theoretic case

A type-checker (or full LSP) does work the LLM cannot easily redo: resolving obj.method() to the actual method given the static type of obj, propagating types through generics, walking an inheritance chain to the method actually invoked. That requires the full compilation context — every transitive import, every type definition, every generic instantiation. The cost is high (a build environment, slow cold start, breaks when the build breaks) and the benefit is narrow: precise semantic resolution that's genuinely hard to reconstruct from local context.

A syntactic extractor does different work. It makes the structure of the source queryable, but only the structure that's locally apparent: "function dispatch defined at hono-base.ts:406, calls match here, imported from router." No types, no generics, no runtime binding — but no compilation either.

The information-theoretic question is: given an LLM that's good at semantic reasoning but bad at structural enumeration, what's the right split between what the index provides and what the LLM provides?

CodeGraph's answer: hand the LLM the structural skeleton — what calls what, what's defined where, what imports what — because enumerating that across thousands of files is exactly the part the LLM is bad at and would burn dozens of tool calls trying to do by hand. Leave the semantic resolution — what does this call actually invoke at runtime under dynamic dispatch? — to the LLM, because the LLM is reasonable at that once the relevant code is in its context, and baking a type resolver into the index would multiply the build cost for a recovery the LLM mostly doesn't need.

The clean way to see this boundary is the contains + calls + references edges (7,059 of the 8,225) versus the things that aren't edges at all. When the companion benchmark's Q1 asked how a GET /users/:id request reaches its handler, what CodeGraph gave Claude Code was the call chain — fetch → dispatch → match — as graph edges. What it did not give, and didn't try to, was which concrete match implementation runs given Hono's SmartRouter picking RegExpRouter at runtime. The graph located the players; the LLM read the three files and resolved the dispatch. That's the split working as designed: enumeration from the index, resolution from the model.

The boundary is a literal table

Here's the detail that turns this from an argument into an observation. When tree-sitter sees a reference it cannot statically resolve to a definition, CodeGraph does not invent an edge. It writes a row to a separate unresolved_refs table — name, location, the node it came from, no target. The schema has a first-class place for "I saw a use here, I could not prove what it binds to."

On Hono, unresolved_refs has zero rows — and, as it turns out, so did every other repo I indexed to check it (Section 3 has that result, and it's not the one I expected). The empty table isn't the interesting part; the table existing is the architecture stating its own boundary. A tool that faked those edges — guessed a target to make the graph look complete — would be lying to the LLM in exactly the way that produces confident wrong answers. CodeGraph's choice to record the unresolved reference as unresolved is the same discipline a good cache has when it marks an entry stale instead of serving it: the honest move is to represent "don't know," not to paper over it.

Why this matters beyond CodeGraph

This boundary — syntactic graph for the index, semantic reasoning for the LLM — is the line the next generation of LLM-coding tools will either hold or violate. The violations are predictable:

Too far toward semantics in the index: a tool that tries to be a full LSP-plus for the LLM. High build cost, slow cold start, fragile on broken builds, marginal benefit because the LLM can do that resolution from local context anyway.
Too far toward raw text in the index: a tool that's just "grep with nicer indexing" — fast and broad, but it doesn't hand the LLM the structural skeleton it actually needs. That's the position grep+loop already occupies; an index there adds little.

CodeGraph sits in the middle, and that position is right for current LLM capability. As models get better at semantic resolution the line will move one way; as tool-loop iteration gets cheaper it will move the other. But the principle — that there's an information-theoretic boundary worth picking, and that picking it requires modeling the LLM's real strengths and weaknesses — is the durable take. The right way to evaluate any new LLM-retrieval tool starts here: what does it choose to extract, what does it leave for the LLM, and is that split calibrated for what an LLM is actually good at?

Section 2 — SQLite + FTS5 vs vector DB: the cost curve, recursed

CodeGraph stores its symbol graph in a local SQLite database. Not Chroma. Not Pinecone. Not Weaviate. Not Qdrant. The full table list from Hono's index:

nodes              edges              files
unresolved_refs    nodes_fts          schema_versions
project_metadata   (+ FTS5 shadow tables: nodes_fts_data/idx/docsize/config)

nodes and edges are plain relational tables. nodes_fts is an FTS5 virtual table. Searching the whole schema for an embedding column, a vector type, a float array — anything ANN-shaped — returns nothing. The only BLOB columns are FTS5's own internal segment storage (nodes_fts_data), not vectors. There are no embeddings in CodeGraph. That's not an omission; it's the architecture, and it's the same call the prior post made one level down.

The cost-curve frame, recursed

The prior post argued vector RAG over a codebase pays a build cost (chunk + embed every file), a maintain cost (re-embed on change, reconcile cross-chunk references), and a low per-query cost (ANN search + rerank) — and that for most repos this loses to grep+loop's (zero build, zero maintain, per-query round-trips).

Apply that exact frame to CodeGraph's own storage. If CodeGraph used a vector DB for its symbols, it would pay: embed every symbol's signature and body on index; re-embed on every file save (the file-watcher would have to fire embedding calls); ANN search per query. That's the same curve the prior post argued against — and CodeGraph's workload doesn't justify it, because the queries it serves are exact lookups, not similarity searches. The schema proves the queries are exact by the indexes it builds for them:

"Find symbol getUserById" → idx_nodes_name, and idx_nodes_lower_name for case-insensitive matches. A B-tree probe, microseconds. FTS5 (nodes_fts over name, qualified_name, docstring, signature) handles the fuzzier "name contains" variants. No similarity math.
"Who calls Context.set?" → idx_edges_target_kind (a reverse-edge index on (target, kind)). Reverse adjacency lookup, deterministic.
"What does dispatch call?" → idx_edges_source_kind (the forward-edge index). Forward adjacency, deterministic.
"Trace fetch → db_query" → repeated forward-edge hops over those same indexed edges. Graph traversal on stored adjacency, no vectors anywhere in the loop.

Those forward and reverse edge indexes are the whole ballgame. Callers and callees — the queries a code-intelligence tool exists to answer — are a single indexed adjacency lookup in each direction. Vector search cannot do this better; it can only do it fuzzier and more expensively, because "who calls this function" has an exact answer that an approximate-nearest-neighbor index would blur.

The only queries where vector search genuinely helps are semantic ones with no symbol to anchor on — "show me the code that does authentication." CodeGraph doesn't serve those. The LLM does, by issuing a sequence of exact structural queries and reasoning across the results. The division is the same one from Section 1: the index answers the exact-lookup questions deterministically; the LLM answers the fuzzy-intent questions by orchestrating exact lookups. Neither needs an embedding.

The recursion as a design principle

What's elegant — and worth surfacing for its own sake — is that CodeGraph's storage choice is consistent with the retrieval philosophy from the prior post, one level up. Both arguments are the same sentence: exact-lookup workloads should use exact-lookup tools; approximation overhead is paid only where approximation pays back.

If CodeGraph had reached for Chroma over FTS5, it would have violated its own retrieval philosophy — paying embedding and ANN cost to answer questions that have exact answers. That it didn't, that the designer recognized the symbol-graph workload is exact-lookup-shaped and picked the cheapest exact-lookup storage available, is what makes the architecture coherent across layers rather than just locally clever.

The next tool in this class will face the same fork, and most will reach for a vector DB by default, because "AI tooling = vector store" is the reflex. CodeGraph's choice is the corrective: ask what your workload needs, not what the category's fashion suggests. That's the cost-curve frame functioning as a meta-design tool — every time you add a layer to an LLM stack, ask which side of the curve the new layer's workload sits on, and pick storage and algorithm from the answer, not the trend.

Section 3 — Where CodeGraph's abstraction leaks

Every index lies a little. The question is where it lies and whether you can tell when it does.

CodeGraph's graph is built from syntactic extraction, so anywhere the runtime semantics diverge from the syntactic structure, the graph is incomplete in a way that's hard to detect from the index alone. The leak isn't a bug; it's the abstraction working as designed, at a layer that structurally cannot see certain phenomena. There's a tell for it in the schema, and there's a part the schema can't tell you about — and the difference between those two is the whole point.

The honest part: the provenance column

CodeGraph stamps every edge with a provenance value. On Hono, 8,218 of the 8,225 edges have empty provenance — meaning direct from the syntax tree — and exactly 7 carry the value heuristic. Those seven are edges CodeGraph's framework adapters inferred from a recognized pattern rather than read off the AST: route registrations, framework binding conventions, the handful of cases where a tool that "supports Hono / Flask / Spring" pattern-matches a known idiom and synthesizes an edge the raw syntax doesn't spell out.

That heuristic tag is the architecture being honest. It is, in the vocabulary of the memory post in this series, an arrow: every edge points back to how it was derived, and the seven guessed edges are flagged as guesses. A consumer that cared could treat heuristic edges with less trust than syntactic ones. That's good cache hygiene — the index records the confidence of its own entries instead of presenting all of them as equally certain.

The part the schema can't tell you about

Here's the catch, and it's the one that matters: the provenance column only flags edges that exist. The dangerous leak isn't a guessed edge that's marked as guessed. It's the edge that should exist and isn't there at all — because the relationship lives in a layer tree-sitter cannot see, so there's nothing to extract, nothing to tag, and nothing to warn you. The four big zones where this happens:

Macro-heavy code. In Rust, vec![1, 2, 3] expands at compile time into a call sequence the AST never contains; the graph shows a vec! invocation, not the Vec::new() + push() that actually runs. For procedural macros (#[derive(...)], attribute macros), the generated implementation is what executes and CodeGraph can't see into it without running the compiler — which would forfeit the no-compile property that Section 1 showed is the whole point. Same shape in C/C++ preprocessor-heavy code, Lisp/Clojure macros, Elixir compile-time metaprogramming.

Metaprogramming. Python decorators routinely rewrite functions: @dataclass synthesizes __init__/__repr__/__eq__; @app.route("/users") registers a handler with a router. Tree-sitter sees the decorator and the function as adjacent syntax, not the synthesis or the registration. CodeGraph's framework adapters catch the common cases — and that's literally what the 7 heuristic edges on Hono are — but arbitrary user-defined decorators that mutate behavior are invisible. Ruby method_missing, Python __getattr__, Java reflection: same story. The graph confidently returns "no callers" for a method invoked entirely through reflection, and the LLM, trusting structured output, may hand you a confidently wrong blast radius.

Generated code. Protobuf, GraphQL codegen, OpenAPI clients, ORM model generation (Prisma, SQLAlchemy declarative), JSX/Svelte compilation — the code the runtime executes isn't the code in source control. It lives in build/, dist/, .cache/, places .gitignore excludes. CodeGraph indexes what's checked in; the generated layer is outside the boundary. "Who implements UserService?" returns the hand-written interface, not the generated stub that implements it on the wire. Any source-only index has this; it's worth naming because it interacts badly with the user's instinct that an "AST graph" must be complete. It's complete over the source it indexed — and the generated layer was never in that source.

JIT and runtime-registered bindings. DI containers (Spring, Guice, Dagger, ASP.NET service collection), FastAPI Depends, plugin systems with runtime registration, and — the one the companion benchmark hit directly — middleware chains composed at app startup. Hono's app.use(...) builds the middleware array at runtime; tree-sitter sees the use call sites and the handler as unconnected syntax. When the benchmark's Q2 asked Claude Code to trace the middleware call stack, what codegraph_trace could return was the syntactic call chain through compose() — accurate as far as it goes, and genuinely fewer steps than baseline grep — but the actual runtime ordering of middlewares is assembled by app.use calls scattered across the app, which the graph doesn't compose. The trace looked authoritative and was structurally real; it just wasn't the runtime composition, and only someone who knew the leak zone would know to check.

The empirical check, and the null result that sharpens it

I expected unresolved_refs to be where this shows up — index a macro-heavy repo, watch the table fill. So I indexed three to test it: Hono (TypeScript), click (Python, decorator-heavy), and ron (a Rust crate leaning on derive macros and serde). unresolved_refs was zero on all three; heuristic edges were 7, 0, and 0. The null result is the finding. A #[derive(Serialize)] impl never appears as an unresolved reference, because nothing in the source ever wrote a reference to it to leave dangling — the impl only exists after macro expansion. codegraph callers serialize on ron returns its seven real syntactic callers and silently omits whatever the derive generates, with no flag and no empty-table warning, because from the index's point of view nothing is missing. And that is the trap. An empty unresolved_refs table reads like a clean bill of health, but on derive-heavy or reflection-heavy code it means the opposite of "everything resolved" — it means the thing that didn't resolve never left a trace to flag. The table catches references it can't resolve; it cannot catch code that was never written down to reference. That's the leak that costs you: not the guess that gets flagged, but the absence that looks exactly like completeness. It's the same failure shape as the memory post's "could" stored as "did" — the dangerous error is always the one that wears the face of a correct answer.

Why mapping the leaks matters

A tool you trust everywhere is a tool you stop checking. The four zones above are where the LLM, trusting the graph, gives you confidently wrong answers — and those are the failures that cost real engineering time, because the answer looks right and you have no reason to second-guess it.

The practical rule is small. Inside one of these zones — heavy macros, reflection/DI, codegen-heavy projects, runtime-composed bindings — CodeGraph is still a fine starting point, but the LLM's answer has to be cross-checked against the runtime, not against the graph. Outside them — most application code in most languages, which is most of what most people query — the graph is enough. The provenance column tells you which present edges were guessed; nothing tells you which absent edges were never seen. That asymmetry is the actual trust boundary, and it's the thing to internalize before you wire any syntactic index into an agent's decision loop. Joel Spolsky named this pattern for compilers and frameworks twenty years ago — every abstraction leaks, and you pay for the leak precisely when you've forgotten the abstraction is there. CodeGraph is the latest data point in a very old series.

Mapping CodeGraph to the six conditions

Field-by-field, how CodeGraph hits each condition from Agent Retrieval Is a Cost Curve Problem. Compressed; the prior post defines the conditions, the companion post applies them empirically.

1. No-compile parsing. Tree-sitter parses source into an AST with no build invocation, no dependency resolution, no language environment. On Hono, 362 files indexed to 4,128 nodes and 8,225 edges in 1.7 seconds; the published 7-repo benchmark reports first-index on the order of minutes for VS Code-scale (~30k files), all subsequent updates incremental. LSP needs tsc / cargo check / mvn; CodeGraph reads raw text. Met.

2. Language portability. ~19 languages via tree-sitter, plus framework adapters for route-aware extraction (Hono's 873 route nodes come from one of them). One binary, no per-language server. Met.

3. LLM-shaped API. Here the scaffold version of this post — and a lot of the casual coverage — gets a fact wrong worth correcting precisely. The CLI exposes a dozen commands (query, callers, callees, impact, affected, context, …). But the MCP server exposes exactly five tools to the agent: codegraph_search (locations only), codegraph_context (described in its own schema as the PRIMARY tool, call FIRST for any how-does-X-work question), codegraph_node (one symbol plus its callers/callees trail), codegraph_explore (several related symbols in one capped call), and codegraph_trace (the call path between two symbols). The narrowing is the design: the human CLI gets impact and affected as separate verbs; the agent gets a context-first surface of five flat tools, each returning {symbol, file, line, snippet, related[]}-shaped records, with the instruction snippet steering it to codegraph_context before anything else. Ten tools would be worse for an LLM than five; CodeGraph picked five. Met, deliberately.

4. Coverage breadth. Symbol graph for structure; FTS5 over name, qualified_name, docstring, signature for text-fallback; Claude Code's native Grep stays enabled for everything outside the index. Partially met — the correct partial.

5. Live update without reindex. OS file-watcher with a short debounce; a save re-parses the touched file and re-resolves dependents' import edges. Met.

6. Zero-config install. Single binary, one-line install, auto-detects the agent, writes the MCP config and the instruction snippet, then codegraph init -i builds the index. Ten minutes from curiosity to working under ~1,000 files. Met.

Six for six. The architecture the prior post argued was theoretically right but practically missing exists, in production, with a working installer — and, read against its own schema, the choices hold up under inspection rather than just on the landing page.

What this says about LLM retrieval as a discipline

Three things, in increasing order of generality.

1. The right LLM-index design is not a copy of human-IDE design. Sourcegraph and LSP were built for a human reading one precise answer; an LLM reads many cheap rounds and reasons across them. The architectures should differ, and CodeGraph's choices — tree-sitter not LSP, five flat MCP tools not a nested LSP API, FTS5 not vectors — are evidence of someone designing for the actual consumer instead of porting an existing design. The framework predicts the design space, and the interesting variation between the tools that will fill it is not in the six conditions (those are now the table stakes) but in the ranking layer — how each one orders the symbols a query surfaces. That's where the next tool will try to win, and where the next benchmark should aim.

2. The cost-curve frame is recursive. It applies to every layer of an LLM stack, including the tools that wrap the LLM. CodeGraph's FTS5-not-Chroma choice is the same shape as the original grep-not-RAG choice. Use it as a meta-design tool: at every layer, ask which side of the curve the workload sits on, and let that pick the storage and the algorithm.

3. The abstraction leaks are the trust boundary — and trust, in the end, has to terminate at the source. This is the thread that runs through the whole series. CodeGraph's graph is a derived view of the source: a cache. Its heuristic provenance tags and its unresolved_refs table are the parts where it keeps an arrow back to that source and is honest about what it did and didn't see. But a syntactic graph is still a lossy projection of a running program, and the leak zones are exactly where the projection drops information that only exists at runtime. The discipline that falls out of this is the same one the retrieval post and the memory post arrived at from their own directions: a derived artifact is trustworthy only where you can check it against the source that produced it. CodeGraph is fast and exact in the 80% of code where syntax determines structure, and quietly incomplete in the 20% where it doesn't — and the only way to stay out of the failure modes is to remember the graph is a cache and keep the real code, the actual runtime, as the thing that wins every conflict.

The bigger move CodeGraph represents — third-party MCP tools filling the retrieval gap the foundation model's main agent doesn't fill — is the ecosystem direction the feature-flag analysis in the prior post suggested Anthropic is hedging toward. Whether Anthropic eventually builds tree-sitter symbol-graph functionality natively or leaves it to the CodeGraph-class ecosystem is a product call. The technical case for "let MCP fill it" is strong: the design space is still settling, and locking one approach into Claude Code spends option value the ecosystem is currently pricing for free.

Closing — the mini-series arc

This is the third of a three-part Lab series on Claude Code's retrieval and memory architectures:

Agent Retrieval Is a Cost Curve Problem (2026-05-25) — why grep+loop, not RAG, for most projects
Agent Memory Is a Cache Coherence Problem (2026-05-28) — why hand-curated Markdown, not lossy vector recall, for cross-session memory
This post (2026-06-08) — what lives above the cost-curve crossover: CodeGraph as the architecturally coherent symbol-graph companion the first post argued was missing, read first-principles against its own index for what its choices say about the discipline

Read together, the three describe one stance on agent retrieval and memory: choose lossless and exact by default; expose MCP as the integration substrate; let third-party tools fill the gaps you don't want to own; and keep an arrow back to the source everywhere, because every derived view is a cache and the source is the only thing that can't drift from itself. The cost-curve frame is the math, the cache-coherence frame is the failure taxonomy, and the first-principles read of CodeGraph is what the architecture, looked at carefully, says about where LLM retrieval is going.

If you're building agent retrieval, the three frames are now in your toolkit. The companion empirical post gives you the install-or-not decision; this one gives you the lens for the next ten tools that ship in the same space.

Companion piece 1 (this is the third in a 3-post Lab series): *Agent Retrieval Is a Cost Curve Problem: Why Claude Code Doesn't Use RAG***
Companion piece 2: *Agent Memory Is a Cache Coherence Problem***
Empirical pair on the Operator track: *I Tested CodeGraph on Hono. The Tool-Call Savings Reproduce — the Cost Savings Don't.***
Background: *Consistency in Distributed Systems: Scenarios, Trade-offs, and What Actually Works***
CodeGraph repo: *https://github.com/colbymchenry/codegraph***

I Tested CodeGraph on Hono. The Tool-Call Savings Reproduce — the Cost Savings Don't.

Harrison Guo — Mon, 01 Jun 2026 22:39:59 +0000

Two weeks ago CodeGraph hit GitHub trending — tree-sitter + SQLite/FTS5 + MCP for Claude Code, 19k+ stars in a week. The team published a benchmark on 7 repos showing 35% cheaper, 57% fewer tokens, 46% faster, 71% fewer tool calls vs. baseline.

Those are big numbers. They're also numbers from a benchmark designed by the team that built the tool, on repos they chose. Designer bias is the #1 risk in any retrieval benchmark — when you pick the test repos and write the ground truth, you'll consciously or unconsciously favor your own tool's strengths.

So I ran an independent test on an 8th repo — Hono (TypeScript, ~280 source files, in neither CodeGraph's published 7-repo suite nor any other published benchmark I could find). 5 architectural questions covering different retrieval shapes, with a deliberate control case (Q5) where the tool should not win. Two conditions (baseline grep+Read+Glob+Explore vs. CodeGraph active), 4 repeats per question per condition. 40 runs on Claude Opus 4.8 — and, critically, every CodeGraph run was verified to have connected, and actual codegraph_* tool usage was recorded per run (more on why that sentence exists below).

The result splits in a way the single published headline number hides — and the split is the useful part.

tl;dr — On Hono, CodeGraph delivers a large, consistent reduction in tool calls (-55%, 14.0 → 6.3 avg) and a smaller latency win (-20%) — the published 7-repo direction reproduces here. But cost is a wash: +6.8%, not the published −35%. On narrow-scope questions (route lookup, middleware trace) CodeGraph is actually 20-43% more expensive, because each structural lookup loads a big chunk of graph context that costs more in cached tokens than the grep round-trips it replaces. The cost win only appears on broad multi-file navigation (Q3 multi-runtime adapters: −29% cost, −80% tool calls, −53% latency). A second finding: baseline grep+Read has high variance — the agent occasionally spiraled to 47-52 tool calls on the broad questions, while CodeGraph never exceeded 16. Net at Hono's size: CodeGraph makes the agent take fewer steps and finish faster, but not for fewer dollars. Total cost of the 40 valid runs: ~$14 of Opus 4.8 calls. Raw per-run CSV and the 5 verbatim prompts are below.

What "tool calls down, cost flat" actually means

CodeGraph's published 7-repo suite (VS Code, Excalidraw, Django, Tokio, OkHttp, Gin, Alamofire) skews larger and more architecturally complex than Hono. Hono is ~280 TypeScript source files (362 files indexed by CodeGraph, including tests and configs), 16MB on disk — small enough that a thoughtful agent with grep + Read can finish most architectural questions in a handful of tool calls.

The interesting result is that the axes come apart. CodeGraph replaces several grep+Read round-trips with one or two structural lookups — so step count drops hard (-55%). But each codegraph_context / codegraph_explore call returns a sizeable chunk of graph context, which then rides along in the conversation cache and gets re-read every turn. At Hono's size, the dollar cost of carrying that cached payload roughly equals the dollar cost of the grep round-trips it replaced — so dollars stay flat (+7%) even as steps fall by more than half.

That's not a contradiction of the cost-curve thesis from the prior post in this mini-series — it's a sharper reading of it. Hono sits above the step-count crossover (the index already saves tool calls) but below the dollar crossover (it doesn't yet save money). On a much bigger repo, the grep path churns through far more files and the index pays back on dollars too. Hono just happens to land in the gap between the two crossovers.

A useful complementary benchmark answers three things the published one doesn't:

Cross-validation on a repo not chosen by the tool's team — do the published advantages generalize?
Within-repo variance across question types — does the win concentrate on certain question shapes? (It does — heavily.)
A control case where the tool shouldn't win — Q5 (text search) tests whether the agent correctly declines to use the structural engine when grep is the right tool.

Setup — install CodeGraph, ~10 minutes

# install (downloads a single binary, no Node/npm required)
curl -fsSL https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.sh | sh

# clone the test repo + index it
git clone https://github.com/honojs/hono.git ~/tmp/hono
cd ~/tmp/hono
codegraph init -i

Index build time on Hono (362 files, 4,128 nodes, 8,225 edges): 1.7 seconds. On-disk index: 7.1 MB.

Per-condition setup for the two arms:

Baseline (control): a clean copy of Hono via rsync -a --exclude='.codegraph/' to a separate directory so Claude couldn't accidentally grep into the index. No MCP servers registered. Agent uses native Glob + Grep + Read + Explore + Task.
CodeGraph active: original Hono directory with .codegraph/ present, MCP server registered:

{"mcpServers": {"codegraph": {"command": "codegraph", "args": ["serve", "--mcp"]}}}

Both arms run claude --print --output-format stream-json --model opus so the model and the rest of the agent loop are identical; the only varying input is whether the CodeGraph MCP server is in the loop. Each run is a fresh session with no prior context.

Verifying the tool actually ran (this is not optional)

A retrieval-tool benchmark is only valid if the tool is actually in the loop — and I learned that the hard way. My first pass at this benchmark silently ran with CodeGraph's MCP server never connected: the config was missing the --mcp flag, and Claude Code proceeds without a server that fails its hand-shake in time rather than erroring out. Every "CodeGraph" run was really just grep+Read. The comparison was noise, and the numbers looked plausibly small — which is exactly how a broken benchmark slips through.

So for the data here, every run is instrumented:

--strict-mcp-config — only the server under test is loaded, with no contamination from other globally-registered MCP servers.
Pre-warmed daemon + MCP_TIMEOUT=30000 — CodeGraph's stdio server attaches to a warm daemon and finishes its handshake before the agent loop starts. (MCP connection is async; on a fast question the agent can otherwise finish before a cold server is ready.)
A per-run assertion that CodeGraph reached connected status, plus a record of whether the agent actually invoked a codegraph_* tool. Runs that didn't connect were discarded.

All 20 CodeGraph runs in this post connected. The agent invoked CodeGraph on Q1-Q4 (4/4 repeats each) and — correctly — chose grep on the Q5 control (0/4). Most vendor benchmarks never report this check. After watching mine fail it silently, I won't publish a retrieval benchmark without it.

The 5 questions

Full verbatim prompts in the Appendix. Brief overview:

#	Question	What it tests	Hypothesis for CodeGraph
Q1	Route resolution: `GET /users/:id` → handler	Route-aware extraction	Strong win
Q2	Middleware chain trace through `app.use`	Dynamic dispatch tracing	Decisive win via structural lookup
Q3	Multi-runtime adapter architecture	Cross-file abstraction	Mid-strong win
Q4	Refactor impact: add mandatory `requestId` to `Context`	Impact propagation + completeness	Strong win (what `codegraph_impact` is built for)
Q5	Text search: every literal `'Content-Type'`	Keyword search baseline	~Parity; agent should decline the tool

Q5 is the CONTROL — the tool should not win here, and whether the agent even reaches for it is itself a signal.

The data

Each row averaged over 4 repeats. Cost is Claude's own total_cost_usd (the API's authoritative figure, not my own multiplication); wall latency from request to final token; tool calls counted from unique tool_use blocks in the transcript; tokens are input + output (cache tokens tracked separately in the CSV).

Cost / tokens

Q	Baseline cost	CodeGraph cost	Δ cost	Baseline tokens	CodeGraph tokens	Δ tokens
Q1 route	$0.321	$0.393	+22.5% ❌	10,115	6,045	−40.2%
Q2 middleware	$0.212	$0.303	+43.4% ❌	7,233	5,649	−21.9%
Q3 multi-runtime	$0.490	$0.348	−28.9% ✓✓	11,582	7,048	−39.1%
Q4 refactor	$0.402	$0.509	+26.5% ❌	9,119	8,567	−6.1%
Q5 text (ctrl)	$0.267	$0.253	−5.3%	8,874	8,998	+1.4%
Aggregate	$0.338	$0.361	+6.8%	9,385	7,261	−22.6%

Tool calls / wall latency

Q	Baseline calls	CodeGraph calls	Δ calls	Baseline latency	CodeGraph latency	Δ latency
Q1 route	7.8	6.8	−12.9%	49.8s	51.2s	+2.8%
Q2 middleware	5.0	4.0	−20.0%	41.2s	43.1s	+4.5%
Q3 multi-runtime	35.2	7.0	−80.1% ✓✓	123.7s	58.4s	−52.8% ✓✓
Q4 refactor	19.8	11.8	−40.5% ✓	87.9s	85.9s	−2.2%
Q5 text (ctrl)	2.2	2.0	−11.1%	51.2s	43.5s	−15.0%
Aggregate	14.0	6.3	−55.0%	70.8s	56.4s	−20.3%

Two headline rows: −55% tool calls (real and consistent — CodeGraph used fewer tools on every single question) and +6.8% cost (CodeGraph is not cheaper on Hono). The latency win (−20%) is real but concentrated: almost all of it is Q3; on Q1/Q2/Q4 latency is within ±5%.

The variance story — CodeGraph bounds the worst case

Averages hide the most interesting result. Baseline tool-call counts, per repeat:

Q	Baseline (4 repeats)	CodeGraph (4 repeats)
Q3 multi-runtime	14, 23, 52, 52	5, 6, 8, 9
Q4 refactor	9, 10, 13, 47	9, 10, 12, 16

On the broad questions, baseline grep+Read occasionally spiraled — the agent without an index wandered to 47-52 tool calls chasing files. Across all 40 runs, baseline ranged from 2 to 52 tool calls; CodeGraph ranged from 1 to 16. A large part of CodeGraph's value here isn't the mean — it's that it bounds the worst case. When the structural answer is one graph query away, the agent can't spiral. That's a reliability property, not just an efficiency one, and it doesn't show up in a single average.

A caveat on sample size: this is 4 repeats on one repo. Treat the magnitudes as indicative and the directions as robust — CodeGraph used fewer tool calls in every question and nearly every repeat, and the cost direction was consistent within cells (more expensive on Q1/Q2/Q4, cheaper only on Q3). What I would not over-read is the exact percentages; a 47-vs-9 baseline spread on Q4 means the per-question means carry real uncertainty even at n=4.

Per-question narrative

Q1 (route resolution) — CodeGraph used, but more expensive. Both arms traced app.fetch → #dispatch → this.router.match() and read SmartRouter → RegExpRouter. CodeGraph used codegraph_context + codegraph_trace (2 calls/run) and cut tool calls 13% and tokens 40% — but cost rose 22.5% and latency was flat. The structural context it front-loaded was heavier than the 1-2 grep steps it saved. Hono's router is small enough (5-6 files for the full picture) that grep finds it directly.

Q2 (middleware chain trace) — used, 43% more expensive. CodeGraph landed the app.use → middleware array → compose() chain in 4 tool calls vs baseline's 5, but cost jumped 43%. Same mechanism as Q1, more pronounced: the call-chain context payload dominated a question baseline answered cheaply in 5 small steps. The clearest example of "fewer steps, more dollars."

Q3 (multi-runtime adapter) — the unambiguous win. Enumerating 6 adapter directories (Cloudflare Workers / Deno / Bun / Node / AWS Lambda / Vercel Edge) is exactly where one graph query beats many Glob+grep iterations. Baseline averaged 35 tool calls and 124s (and spiraled to 52 twice); CodeGraph: 7 calls, 58s, −29% cost. This is the question shape where structural retrieval pays back on every axis at once — and the only one where it saved money.

Q4 (refactor impact: add requestId to Context) — tools halved, cost up. Supposed to be CodeGraph's strongest case (codegraph_impact is built for blast-radius). It did cut tool calls 40% (and tamed baseline's 47-call spiral), but cost rose 26.5%: the impact-graph walk pulled wide context the agent didn't fully need at Hono's size. Completeness was comparable across arms (both identified the Context constructors in src/hono-base.ts, the Variables plumbing, and the per-method handler signatures). Fewer, more-bounded steps — but the propagation graph isn't wide enough here to pay back on dollars.

Q5 (text search, control) — the agent declined the tool, and that's the point. On a pure literal-'Content-Type' search, the agent never invoked CodeGraph in any of the 4 repeats — it reached straight for grep. Result: near-parity (−5% cost, −15% latency, both inside the noise). The old version of this post claimed an "FTS5 fallback" win here; that was an artifact of the broken first run. The truth is simpler and better: with CodeGraph connected and available, the agent correctly chose grep for a grep-shaped task. No over-engineering. That's the table-stakes behavior you actually want from a retrieval tool, and it's worth more than a fabricated 1-step saving.

Cross-validation with CodeGraph's published 7-repo benchmark

Metric	Published (7 repos)	This test (Hono, n=4)	Reproduces?
Tool calls	−71%	−55%	✓ Yes — same ballpark
Latency	−46%	−20%	~ Directionally, ~half the magnitude
Tokens	−57%	−23%	~ Directionally, smaller
Cost	−35%	+6.8%	✗ No — opposite sign

The tool-call reduction is the part that generalizes cleanly to a repo the team didn't pick. The cost reduction is the part that doesn't — and that's not an attack on CodeGraph, it's a statement about repo size. Their published suite skews large (VS Code is 30k+ files; Tokio is mid-thousands), and their own published table is non-monotonic in file count — Gin (~110 files) shows a 21% cost win while OkHttp (~645 files) shows ~2%, and Tokio (~790 files) shows 82%. Repo size matters, but it isn't a clean threshold; question shape matters at least as much. A single repo can't locate a universal crossover. What Hono shows is one clear data point: at ~280 files, the step-count win is already here, the dollar win isn't yet.

Decision matrix — install CodeGraph when

Signal	Install CodeGraph	Skip (baseline grep+Read is enough)
You care about fewer agent steps / lower latency	✓ (−55% tool calls even on a small repo)
You're optimizing dollar cost on a sub-~500-file repo	(may cost slightly more)	✓
Repo is large (low thousands of files+)	✓ (dollar win should appear too)
Workload is broad multi-file navigation / architecture	✓ (this is where it wins on every axis)
Workload is narrow single-symbol lookups	(fewer steps, but not cheaper)	(grep is fine)
Static-typed (TS / Rust / Go / Java / Swift / C#)	✓
Dynamic-typed (Python / Ruby / untyped JS)	⚠️ partial (tree-sitter misses runtime semantics)
Workflow is text-search dominant	(no penalty — the agent declines the tool)	✓
Agent reliability matters (bounding worst-case exploration)	✓ (caps the 50-tool-call spirals)
You're not sure	install it; ~10 min, <2s to index a Hono-sized repo, uninstall is one command

Key call from the data: at Hono's scale the reason to install CodeGraph is fewer steps, lower latency, and bounded worst-case exploration — not a lower bill. If your decision rule is purely dollars-per-query on a small repo, baseline grep+Read is still competitive. If it's agent speed, predictability, or you're working in a larger codebase, the index earns its place.

What I'd want to test next

Larger TS repo head-to-head — same 5 questions on Prisma (~2,000 TS files) or TanStack Query (~600 files) to find where the dollar crossover actually is. Hypothesis: cost flips negative somewhere in the high hundreds to low thousands of files.
Dynamic-typed repo — same 5 questions on FastAPI or Django REST to see how much of the step-count win survives when tree-sitter can't resolve dynamic dispatch.
Long-session compounding — single-question runs miss the multi-turn agent context. Does the per-query step saving compound across a real session, or stay linear?

All future content. None block the install-or-not decision the data above already answers.

One-line verdict

On TypeScript / Rust / Go projects, install CodeGraph if you want fewer agent steps, lower latency, and bounded worst-case exploration — those reproduce on an independent repo. Don't install it expecting a lower bill on a small codebase: at Hono's ~280-file scale it was ~7% more expensive, and in this benchmark a cost win appeared only on broad multi-file navigation (Q3) — the published −35% likely needs much larger repos.

For the architectural deep-dive on why this class of tool works and where the abstractions leak, see the companion Lab piece Agent Retrieval Above the Crossover: A First-Principles Read of CodeGraph (publishing 2026-06-08).

For the broader cost-curve framework this benchmark applies, see the prior Lab post: Agent Retrieval Is a Cost Curve Problem.

Appendix: Benchmark Questions

The 5 prompts used, verbatim. Each was sent to Claude Code in a fresh session (claude --print --model opus), 4 times per arm. No follow-up prompts within a run.

Q1 — Route resolution

When a request hits GET /users/:id in a Hono app, walk me through how Hono's routing finds and invokes the right handler. Where in the source does the URL → handler matching happen, and what data structure stores the route table?

Q2 — Middleware chain trace

Hono middleware is chained via app.use(middleware). When a request flows through several middlewares before hitting the handler, what's the actual call stack from the incoming request to the handler? Specifically — how does Hono ensure middleware runs in order, and how is c (context) + next passed between them?

Q3 — Cross-file abstraction navigation

Hono supports multiple runtime adapters (Cloudflare Workers, Deno, Bun, Node, AWS Lambda, Vercel Edge). How is this multi-runtime abstraction implemented? What's the shared interface, and where do the per-runtime adapters live? Show me the architecture.

Q4 — Refactor impact

Imagine I'm planning to add a mandatory new property requestId: string to Hono's Context class. What files and functions across the codebase would be affected? Give me the full blast radius — where Context is constructed, where it's typed in signatures, and where mandatory-property additions would break.

Q5 — Text search (control)

Find every place in the Hono codebase where the literal string 'Content-Type' (the exact HTTP header name, case-sensitive) appears. Include source code, tests, comments, and documentation.

Scoring & environment

Each question evaluated on cost (total_cost_usd), tokens (input+output, cache tracked separately), wall latency, unique tool-call count, and a manual correctness/completeness check (both arms agreed on the same answer in every Q). Every CodeGraph run was additionally checked for connected MCP status and actual codegraph_* tool invocation.

Environment: Hono @ commit 2cbeadda (2026-05-28) · CodeGraph 0.9.7 · Claude Code 2.1.159 · model claude-opus-4-8 (Opus 4.8) · macOS. Raw per-run data (cost, tokens, tool calls, latency, connection status) for all 40 runs: CSV.

Appendix: Why there's no third arm (knowing)

I intended to include knowing (Blackwell Systems) as a third arm. In headless batch mode its MCP server connected only intermittently — knowing advertises asynchronous tools/listChanged, which races Claude Code's MCP startup window, so on most runs the agent never saw knowing's tools and silently fell back to grep+Read.

Reporting task-cost numbers from runs where the tool wasn't actually in the loop is exactly the trap that invalidated my first attempt at this benchmark (see Verifying the tool actually ran), so I'm not publishing knowing figures. That's a limitation of my batch harness, not a verdict on knowing — a persistent / pre-warmed MCP host or an interactive session would likely fix it. If I get a reliable knowing setup, I'll benchmark it on its own terms and publish separately.

Lab companion (first-principles architectural read of CodeGraph and the class of tools it represents): **Agent Retrieval Above the Crossover* — publishing 2026-06-08.*
Prior Lab post in the retrieval / memory mini-series: *Agent Retrieval Is a Cost Curve Problem***
Companion Lab post on cross-session memory: *Agent Memory Is a Cache Coherence Problem***
CodeGraph repo: *https://github.com/colbymchenry/codegraph***

Agent Memory Is a Cache Coherence Problem

Harrison Guo — Fri, 29 May 2026 15:17:06 +0000

This post is one half of a pair. The other half — Agent Retrieval Is a Cost Curve Problem — argues that Claude Code's within-session code retrieval avoids RAG because the cost curve says it should. This piece argues something parallel about cross-session memory: the lossy auto-capture systems being marketed as "AI memory" are, in classical distributed-systems vocabulary, caches. They inherit every problem caches have always had, and the hype around them is mostly arguing for one side of a write-back vs write-through trade as if the other side didn't exist.

Sequel to Consistency in Distributed Systems: Scenarios, Trade-offs, and What Actually Works. If you remember that piece, you'll recognize the move: take a problem space the AI community is debating with fresh vocabulary, and notice that the database community already mapped the failure modes thirty years ago.

tl;dr — Cross-session agent memory varies on two independent axes, not one: fidelity (lossless vs lossy) and retrieval (exact lookup vs approximate vector). Claude Code's built-in memory plus a hand-written CLAUDE.md lives at lossless + exact. The currently-trending claude-mem (70k+ GitHub stars as of May 2026) lives at lossy + approximate — auto-capture passed through a Haiku compression step and recalled via SQLite-FTS5 + Chroma vectors. The second is, structurally, a cache: a derived lossy view of the source of truth, retrieved approximately. It inherits every cache problem the distributed-systems literature already named — staleness, wrong-row retrieval, no coherence with the source. I ran claude-mem under controlled conditions and compared it against the deterministic CLAUDE.md baseline; the numbers (and the kinds of failures) line up with the classical cache framing. The most interesting failure isn't tokens or latency. It's that compression flattens modality — a hedged hypothetical becomes a flat fact, indistinguishable from a firm decision. An agent confidently acting on a maybe-it-said-yes is worse than an agent with no memory at all.

The Two Axes (Don't Collapse Them Into One)

Most takes on agent memory collapse the design space onto a single axis: "lossless and limited" vs "lossy and powerful." That framing hides the failure modes.

The real space is two-dimensional:

Fidelity — lossless (verbatim, what-you-wrote-is-what-was-stored) vs lossy (LLM-compressed: a summary written by a smaller model over the raw events).
Retrieval — exact / curated (you wrote an index entry; the system reads it back) vs approximate / semantic (vector embeddings; cosine similarity; top-K nearest neighbors).

The interesting failures live in the bottom-right quadrant — lossy + approximate — because the failures of one axis are invisible to a user evaluating along the other. The system loses information at write time and approximates at read time, and the user sees a single "answer" that fused both losses. Debugging means asking: was that wrong because the original event was corrupted in compression, or because retrieval surfaced the wrong row? You usually can't tell.

Most takes conflate "lossy = unreliable" with "vector = powerful." They're orthogonal. You can have lossless + vector (raw logs, vector-indexed — fine but storage-heavy and signal-weak). You can have lossy + exact (compressed summaries, FTS-indexed — works for some applications). Lossy + approximate is what's being marketed as "AI memory," and it's the quadrant most exposed to compounding failure modes.

The Baseline: Lossless + Exact (`CLAUDE.md` + built-in memory)

Claude Code's built-in memory system, paired with a hand-written CLAUDE.md, sits firmly in the lossless + exact quadrant. The design choices, made explicit:

Verbatim storage. What the author wrote is what gets stored. Markdown in, Markdown out. There's no compression step.
Always-loaded index + on-demand body. MEMORY.md (the index) gets injected into every session — capped, deliberately, around 200 lines to avoid context bloat. Individual memory files are read on demand, when the index entry suggests one is relevant.
Curated, not a firehose. A human (or a structured prompt) decides what is worth storing. Not every tool call. Not every file read. Only the durable, surprising, cross-session-useful facts.
Exact recall. The model reads a specific file. Either it's there and is read verbatim, or it isn't. No fuzzy near-matches; no confidence score.

Mental model: a hand-written WAL (write-ahead log) plus a curated index. Both are close to a source of truth — the author's deliberate decision — and they recall exactly what was committed.

Tradeoffs are visible from this framing:

✅ Precision: 100%. What you stored is what you get back.
✅ Auditability: you can read the file yourself. No black box.
✅ Token economics: index sits in context; bodies fetched only when needed. Cheap.
❌ Coverage: limited to what the author bothered to write down.
❌ Upkeep: manual. Memories rot; updating them is a chore.

The earlier post in this series, Claude Code Deep Dive Part 4: Why It Uses Markdown Files Instead of Vector DBs, walks through the specific design choices in the publicly circulated build snapshot. Here I'll focus on what happens when you put the lossless+exact baseline next to the lossy+approximate contender.

The Contender: Lossy + Approximate (`claude-mem`)

claude-mem is among the highest-starred entries in the agent-memory category right now (70k+ GitHub stars as of May 2026). I tested v13.2.0. The architecture, summarized:

Auto-capture firehose. Lifecycle hooks (SessionStart, UserPromptSubmit, PostToolUse, Stop, SessionEnd) fire on essentially everything the model does. The hooks pipe events to a Bun worker on localhost:37701.
Compression to facts/narrative. At session boundaries the worker invokes Haiku 4.5 to compress raw observations into structured facts (a JSON array) and a narrative (a paragraph). This compression runs on your own Claude subscription — billed to your quota, ~5,150 compression tokens per session in my test.
Hybrid index. Compressed observations are written to two indexes simultaneously: SQLite-FTS5 (full-text keyword) and Chroma (vector embeddings). Recall is hybrid — keyword and ANN together.
Disables built-in memory. Installation sets CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 in ~/.claude/settings.json. The built-in CLAUDE.md path is turned off; claude-mem is meant to replace, not augment.
Hardcoded data dir. Despite respecting CLAUDE_CONFIG_DIR for plugin config, the data store path (~/.claude-mem) is hardcoded. Sandboxing is partial.

Mental model: a derived, lossy materialized view of session events, plus a similarity cache for retrieval. Two layers of approximation: a lossy write transform (Haiku compression) and an approximate read transform (ANN). Each compounds the other.

This is exactly what the bottom-right quadrant looks like in deployment.

The Test (Real Numbers, 2026-05-20)

To make the comparison concrete I built a small URL-shortener as the test bed: simple enough that the "right answer" was unambiguous, structured enough that real architectural decisions had to be recorded. The setup:

Test arm: claude-mem v13.2.0, sandboxed via CLAUDE_CONFIG_DIR=/tmp/cmem-test/dot-claude. Built-in memory disabled (per install default). Session 1 established four decisions about the codebase; Session 2 asked for them back. This is the arm I measured.
Baseline: the same four decisions written into a hand-curated CLAUDE.md — 1,075 chars, ~269 tokens. Built-in memory intact. The baseline numbers in the table below are deterministic properties of how the built-in CLAUDE.md path works (verbatim recall, no extra round-trip, no compression bill), not a separately measured session.

The decisions in Session 1 (so the comparison is fair):

Use Redis for URL storage.
Generate short codes with base62.
Add a 30-day TTL.
"We could refresh the TTL on each access if we want sliding expiration." (Note the hedge — this is the modality test that matters.)

Numbers, side by side:

Measure	claude-mem (v13.2.0)	bare CLAUDE.md
Per recall-cycle tokens	~6,700	~280
↳ Passive context injection on session start	~1,050	~269 (full file)
↳ `mcp-search` retrieval round-trip	~502	0
↳ Haiku 4.5 compression cost (charged to your quota)	~5,150 / session	0
Extra round-trip for details	yes (~22s)	no
Fidelity of recall	lossy (see below)	100% verbatim
Built-in memory state	disabled (`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`)	intact
Compression cost	on user's Claude subscription	none
Headless capture (`claude -p`)	zero events	n/a
Upkeep	automatic	manual edit

The token gap — 6,700 vs 280 — is meaningful but not the headline. The headline is the fidelity row.

The Sharpest Failure: Compression Flattens Modality

The four decisions written in Session 1 included three firm choices and one hedge:

"We use Redis... base62 short codes... 30-day TTL... we could refresh the TTL on each access if we want sliding expiration."

When I read the raw observation row that claude-mem wrote, the four items appeared as a flat JSON array — facts: [...] — with no modal marker distinguishing the hedge from the decisions. The hedge had been flattened into the same shape as the firm choices.

Session 2 confirmed it. I asked the recalling agent to describe the TTL design. It cheerfully reported "we refresh the TTL on each access for sliding expiration" — as though that had been decided. When I challenged it directly, its own reply was that it could not distinguish firm decisions from options that had merely been considered. The compressed facts[] row it was reading from preserved the content of each item but not its modal status — what I'll call its epistemic status throughout the rest of this post.

That's the failure. The lossy layer loses epistemic status, not just bytes. A maybe becomes a decision. The recalling agent has no way to know it shouldn't trust the row.

This is worse than no memory. An agent with no memory has to ask, or reread, or check the code. An agent with confident-wrong memory acts. The cost of acting on a fabricated decision compounds: now there's code (or a PR, or an architectural note) committed under the false premise, and that will be the next round's input.

The generalization: any LLM compression step that maps "speech-act varieties" (decisions, hypotheses, questions, jokes, hedges) onto a single typed structure — like a facts[] array — loses the modal axis. To preserve it, you'd need to compress into a richer schema (with kind: 'decision' | 'option' | 'question' per item), and you'd need the compression model to reliably tag the modality. Haiku 4.5 didn't tag it. Whether a more careful prompt or schema would is an open question, but it's a design question the current tool doesn't even pose.

Six Measured Findings, Versioned to v13.2.0

In one place, six things I measured. Versioned because tool behavior changes:

Replaces, not augments. Install sets CLAUDE_CODE_DISABLE_AUTO_MEMORY=1. Built-in memory is turned off. Default deployment is single-system, not hybrid.
Partial sandbox. CLAUDE_CONFIG_DIR redirects plugin config, but the data store path ~/.claude-mem is hardcoded. Multi-tenant isolation is incomplete.
Compression runs on your subscription. Haiku 4.5 compresses observations to ~5,150 tokens per session, billed to your Anthropic quota. Free tools that consume your paid quota deserve a footnote.
Invisible to headless mode. claude -p runs (non-interactive) emit zero capture events in my tests. The lifecycle hooks fire only in interactive sessions. CI users and automation pipelines get no memory at all.
Compression flattens modality (the sharpest finding, detailed above). A hedge becomes a flat fact, indistinguishable from a firm decision in the compressed facts[] schema.
Token economics lose on small projects. ~6,700 tokens per recall cycle (passive inject + mcp-search round-trip + Haiku compression) versus ~280 deterministic, 100%-faithful tokens for the CLAUDE.md baseline. On a 1,000-line project, the per-token cost gap is wider than any retrieval benefit it provides.

Tool-version drift is real — by the time you read this, some of these may have been fixed. The cache-coherence framing in the next section is version-independent and was the actual reason I wrote the post.

Why This Is a Cache Problem, Precisely

The distributed-systems vocabulary for this design is materialized view of a source, served from a similarity cache.

Write-time lossy transform = a materialized view that can drift from the source of truth (the actual codebase, the actual decisions). The source is the user's intent and the live code; the view is the compressed facts/narrative. Each write step can lose information that the view will never recover.
Read-time ANN = approximate retrieval. Top-K nearest neighbors. False positives are structural, not a bug — a sufficiently-similar wrong row will be returned with confidence indistinguishable from the right one.
No coherence with the source. Classical caches have invalidation hooks — write-through, write-back, snoop protocols, MESI states. They tie cache lines back to the canonical source so that writes propagate and stale lines get evicted or rewritten. claude-mem has no tie to the codebase or to user-issued corrections. You reverse a decision in conversation, the memory still believes the original.
Staleness without expiry. Even without explicit invalidation, classical caches use TTLs to bound staleness. claude-mem has no TTL on facts. A fact written six months ago competes for retrieval with one written yesterday, and the older one might win the vector hop.

When the cache-coherence frame is the right frame, the literature is rich and useful. Pat Helland's Immutability Changes Everything (ACM Queue, 2015) and the broader databases-and-OS literature on cache-coherence protocols (MESI / MOESI), materialized-view invalidation, and write-through vs write-back are the right starting reading. The trades they describe — staleness vs cost, eventual vs strong coherence, when to flush, when to invalidate — are the same trades the agent-memory community is rediscovering with fresh names.

What's missing from this diagram, and crucially: a backedge from "source of truth" to the materialized view that fires when the source changes. That's the invalidation arrow. Its absence is the structural reason claude-mem gets wrong-row retrieval on decisions the user has reversed. Until something supplies that arrow, the system is best understood as a write-only cache.

When Each Wins

Cost-curve thinking (the same frame used in the companion piece) gives a clean answer:

Lossless + Exact wins when:

Project size is small or scope is clear. Curation is cheap; the manual upkeep budget is small.
Fidelity matters. You need to recall the exact decision, not a vibe of it. Coding agents, design decisions, security policy.
The author exists and is engaged. Someone is willing to write three lines into MEMORY.md when a decision is made.

Lossy + Approximate wins when:

History is too big to hand-curate. A year of conversations across multiple contributors, none of whom can be expected to maintain a MEMORY.md.
Coverage matters more than precision. You'd rather have a fuzzy memory that something was discussed than no memory at all. Customer-support agents over a year of tickets; team retrospectives over a quarter of standups.
The cost of acting on a fabricated fact is low. A confident-wrong recall in a support agent says "sorry let me check"; the user corrects it. The same recall in a coding agent ships broken code to production. The blast radius of a false positive determines the budget for accepting one.

The rule of thumb: fuzzy-but-present beats precise-but-absent, but only when the false-positive cost is low enough to absorb. For coding work on a 5,000-LOC project, it isn't.

A Decision Framework

Signal	Lossless + Exact	Lossy + Approximate	Hybrid
Single project, < 50k LOC	✓
Multi-project / multi-year history		✓	✓
Decisions need exact recall	✓
Vague-but-present recall is acceptable		✓
Author is engaged (willing to curate)	✓		✓
No human curator available		✓
Cost of confident-wrong is high (production code, money)	✓
Cost of confident-wrong is low (suggestion, search)		✓
You can pay the Haiku compression bill from your quota	n/a	✓	✓
You operate headlessly or via CI	✓

For most readers of this blog — engineers working on a single non-trivial codebase, where decisions matter and confident-wrong is expensive — the columns lean hard left.

What a Coherent Agent Memory System Would Need

The interesting question, once you accept the cache-coherence frame, is: what would the lossy + approximate corner look like if it were built like a real cache instead of a write-only one?

A short list of capabilities the current generation of "AI memory" tools is missing, and which any serious system in this space will eventually have to ship:

Source pointer (provenance). Every fact carries a back-pointer to the originating event: timestamp, session ID, the raw transcript turn or tool result it was derived from. Without this, you can't audit a wrong recall — you only see the fact, never its lineage.
Modality tagging. Every fact tagged with epistemic status — decision | option_considered | hypothesis | question | observation — at write time, by the compression model. Without this, the system loses what the failure section above showed: the difference between we will and we could.
Supersedes / invalidation chain. A later fact can declare an earlier fact superseded ("decision A was reversed on date T by B"). Recall surfaces the latest applicable fact, not the most semantically similar one. This is the in-band invalidation classical caches use; agent memory currently has none.
Expiry / TTL by class. Decisions might be permanent; observations rot fast ("the build was passing this morning" should not influence behavior at 4 PM). Different fact classes get different TTLs.
Invalidation hook tied to the source of truth. When the underlying codebase or document changes in a way that contradicts a stored fact, the fact gets flagged for re-validation. This is the write-through arrow in the cache diagram earlier — currently absent.
Confidence surfaced to the caller. Instead of returning a flat string, return {value, confidence, provenance}. The recalling agent then knows when to trust, when to double-check, when to ignore.

None of these is novel. All of them are standard in production cache systems, query planners, and event-sourcing stores. They're hard to retrofit onto a system that wasn't designed with provenance and invalidation as first-class concerns. They're not hard to design in from the start — but doing so means giving up the "just drop a hook on everything, ship next week" simplicity that makes the current crop of tools accumulate stars.

If you're building an agent that has to act on its memory rather than just display it, this list is the spec.

Closing — "Persistent Memory" Is a Trade, Not a Feature

"Persistent AI memory" gets talked about like a feature you turn on. It isn't. It's a choice on the two-axis design space above, and every position on that space has known failure modes. The lossless-and-exact corner has the upkeep cost and the coverage limit. The lossy-and-approximate corner has the staleness, the wrong-row retrieval, and — the finding I came away most surprised by — the loss of modality.

Two takeaways worth carrying out:

If someone is selling you "automatic AI memory," ask which quadrant. If the answer is lossy + approximate, ask the six questions from the section above: provenance, modality, supersedes, TTL, invalidation, confidence. If the answer to most of them is "the embeddings handle it," you're being sold a cache wearing the word memory.
The classical literature is the right starting point. Cache coherence, write-back vs write-through, eventual vs strong consistency, materialized-view invalidation — the database and OS communities have spent forty years working through these tradeoffs. Reading their writing is more useful than reading the latest agent-memory thread.

The companion to this post — Agent Retrieval Is a Cost Curve Problem — argues a parallel thing about within-session code retrieval: that Claude Code's "use grep, not RAG" choice isn't romance ("trust the model") but math (cost curves), and that the source code shows Anthropic A/B-testing alternative retrieval architectures (Explore vs Fork) in production. Read together, the two pieces add up to a coherent stance about Anthropic's bets across the fidelity × retrieval design space: in both within-session code search and cross-session memory, the default is lossless + exact, and the alternative branches are kept gated behind feature flags so the decisions can flip when the cost curves do. The memory side alone has at least four such gates visible in the snapshot I reviewed — tengu_coral_fern, tengu_herring_clock, tengu_passport_quail, tengu_slate_thimble — plus the build-time KAIROS, TEAMMEM, and EXTRACT_MEMORIES gates in src/memdir/.

That stance is not "we trust the model." It's: read the cost curves, build for the curves' current shape, leave the toggles in for when they shift.

Appendix: How I Measured This

For the reader who wants to reproduce — or, more usefully, who wants to know exactly what was and wasn't measured.

Versions and environment.

claude-mem v13.2.0 (npm install via the project's standard install script)
Claude Code: current public release at time of test (2026-05-20)
macOS 25.4.0, zsh
Sandbox: CLAUDE_CONFIG_DIR=/tmp/cmem-test/dot-claude for the plugin config; data store at ~/.claude-mem (this directory location is hardcoded inside the tool — see Finding #2)

Test project.
A small URL-shortener spec written from scratch, with four decisions in Session 1: (1) Redis for URL storage, (2) base62 short-code generation, (3) 30-day TTL, (4) the hedged sliding-expiration option that became the modality test.

Commands and protocol.

Fresh claude-mem install into the sandboxed CLAUDE_CONFIG_DIR. Verified install behavior set CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 in settings.json (Finding #1).
Session 1: interactive Claude Code session, walked through the four decisions with the tool actively capturing via lifecycle hooks. Watched the Bun worker on localhost:37701 accept events.
Session ended; session-boundary compression fired; Haiku-compressed facts[] and narrative written to SQLite-FTS5 + Chroma. Token count for the compression call read from the API trace.
Session 2: fresh interactive session; queried for each of the four decisions; observed the recall path (mcp-search round-trip; ~22s extra latency).
Compared retrieved content against the original Session 1 transcript byte-by-byte to identify the modality flattening (Finding #5).
Repeated the install + Session 1 pattern in headless claude -p mode to confirm Finding #4 (no events captured).

What I'm explicitly not claiming.

This is a single test run on a deliberately small project. The N is 1.
The CLAUDE.md baseline column in the test table is not a separately measured comparison session. It reflects deterministic properties of the built-in CLAUDE.md path (verbatim recall, no compression bill, no extra round-trip) that follow from the design — not a measured outcome.
I didn't benchmark Chroma vector recall quality across many queries. The modality finding came from a single targeted probe (the TTL question); the cache-coherence framing predicts the same class of failure across many queries, but predicting and measuring are different.
I tested v13.2.0. The tool is actively developed; specific findings may have been addressed in later releases by the time you read this. The cache-coherence framing is version-independent.

Artifacts. Session 1 / Session 2 transcripts, the raw claude-mem SQLite + Chroma snapshots, and the token-cost API traces from the test run are kept locally and can be made available on reasonable request — get in touch.

Companion piece: *Agent Retrieval Is a Cost Curve Problem: Why Claude Code Doesn't Use RAG***
Background: *Consistency in Distributed Systems: Scenarios, Trade-offs, and What Actually Works***
For the design rationale behind Claude Code's built-in memory in particular: *Claude Code Deep Dive Part 4: Why It Uses Markdown Files Instead of Vector DBs***

Agent Retrieval Is a Cost Curve Problem: Why Claude Code Doesn't Use RAG

Harrison Guo — Tue, 26 May 2026 03:12:20 +0000

There's a popular interview question making the rounds: "Why doesn't Claude Code use RAG to retrieve code? Why grep?"

The popular answer goes: chunking breaks code structure, vectors approximate when code demands exact, indexes go stale, cold-start is slow, retrieval is a black box. All five are real. None of them are the reason.

They're symptoms. The reason is older than RAG, older than LLMs, older than the term retrieval. It's a cost curve.

tl;dr — Index-based retrieval pays a high build cost plus a nonlinear maintenance cost in churn × index complexity. LLM tool-loop retrieval pays nothing up front and a per-query cost that's roughly project-size-independent for queries an LLM actually issues. For most small-to-mid-size repos the crossover is never reached. The "Anthropic trusts the model" framing is romantic; the actual answer is colder — build cost zero, per-query cost amortizes faster than index drift, so the math says grep.

There's also a precision axis, which most engineers care about more than cost. Vector RAG is approximate by design — getUserById returns alongside getUserByEmail because they're semantically adjacent. Code usually wants exact, which grep gives you for free. Symbol-graph indexes (Sourcegraph, Kythe, LSP) are precision-first but haven't become the LLM companion either — covered below.

Audited against a publicly circulated build snapshot of Claude Code with file:line citations. The kicker: the Explore subagent's "use this when you'll need more than 3 queries" rule is gated behind a feature flag (tengu_amber_stoat) and being A/B-tested against a parallel architecture (Fork). The canonical answer is conditional. That is the answer that gets you the offer.

The Frame That Matters: Total Cost Over Time

Pick any retrieval system and you pay for three things, on different schedules:

Build cost — one-time work to assemble whatever structure makes lookup fast. For an index, this is chunking + embedding + insert. For tool-loops, this is zero.
Maintain cost — ongoing work to keep the structure honest as the underlying data changes. For an index, this is invalidation, reindex, drift reconciliation. For tool-loops, this is also zero — the "structure" is the live filesystem.
Per-query cost — work done when a question arrives. For an index, this is a vector search + a few reranks + an LLM call. For tool-loops, this is N LLM-tool round-trips, where N varies.

The temptation is to compare per-query cost: "vector search is one round-trip, tool-loop is six." That's why RAG looks dominant on a whiteboard. But you ship a system, not a whiteboard. The bill is:

total_cost = build_cost + maintain_cost × time + per_query_cost × queries

For a project that changes daily and gets queried hourly, the term that actually grows is maintain_cost × time. For index-based retrieval on a churning codebase, maintain cost grows at least linearly with churn — and can grow faster than teams expect, because cross-chunk and cross-file references force cascading re-embeddings and symbol-graph consistency checks. A naive incremental indexer is linear; a correct one tracking cross-file refactors is often worse than linear in the worst case. For tool-loops, maintain cost is identically zero, because the loop has no persistent structure.

The build/maintain term dominates anything you save on per-query cost, until your project is large enough that per-query cost itself becomes the bottleneck. For most small-to-mid-size repos, the crossover is never reached.

That's the whole argument. The rest of this post is evidence — what the cost-curve choice looks like in source code, and where Anthropic is hedging.

A teaser for the punchline two sections down: the canonical "Claude Code spawns an Explore subagent for open-ended search" rule that most explainers quote is gated behind a feature flag (tengu_amber_stoat) and A/B-tested in production against a second architecture (Fork) that takes the opposite trade. Anthropic is hedging on the retrieval design itself. We'll come back to this in "The Subagent Twist Nobody Quotes Correctly."

The Popular Answer, Charitably

Before deflating it: the popular answer isn't wrong, it's just downstream. Briefly, with the steel-manned version:

Chunking breaks structure. A function split across chunks loses both halves of an if/else, and call-graph relationships fragment between chunks. AST-aware chunkers exist; they're better, not solved.
Vectors approximate. getUserById returns alongside getUserByEmail and getUserByName because they're semantically adjacent. Exact-symbol search beats this trivially. Important distinction: this is true of vector RAG specifically. Symbol-graph indexes — Sourcegraph, Kythe, Glean, LSP-backed code search — are a different category: they index by function/class/reference, not by chunked vector. They give exact answers and are what mega-monorepos at Google and Meta actually run for code search. When this post says "RAG," it means vector RAG. The cost-curve argument applies to vector RAG; symbol-graph has its own cost curve and lives higher up the scale spectrum. It has its own separate reason for not becoming the LLM companion — covered in the next section.
Indexes go stale. Every commit invalidates some subset of chunks. Incremental update has edge cases (renames, file moves, cross-file rename refactors). Full reindex is expensive enough to discourage frequent commits.
Cold start. Minutes-to-first-query is a non-starter for "open the tool, start working" UX.
Black-box recall. Top-K vector hits are not human-auditable. When the LLM returns a wrong answer, you can't tell whether retrieval failed or reasoning did.

All five are pains. Each has a counter — better chunkers, hybrid retrieval, incremental indexers, warm pools, attribution layers. The counters cost engineering. Some teams spend the engineering and ship working systems. Anthropic looked at the bill and decided not to.

Why? Because the baseline for code retrieval — grep over a clean filesystem with an LLM in the loop — already works well enough that adding an index is paying engineering cost to solve symptoms whose root cause is the index itself. Removing the index removes the pains. The remaining cost is per-query LLM round-trips, which the cost-curve frame says is acceptable below the crossover.

That's why grep. Everything else is engineering details on top of that decision.

So Why Hasn't Symbol-Graph Become the LLM-Companion Either?

If symbol-graph indexes are precision-first, language-aware, and battle-tested at FAANG scale, the natural question is: why didn't they become the default companion to LLM coding agents? Why grep, not LSP-over-MCP?

The answer is the same shape as the vector-RAG answer — high friction in places that don't show up on a feature comparison — but the specific frictions are different:

Build cost is high in a different way. Symbol-graph indexes need to compile (or semi-compile) the project to resolve symbols. For Rust, C++, large TypeScript or Java codebases this is minutes to tens of minutes per cold start. "Open Claude Code and start working" can't pay that toll.
Language-specific, not portable. LSP is one server per language. Tree-sitter coverage helps but isn't uniform. A grep-backed agent works on any text in any language with zero setup; a symbol-graph-backed agent inherits the project's language-server matrix.
API/format mismatch with how LLMs reason. LSP returns deeply nested JSON (locations, ranges, document hierarchies); grep returns file:line: content. The second is almost literally an LLM's native dialect; the first needs adapting. The translation tax is real.
Coverage is narrower than it looks. Symbol-graph models code-as-structure. It misses config files, comments, strings, generated code, markdown, env files, shell scripts, the README — all of which are first-class context for a real coding session. Grep covers anything that's text.
The win is for structure questions, not intent questions. "Where is getUserById defined?" — symbol-graph is exactly right. "How does the login flow work?" — back to grep + read. A real coding day has both kinds; building infrastructure that only solves one kind is paying a high fixed cost for half the answer.
The constraint reversed under it. Symbol-graph was designed for a world where the constraint was human attention bandwidth — give a developer one precise answer they can read. LLMs don't have that constraint; they can read 30 grep hits cheaply and reason across them. The bottleneck moved from "precision of retrieval" to "fluency of the model reading the retrieval." Symbol-graph is optimizing the part that's no longer expensive.

One-line summary: symbol-graph is precision tooling built for human IDEs. LLMs are not human IDEs. Their retrieval bottleneck is different — they prefer many cheap rounds over one expensive precise call. Installing a symbol-graph for an agent that can grep thirty times in a session is, roughly, hiring a second driver for someone who already drives.

This is also why the few existing LLM ↔ symbol-graph integrations (Cursor's @symbol references via LSP, Sourcegraph Cody, Codeium with LSP backend) are additive niceties in those products, not the retrieval backbone. The backbone is still the same as Claude Code's — grep over text.

The Three Primitives, Audited

Source paths below are from a publicly circulated, non-public build snapshot of Claude Code that I have on disk for analysis purposes. APIs and exact line numbers will drift; the design choices below have been stable in the snapshot I reviewed, and match observed runtime behavior on current public Claude Code releases.

Grep — ripgrep, with structured output and a "don't shell out" enforcement clause

The Grep tool description, from src/tools/GrepTool/prompt.ts:7-16, is short and pointed:

A powerful search tool built on ripgrep

  Usage:
  - ALWAYS use Grep for search tasks. NEVER invoke `grep` or `rg` as a Bash command. The Grep tool has been optimized for correct permissions and access.
  - Supports full regex syntax (e.g., "log.*Error", "function\s+\w+")
  - Filter files with glob parameter (e.g., "*.js", "**/*.tsx") or type parameter (e.g., "js", "py", "rust")
  - Output modes: "content" shows matching lines, "files_with_matches" shows only file paths (default), "count" shows match counts
  - Use Agent tool for open-ended searches requiring multiple rounds
  - Pattern syntax: Uses ripgrep (not grep) - literal braces need escaping (use `interface\{\}` to find `interface{}` in Go code)
  - Multiline matching: By default patterns match within single lines only. For cross-line patterns like `struct \{[\s\S]*?field`, use `multiline: true`

Three things to read out of this:

The ALWAYS / NEVER is doing work. The model has Bash. It could shell out to rg or grep directly. The prompt forbids it. Why? Three reasons, in order of importance:
- Permission surface. Bash is a universal tool. Auditing what the model can do with Bash means auditing every shell command. Audited Grep means audited Grep, period.
- Output discipline. A Bash rg dumps raw text into the context window. Grep returns one of three structured modes — content, files_with_matches, count — letting the model pick the cheapest mode that answers the question. This is a token-budget decision, not a feature decision.
- Backend swap. Today the implementation is ripgrep. Tomorrow it might be bfs/ugrep (the source already has a branch for embedded-search tools — see hasEmbeddedSearchTools() in src/utils/embeddedTools.ts). A tool boundary makes the swap invisible to the model.
The default output mode is files_with_matches. Not content. The model has to opt in to seeing actual lines. This is a token-conservation default: most of the time, the model wants to know which files matched so it can narrow further; only when it's ready to read does it ask for the lines.
Multiline is opt-in. Default ripgrep is line-bounded — a deliberate restriction, because cross-line regex on a large tree is a perf cliff. The model can opt into multiline when it knows it needs to, paying the cost only then.

These are all small choices. Each one shaves a tail off the per-query cost. Cumulatively, they're why the per-query term in our cost equation stays low enough that the build/maintain term — zero — dominates.

Glob — filename patterns with a recency heuristic and a hard cap

Glob is the "find files by name" primitive. Two design tells in its construction:

Results are sorted by mtime descending. Most-recently-modified file first. The heuristic is that in any given session, the files you've touched recently are the files you're about to touch again. This is the same logic IDEs use for the "Recent Files" list, and it's empirically right far more than it's wrong.
A hard cap of 100 results. Past that, output truncates. The model can tighten the pattern and re-call. The cap exists because an LLM that consumes 800 file paths because it asked for **/* is an LLM that's burned a quarter of its context on noise.

Both are token-budget decisions disguised as ergonomic ones. The mtime sort means a small N typically covers the relevant set. The 100-file cap means a careless query degrades gracefully instead of catastrophically.

Read — bounded, fresh, stat-checked

src/tools/FileReadTool/prompt.ts is the most interesting of the three because of what it constrains:

- By default, it reads up to 2000 lines starting from the beginning of the file (...)
- When you already know which part of the file you need, only read that part. This can be important for larger files.

(MAX_LINES_TO_READ = 2000 at line 10; the "only read that part" guidance is OFFSET_INSTRUCTION_TARGETED at line 20-21, swapped in dynamically based on context.)

A 2000-line default cap. offset and limit parameters for targeted reads. And — critically — every read calls stat on disk. No cache. No index. No staleness layer.

The implication is that Read is always live. The model that just edited a file and wants to see the result reads the file and gets the new bytes. The model that's iterating on a fix doesn't fight cache invalidation because there is no cache to invalidate. This is the "no maintain cost" term in the cost equation, made concrete: the live filesystem is the index, and the filesystem is always up to date with itself.

The OFFSET_INSTRUCTION_TARGETED template is worth noting separately. It's swapped in over the default when the prompting context suggests the model already knows what range it wants. It's a tiny prompt-engineering detail, but it's also a piece of evidence that the team thinks carefully about teaching the model to read selectively. The lesson the model is being taught, every prompt, is don't be greedy. That's exactly the discipline that keeps per-query cost from blowing up.

The composition

Walk through a real query: "Where's the login flow in this project?"

Glob `/login.{ts,tsx,js}`** — returns up to 100 files, most-recently-modified first. Usually under ten matches; usually the right one is in the first three.
Grep passport|auth|login --glob '' — narrows to specific lines. Three output modes available; the model picks the cheapest one that disambiguates.
Read the file, with offset/limit targeted at the matched region — reads only what's needed.

Three primitives. One realistic query. Total cost: three round-trips, a few hundred tokens of output total. No index, no embedding, no rebuild. Filesystem unchanged.

Now add ten more iterations as the model investigates a bug. Each one is the same three primitives, in different combinations. The cost grows linearly with iterations; it doesn't grow super-linearly with the codebase. That's the curve.

The Loop, Sketched Honestly

You can find pseudo-code versions of Claude Code's main loop in essays and threads. They all look something like:

while (true) {
  const response = await callLLM(messages);
  if (!response.toolUses.length) break;
  for (const use of response.toolUses) {
    const result = await execute(use);
    messages.push(result);
  }
}

That sketch is conceptually right. It is also missing roughly 1,415 lines.

The real loop is at src/query.ts:307 (while (true) {) and closes at :1728 (} // while (true)). The body is 1,421 lines. The full file is 1,729. If you want a guided tour, the Claude Code Deep Dive Part 2 walks through it section by section. The summary for our purposes here is shorter:

The Platonic loop — call model, run tools, repeat — is six lines. The other 1,415 lines are doing one of four things:

Streaming tool execution. Tool calls don't wait for the full model response before they start running; they execute as the streaming tool_use blocks arrive. This is non-trivial because the model can emit a partial tool_use, retract it, or continue thinking before the rest of the call arrives.
Cache and compaction management. Microcompact maintains tool-result cache coherence by tool_use_id without inspecting content. The auto-compact path triggers when the message stream nears a context budget. Both interact with the prompt cache, which has its own coherence rules.
Failure recovery. A nested while (attemptWithFallback) at line 654 handles model fallback when the primary returns a recoverable error. There's a separate path for max_output_tokens recovery (when the model would emit a tool_use but truncate before the tool_result). Orphan tool_results are aggressively pruned so a retry doesn't carry stale IDs forward.
Thinking-block preservation. Reasoning content has to span the assistant trajectory — same turn, plus any subsequent tool_use/tool_result chain — because the model expects to see its own prior reasoning to continue coherently. The loop preserves these blocks across iterations.

The point is not "the loop is complicated." The point is: the cost of running an agent on tool-loops moves into the loop, where it's controllable. There's no second system — no index pipeline, no vector store, no embedding service — with its own failure modes, latency budgets, and consistency guarantees. The loop is the engineering target.

This is the same property that makes a single-process database easier to operate than a microservices fleet, for reasons that have nothing to do with the database. Concentrate complexity where you can attack it.

The Subagent Twist Nobody Quotes Correctly

This is the section where most explanations of Claude Code's retrieval get it wrong, including the article that prompted this post. The popular telling goes: "For open-ended exploration, Claude Code spawns an Explore subagent — Anthropic codifies the rule that if you need more than three queries, you should fork."

The rule exists. The popular telling has two things wrong about it:

The rule is not in AgentTool/prompt.ts. It's in src/constants/prompts.ts:378-379, injected into the system prompt by template:

For simple, directed codebase searches (e.g. for a specific file/class/function) use [searchTools] directly.

For broader codebase exploration and deep research, use the Agent tool with subagent_type=Explore. This is slower than using [searchTools] directly, so use this only when a simple, directed search proves to be insufficient or when your task will clearly require more than [EXPLORE_AGENT_MIN_QUERIES] queries.

EXPLORE_AGENT_MIN_QUERIES is defined in src/tools/AgentTool/built-in/exploreAgent.ts:59 as the integer 3. So the "more than 3 queries" threshold is literal — but it's a single named constant, easy to change, and the guidance is interpolated dynamically per system-prompt build.

The rule is conditional. The two lines above are only emitted when both conditions hold:

// src/constants/prompts.ts:374-381
...(hasAgentTool &&
areExplorePlanAgentsEnabled() &&
!isForkSubagentEnabled()
  ? [
      `For simple, directed codebase searches ...`,
      `For broader codebase exploration ... subagent_type=Explore ...`,
    ]
  : []),

Both areExplorePlanAgentsEnabled() and !isForkSubagentEnabled() have to be true. The first is itself feature-gated:

// src/tools/AgentTool/builtInAgents.ts:13-22
export function areExplorePlanAgentsEnabled(): boolean {
  if (feature('BUILTIN_EXPLORE_PLAN_AGENTS')) {
    // 3P default: true — Bedrock/Vertex keep agents enabled (matches pre-experiment
    // external behavior). A/B test treatment sets false to measure impact of removal.
    return getFeatureValue_CACHED_MAY_BE_STALE('tengu_amber_stoat', true)
  }
  return false
}

BUILTIN_EXPLORE_PLAN_AGENTS is a Bun build feature gate. tengu_amber_stoat is a GrowthBook key. (Aside: tengu_* is the Anthropic-internal naming prefix for Claude Code — any flag you see with that prefix in the source is a Claude Code feature toggle.) The comment in source is the giveaway: "A/B test treatment sets false to measure impact of removal." Anthropic is actively testing what happens when they remove Explore and Plan agents entirely for some fraction of internal users.

Read that again. The canonical "Anthropic uses Explore for exploration" claim is true for the default treatment and being measured against the no-Explore baseline. The interview answer that says "Anthropic always uses Explore" is wrong — or at least, more confident than the source.

This is the half-credit point. Knowing that the rule exists is the first half. Knowing the rule is feature-gated and under measurement is the second half.

The Economics of Explore, in Detail

If you read just the function of Explore, the design looks like a generic subagent: open-ended exploration, returns a summary, isolates from the main context. If you read its parameters, the economics jump out:

// src/tools/AgentTool/built-in/exploreAgent.ts:64-83
export const EXPLORE_AGENT: BuiltInAgentDefinition = {
  agentType: 'Explore',
  whenToUse: EXPLORE_WHEN_TO_USE,
  disallowedTools: [
    AGENT_TOOL_NAME,             // can't spawn nested subagents
    EXIT_PLAN_MODE_TOOL_NAME,
    FILE_EDIT_TOOL_NAME,         // read-only
    FILE_WRITE_TOOL_NAME,        // read-only
    NOTEBOOK_EDIT_TOOL_NAME,
  ],
  source: 'built-in',
  baseDir: 'built-in',
  // Ants get inherit to use the main agent's model; external users get haiku for speed
  // Note: For ants, getAgentModel() checks tengu_explore_agent GrowthBook flag at runtime
  model: process.env.USER_TYPE === 'ant' ? 'inherit' : 'haiku',
  // Explore is a fast read-only search agent — it doesn't need commit/PR/lint
  // rules from CLAUDE.md. The main agent has full context and interprets results.
  omitClaudeMd: true,
  getSystemPrompt: () => getExploreSystemPrompt(),
}

Three economic decisions encoded here:

Explore runs on Haiku for external users. Not the main reasoning model. Exploration is a cheap-tokens job — there's no creative reasoning happening, just iterate-and-filter — and Anthropic uses a fast, small, cheap model for it. The main agent gets the expensive model when it gets the summary back. This is the staffing analogue: junior associate does the deposition review, senior partner reads the brief.
Explore omits CLAUDE.md. The argument in the inline comment: "The main agent has full context and interprets results." CLAUDE.md is for project rules — commit conventions, PR style, lint guidance. Explore isn't doing any of those things. Loading CLAUDE.md into its prompt would be paying tokens for guidance it can't act on.
Explore can't spawn subagents. AGENT_TOOL_NAME is in disallowedTools. No recursion. This is a budget guarantee: an Explore call has bounded depth.

And the system prompt makes the read-only constraint impossible to misread:

=== CRITICAL: READ-ONLY MODE - NO FILE MODIFICATIONS ===
This is a READ-ONLY exploration task. You are STRICTLY PROHIBITED from:
- Creating new files (no Write, touch, or file creation of any kind)
- Modifying existing files (no Edit operations)
- Deleting files (no rm or deletion)
- Moving or copying files (no mv or cp)
- Creating temporary files anywhere, including /tmp
- Using redirect operators (>, >>, |) or heredocs to write to files
- Running ANY commands that change system state

(exploreAgent.ts:26-34. The full prompt also requires Explore to spawn parallel tool calls aggressively — "you must try to spawn multiple parallel tool calls for grepping and reading files" — which is yet another budget squeeze: turn N round-trips into one wall-clock round, lower latency, same token bill.)

Pull the threads together and Explore reads like a budgeted exploration worker: small model, no CLAUDE.md noise, no nested recursion, read-only, parallel-first. It is the most economically tuned piece of the retrieval system, and it's there because once you've decided the model drives retrieval, the next question is which model, and what it gets paid to think about.

The popular telling skips all of this. It says "Anthropic spawns Explore." The source says Anthropic spawns the cheapest possible agent that can do the work, strips its context to the minimum that lets it function, and measures whether spawning it at all beats just letting the main agent loop directly. That last clause is the A/B in the previous section.

Fork — The Architecture Under Test

Here's the other half of the experiment.

src/tools/AgentTool/forkSubagent.ts exports isForkSubagentEnabled(). When that returns true, AgentTool/prompt.ts rewrites the system prompt to introduce a new operation: fork.

The fork section, paraphrased from prompt.ts:80-96:

Fork yourself (omit subagent_type) when the intermediate tool output isn't worth keeping in your context. The criterion is qualitative — "will I need this output again" — not task size. Forks are cheap because they share your prompt cache. Don't set model on a fork — a different model can't reuse the parent's cache.

Don't peek. The tool result includes an output_file path — do not Read or tail it unless the user explicitly asks for a progress check. You get a completion notification; trust it.

Don't race. After launching, you know nothing about what the fork found. Never fabricate or predict fork results. The notification arrives as a user-role message in a later turn; it is never something you write yourself.

Read this against Explore and the trade becomes visible:

	Explore	Fork
Context isolation	Fresh subagent, separate context	Inherits parent context
Model	Haiku (cheap)	Same as parent (no swap allowed)
Prompt cache	New cache, no parent reuse	Reuses parent cache (huge savings on the system prompt and prior turns)
CLAUDE.md	Omitted	Inherited
Recursion	Disallowed	Allowed (forks can fork)
Failure model	Subagent returns summary or error	Notification arrives later as user message
Discipline required	Caller frames the query	Caller writes a directive prompt; trusts the notification

Neither dominates. Explore wins when isolation matters more than cost — the exploration is noisy enough that you don't want any of the tool spew in your main context, and the work is mechanical enough that Haiku can handle it. Fork wins when the work needs the main model's depth and you're willing to pay the cache-reuse savings to get it, accepting that the fork is just a future-you with a directive.

Anthropic is shipping both, gated, and watching which one wins on internal metrics. The interview answer that says "Claude Code uses Explore for exploration" is a snapshot of one branch of the experiment. The next release cycle may settle it differently.

This is exactly the kind of fact you can't get from the popular tellings, because they're written from a single observation of behavior. Reading the source gives you the experimental design.

Version caveat. Flag names (BUILTIN_EXPLORE_PLAN_AGENTS, tengu_amber_stoat, isForkSubagentEnabled) and exact gate semantics reflect the snapshot I reviewed and will drift — Anthropic ships fast. The two specific architectures (Explore, Fork) may be renamed, consolidated, or replaced by the time you read this. What's stable is the pattern: Anthropic running multiple retrieval architectures in parallel, gated, measuring against each other. That pattern outlasts any specific flag name and is what the argument here actually rests on. Verify against the current release before depending on any specific flag.

When RAG Still Wins

The cost-curve thesis predicts when RAG crosses over and dominates. Three regimes:

1. Mega-monorepos where any kind of index is amortized across millions of queries

At Google or Meta or a top-tier hedge fund — codebases large enough that every per-query latency point costs real money across the org — there is an index. But, importantly, the index they actually run is almost always symbol-graph (Kythe, Glean, Sourcegraph) rather than vector RAG, for the precision reasons in the earlier section. The index is built once, maintained by a dedicated team, amortized across the engineering population's queries forever. Per-query cost goes to single-digit milliseconds; index drift is handled.

In this regime, tool-loops are paying the same per-query cost over and over across N engineers, and the math flips toward indexing. Below that scale, the staffing cost of "a dedicated team that owns the code index" is itself a hidden cost that wipes the savings — and the index of choice at smaller scales (per-project Sourcegraph, ctags) is still not vector RAG. Vector RAG specifically tends to win in the next two regimes, not this one.

2. Pure semantic queries where there's no symbol to grep for

"Find the code that handles user-deactivation edge cases when the account is also a billing admin" — there's no specific symbol. There's a conceptual region of the code that you're looking for. A vector search over function-doc embeddings might point you to the right cluster of functions faster than the model would grep.

Claude Code's answer to this is: spawn Explore, let it iterate. That works, but it isn't free. If your workload is dominated by semantic-cluster queries (auditing, security review, refactor planning), RAG starts to pencil out.

3. When LLM context is the scarce resource — cheap, short-context model regime

This is the framing nobody else gives. Cost-curves cut both ways. If your model has a 32k context window and costs $0.50/M tokens, doing six tool round-trips for retrieval is six rounds of context consumption. A one-shot RAG hit lets you spend that context budget on reasoning. RAG dominates when per-query token cost dominates per-query latency cost.

Anthropic optimized Claude Code for the regime where context is cheap and abundant (Opus 4.7 with a 1M-token window, in some configurations) and per-query latency is what users feel. In a different regime — say, on-device coding agents over a small open-source model — the cost-curve flips and RAG is the right tool. Same first principles, different numerical answer.

A Decision Framework

Match your project to the column. The cost-curve answers the question for you.

Signal	Use grep + LLM tool-loop	Use RAG	Either / hybrid
Project size: under ~1M lines	✓
Project size: 1M–100M lines	✓		✓
Project size: 100M+ lines (mega-monorepo)		✓
Codebase changes daily	✓
Codebase is mostly static (knowledge base, archived)		✓
Queries are exact-symbol ("find getUserById")	✓
Queries are conceptual ("how does auth work")		✓	✓
Model context is cheap and large	✓
Model context is scarce/expensive		✓
You don't have a team to own a code index	✓
You have an index team and tooling already (Sourcegraph, Glean, ctags)		✓	✓

The crossover doesn't happen at any one threshold; multiple columns moving in the same direction tips the cost curve. For most non-FAANG projects, the columns sit on the grep side.

The Companion Question

This post is one half of a two-part argument about Claude Code's retrieval and memory choices. The companion — "Agent Memory Is a Cache Coherence Problem" (publishing 2026-05-28) — makes the same kind of argument about cross-session memory: why Claude Code's built-in memory is hand-written Markdown instead of vector-recalled embeddings, even with the world hyping claude-mem (70k+ GitHub stars as of May 2026) as a drop-in upgrade.

Read together, the two pieces add up to a coherent design stance. Anthropic's bet, across both axes:

Fidelity over fuzz. Both within-session retrieval (grep) and cross-session memory (CLAUDE.md) are lossless and exact. Both refuse vector approximation as the default.
Cost curves over romance. Neither choice is justified by "we trust the model." Both are justified by the math: zero build cost + zero maintain cost beats nonlinear maintain cost for the workloads they target.
Experimentation in production. Both architectures have alternative branches under active flag gating. On the retrieval side: tengu_amber_stoat for Explore-vs-no-Explore, with Fork as a parallel architecture. On the memory side: tengu_coral_fern, tengu_herring_clock, tengu_passport_quail, tengu_slate_thimble, plus the build-time gates KAIROS, TEAMMEM, and EXTRACT_MEMORIES — all visible in src/memdir/. The pattern is the same: ship a default, leave the toggles in, keep measuring.

The shape of the design is a careful refusal to lock in. The cost curves favor the current choices today. They might not in a year. The system is built to flip.

That, finally, is the answer the interview question is fishing for. Why not RAG? Because the cost curves don't justify it for this workload, and Anthropic has the engineering culture to refuse the cargo-cult. Will it always be that way? No — and the feature flags in the source say so out loud.

Companion piece: **Agent Memory Is a Cache Coherence Problem* (publishing 2026-05-28)*
Background: *Consistency in Distributed Systems: Scenarios, Trade-offs, and What Actually Works***
For a focused walk through the loop file: *Claude Code Deep Dive Part 2: The 1,421-Line While Loop***

Channels Aren't Message Passing — How Parked Goroutines OOM-Killed a Pod

Harrison Guo — Thu, 14 May 2026 05:26:27 +0000

It's 3am. The Kafka consumer pod that's been running cleanly for six weeks gets OOM-killed. Kubernetes restarts it. Five minutes later: OOM-killed again. Restart. OOM-killed a third time. By the fourth restart I've shelved the dashboard and started reading runtime/chan.go.

The code that died fit on one line:

events := make(chan Event)

I want to tell you that line is the bug. It isn't. An unbuffered channel will happily backpressure a single producer — every send rendezvous with a receiver, the producer cannot run ahead. The channel did exactly what it was designed to do.

What I had built around it didn't. The Kafka consumer loop wrapped events <- parseEvent(msg) inside a go func(msg) { ... }(msg), spawning a fresh goroutine per inbound message. Every one of those goroutines blocked on send, parked on the channel's sendq list, and kept its stack and the parsed event alive in memory. The channel was the gravestone. The unbounded go func fan-out was what filled it.

This is the story of what a Go channel actually is at the runtime level, why "channels are message passing" is one of the most expensive lies in the Go ecosystem, and why the most common channel bug isn't in the channel — it's in the code that calls into it.

tl;dr — A Go channel is not a queue and not a message bus. It's a heap-allocated hchan struct containing a mutex, a ring buffer, and two parked-goroutine lists. The send operation is a memcpy under a lock, not a transmission. Channels only deliver backpressure if the producer side is bounded. The OOM that started this story came not from make(chan Event) — that was working as designed — but from an unbounded go func(msg) fan-out parking thousands of goroutines on sendq, each retaining a 10KB payload. The fix isn't a buffer size. It's making backpressure part of the producer contract: a single long-lived producer with select-based backoff, plus a bounded queue as a safety net. The same architectural mistake shows up at every layer where engineers reach for an "in-process queue" — including the inbound queue of your AI agent.

The Mental Model That Killed The Pod

Here is what I thought a channel did, and I suspect most Go engineers carry some version of this picture:

"A channel is like a Kafka topic in-process. Producers push messages onto it. Consumers pull messages off it. The runtime handles ordering and delivery. It's CSP — Communicating Sequential Processes — Hoare's thing, basically a typed pipe."

Every word of that sentence is wrong in a way that matters. There is no topic. Nothing is being pushed anywhere. The runtime is not a broker. The word passing — borrowed from message-passing concurrency, where independent processes communicate across an isolation boundary — is the most misleading part. In a Go channel, there is no isolation boundary. There is one struct on the heap, and both goroutines reach in and mutate it.

I held the message-passing model long enough that when the Kafka consumer started ingesting a 12-hour upstream replay at full throttle, I had no instinct that the messages were going somewhere bounded. They weren't. They were sitting in a ring buffer that I had failed to give a size to.

What A Channel Actually Is

Crack open runtime/chan.go in the Go source tree and you'll find this (layout stable since Go 1.7, confirmed against Go 1.21–1.25):

type hchan struct {
    qcount   uint           // total data in the queue
    dataqsiz uint           // size of the circular queue
    buf      unsafe.Pointer // points to an array of dataqsiz elements
    elemsize uint16
    closed   uint32
    elemtype *_type
    sendx    uint           // send index
    recvx    uint           // receive index
    recvq    waitq          // list of recv waiters
    sendq    waitq          // list of send waiters
    lock     mutex
}

That's it. That's the channel. A struct with a mutex, a pointer to a circular byte array, two indices to track read/write positions in the ring, and two intrusive linked lists holding parked goroutines that are waiting to send or receive.

When you write ch <- value, the runtime calls chansend, which does roughly this:

Take the lock (lock(&c.lock)).
Check recvq — is there a goroutine already parked waiting to receive? If yes, copy value directly from the sender's stack into the receiver's stack via sendDirect, mark the receiver runnable with goready, release the lock, return. No buffer involved — when a receiver is already waiting, send can hand off directly without ever touching the ring buffer. (In normal operation a buffered channel can't simultaneously have queued data AND parked receivers; if recvq has a waiter, the buffer is empty.)
Otherwise, check buffer space — if qcount < dataqsiz, copy value into buf[sendx], advance sendx, increment qcount, release the lock, return.
Otherwise, park the sender — append the sender's goroutine to sendq, release the lock, and call gopark to suspend execution until a receiver wakes it up.

Receive is the mirror image, calling chanrecv with sendq and recvq swapped.

Here is the shape of it:

Three things are worth burning into memory:

One — there is no transport. The "message" never leaves the heap. Sender writes bytes; receiver reads bytes; the lock arbitrates. This is shared-memory synchronization with the appearance of message passing.

Two — the buffer is just a ring of typed slots. dataqsiz is set exactly once, at make(chan T, N) time. If you write make(chan T), dataqsiz is zero and there is no buffer at all — every send must rendezvous with a receiver or park.

Three — sendq is unbounded. This is the part nobody talks about. The ring buffer has a fixed size. The list of parked senders waiting to write into the ring buffer does not. If a thousand goroutines all hit a full channel, the runtime parks all thousand of them on sendq and each one keeps its stack and any data it was about to send alive in memory.

That third point is what made the OOM I had a different shape from the one I was about to describe.

The Incident, Mechanism By Mechanism

The pod that died had a goroutine topology that looked like this — and the bug is not the make(chan Event) line. Watch the outer loop:

events := make(chan Event)

// Consumer — slow.
go func() {
    for ev := range events {
        process(ev) // ~3ms per event
    }
}()

// THE ACTUAL BUG: outer loop spawns a fresh goroutine per inbound message.
for msg := range kafkaConsumer.Messages() {
    go func(msg kafka.Message) {
        events <- parseEvent(msg) // every blocked send parks on sendq
    }(msg)
}

If you replace the inner go func(msg) { ... }(msg) with a direct events <- parseEvent(msg), the outer loop itself becomes the producer, and the unbuffered channel correctly backpressures it — the loop simply doesn't advance until the consumer is ready. No OOM.

But because each message is dispatched to a fresh helper goroutine, the outer loop never blocks. It keeps spawning. Each helper goroutine reaches the send, finds no waiting receiver, and parks on sendq. Now sendq is the unbounded thing. Here is what actually happened, in order:

1. Sustained baseline: rendezvous works

At 1K msg/sec inbound and ~3ms per process call (~333/sec consumer throughput), the consumer is already behind by 3x at steady state. For weeks this didn't OOM because the Kafka client's own internal buffer absorbed the gap, and lag built up on the broker side — visible in Grafana, ignored by me.

2. Replay: the producer detaches from the consumer's pace

When upstream re-emitted 12 hours of events, the Kafka client's internal pre-fetch buffer filled to capacity (default fetch.message.max.bytes × partition count = several hundred MB) and started backing up Kafka-side without applying backpressure to the consumer goroutine, because the client library was configured with a large internal queue.

3. The actual heap growth: parked sender goroutines

Each call to events <- parseEvent(msg) on the unbuffered channel would either rendezvous (rare during replay) or park. When it parked, the sender goroutine held:

Its own stack (~8KB minimum, grew under load)
The Event value it was about to send (~10KB per event with strings, headers, payload)
A reference into the Kafka message it was parsing (another ~10KB)

Multiply by the number of in-flight parsing goroutines — which kept being spawned by an outer loop that didn't apply backpressure to itself — and you arrive at the 12GB heap. The channel's sendq was the proximate memory sink, not the buffer (which was zero-sized).

The goroutine lifecycle for each parsing goroutine looked like this:

Every goroutine sitting in Parked_on_sendq is reachable (it's on the runtime's wait queue, which is rooted in the hchan struct, which is rooted by both the producer and consumer goroutines). Reachable means non-collectible. The longer the consumer falls behind, the longer the queue grows.

4. GC can't help

Go's GC can only reclaim unreachable memory. Every parked goroutine on sendq is reachable (it's on the runtime's scheduler queue). Every Event it's holding is reachable. The GC ran, found nothing to free, and the heap continued growing until the kernel OOM-killer fired.

5. The cgroup hammer drops

cgroup memory limit was 4GB. Heap crossed 4GB. OOM kill. Kubernetes restarted the pod. The replay was still in progress on the broker side, so the same sequence ran again. And again.

What this looks like in pprof

You don't have to take my word for the mechanism — it reproduces in under a minute. I built a minimal demo at harrison001/channels-oom-demo (cmd/bug) that runs the same workload shape on a laptop. The output of the bug version over 22 seconds, captured with runtime.NumGoroutine() and runtime.MemStats.HeapAlloc:

t=   1s  goroutines=   497  heap_alloc=     5 MB
t=   5s  goroutines=  2462  heap_alloc=    28 MB
t=  10s  goroutines=  4915  heap_alloc=    61 MB
t=  15s  goroutines=  7369  heap_alloc=    89 MB
t=  20s  goroutines=  9828  heap_alloc=   109 MB
t=  22s  goroutines= 10813  heap_alloc=   125 MB

Goroutine count grows at almost exactly 1 per millisecond (the spawn rate). Heap grows at ~5MB/sec, dominated by the 10KB Event payload each parked goroutine is holding. Extrapolate to a 12-hour replay at production volume and you arrive at the original 12GB OOM.

For comparison, the fix version (cmd/fix) on the same workload:

t=   1s  goroutines=     3  heap_alloc=     3 MB  chan_len= 256
t=  10s  goroutines=     3  heap_alloc=     4 MB  chan_len= 256
t=  20s  goroutines=     3  heap_alloc=     5 MB  chan_len= 256

Three goroutines (producer, consumer, pprof listener). Heap flat at 4-5 MB. Channel pinned at its 256-slot bound, meaning the producer is constantly blocked on send and applying backpressure upstream — exactly what we want.

The Fix, And Why It Works

The visible code change was one parameter. The real fix was making backpressure part of the producer contract — two changes, working together:

events := make(chan Event, 256) // (1) bounded queue as safety net

// (2) single long-lived producer goroutine with select-based backoff —
// NO outer loop spawning fresh goroutines per message.
go func() {
    for msg := range kafkaConsumer.Messages() {
        select {
        case events <- parseEvent(msg):
            // sent — loop continues at consumer speed when channel fills
        case <-ctx.Done():
            return
        }
    }
}()

The key word in change (2) is single. There is exactly one goroutine reading from Kafka and writing to the channel. When the channel fills, that goroutine blocks on send; the for msg := range loop stops calling Poll(); the Kafka client's internal pre-fetch queue stops draining; consumer lag accumulates broker-side; the broker simply retains messages until we come back. No go func(msg) helpers. Nothing piling up on sendq. Memory stays bounded because the producer is bounded — the buffer is only a safety net to absorb short bursts.

What this changes, mechanically:

Before (unbounded `go func` fan-out + `make(chan Event)`)	After (single producer + `make(chan Event, 256)`)
One goroutine per inbound message	One long-lived producer goroutine
`sendq` grows unboundedly with parked helpers	`sendq` empty by construction; producer is sole sender
No signal to upstream — outer loop never blocks	Producer blocks on send; outer loop runs at consumer speed
Kafka client keeps pre-fetching, lag invisible	Kafka client's internal queue fills, consumer stops polling, broker-side lag accumulates
OOM	Bounded heap, bounded latency, Kafka rebalances cleanly when behind

A bounded channel buffer alone does not prevent this OOM. If you applied change (1) without change (2), you'd merely increase the OOM-killing rate — the outer go func(msg) fan-out would keep spawning, the buffer would fill in milliseconds, helpers would pile up on sendq exactly as before. Backpressure is not a property of any one component — it is a property of the entire chain having no unbounded buffer (and no unbounded fan-out) anywhere in it.

Every link in this chain is bounded — the database has connection pool limits, the consumer is rate-limited by process() latency, the channel buffer is 256, the Kafka client's internal queue has a configured max, and the broker simply retains messages on disk when its consumer falls behind. When ANY downstream link slows, the pressure propagates back up by the consumer ceasing to pull; the broker doesn't need to be told anything. The whole system runs at the rate of its slowest component.

If any link in that chain has an unbounded buffer, the chain has no backpressure. That link will absorb the load until it OOMs.

Bounded Buffers Are Not About Channels

The lesson is not "use buffered channels." The lesson is:

Any in-process queue without a capacity bound is a latent OOM.

This applies identically across runtimes:

Runtime	The footgun	The fix
Go	Unbounded goroutine fan-out parked on sends (`go func(msg) { ch <- ... }(msg)`); oversized buffered channels	Single long-lived producer + `select` + bounded buffer as safety net
Rust (Tokio)	`mpsc::unbounded_channel()`	`mpsc::channel(N)`
Python (asyncio)	`asyncio.Queue()` with no `maxsize`	`asyncio.Queue(maxsize=N)`
Node.js	Unbounded array of in-flight Promises	`p-limit`, `Sema`, or explicit pool
Erlang/Elixir	Process mailbox grows unboundedly when selective receive can't keep up	Demand-driven flow control: `GenStage` / `Flow` for pipelines, or explicit ack-based protocols in `gen_statem`

Every one of these reaches for the same shape — an in-process queue — and every one of them OOMs the same way when the shape is unbounded.

When Channels Are The Right Tool

I want to be careful not to overcorrect. Channels are not a mistake. They are an excellent primitive used incorrectly. Cases where reaching for a channel is the right call:

Cancellation signaling — context.Done() is a <-chan struct{}. This is canonical.
Fan-out work distribution with a worker pool — a bounded channel feeding N worker goroutines is a clean semaphore. Buffer size = pool size or small multiple of it.
Producer-consumer with a known throughput ratio — yes, with a bounded buffer sized to the latency budget.
Error aggregation from concurrent goroutines — small buffered channel, drain on goroutine completion.
Handoff between pipeline stages — bounded, with explicit close semantics on the upstream stage.

Cases where reaching for a channel is the wrong call:

Cross-process messaging — use a real broker (NATS, Kafka, Redis Streams). Channels do not survive a pod restart.
Persistence — channels are stack-local-ish. If your pod dies, the in-flight data is gone. If you need "at least once" across restarts, you need a real queue.
Bursty load with unknown shape — if you cannot put a meaningful upper bound on the buffer, you have not understood the load. Adding a channel does not give you understanding; it postpones the OOM.
Anything that wants to be a message bus — that's not a channel. That's a message bus. They are different categories of system.

The Same Bug, Different Layer: AI Agent Inbound Queues

The reason this post lives in the SecurityLab track and not just "Go tips" is that the exact same mistake is now happening, at scale, in LLM agent infrastructure. I've seen the pattern repeatedly in recent AI backends — same architectural shape, different runtime.

The pattern: an agent backend exposes an HTTP endpoint. Each inbound request is dispatched to a worker pool via an in-process queue.

# The bug, in a different language
request_queue = asyncio.Queue()  # unbounded

async def http_handler(req):
    await request_queue.put(req)  # never blocks
    return {"status": "queued"}

async def worker():
    while True:
        req = await request_queue.get()
        await llm_call(req)  # 8 seconds, sometimes 30

Steady state is fine: requests arrive faster than they're processed, queue grows slowly, latency creeps up, nobody notices because the HTTP layer keeps returning 200.

Then a launch happens. Or a viral tweet. Or a marketing email goes out. Inbound rate spikes 50x for 20 minutes. The queue accepts everything (it's unbounded). The worker pool can't keep up — LLM calls are inelastic, you can't parallelize past your token-per-minute quota. The queue grows to 200K items. Each item holds a request payload (~50KB with conversation history) and a future. 10GB of heap. OOM. Pod restart. All 200K requests lost. Users see 500s instead of the explicit "rate-limited, try again in 30s" they would have seen with proper backpressure.

The fix is identical to the Go fix:

request_queue = asyncio.Queue(maxsize=100)

async def http_handler(req):
    try:
        request_queue.put_nowait(req)
    except asyncio.QueueFull:
        return Response(status=503, headers={"Retry-After": "30"})
    return {"status": "queued"}

503 is a feature. It is the system telling the client we're at capacity, retry in 30 seconds. It is honest. It is bounded. It is the difference between a system that degrades gracefully and one that dies silently.

Reproducing This Yourself

The numbers in this post come from a minimal Go program that fits in under 100 lines per command. The repo lives at:

github.com/harrison001/channels-oom-demo

git clone https://github.com/harrison001/channels-oom-demo.git
cd channels-oom-demo

# Watch goroutine count + heap climb every second
go run ./cmd/bug

# Switch to the fix — flat at 3 goroutines, 5 MB heap
go run ./cmd/fix

Each program exposes pprof on localhost:6060. While the bug version is running:

# Confirm 10K+ goroutines parked on chansend → runtime_chanrecv1
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=1' | head -20

# Confirm the heap is dominated by Event payloads, not the channel itself
go tool pprof -text http://localhost:6060/debug/pprof/heap

The bug demo has a hard cap at 20,000 goroutines so it won't actually OOM your laptop. Remove the cap if you want to see the kernel finish the job.

What I Wish I'd Known

If I could send one note back to myself eighteen months before the OOM:

When you reach for an in-process queue, you are choosing a backpressure boundary. The buffer size is not a performance tuning knob. It is a contract: under sustained load greater than my consumer's throughput, this is how much memory I am willing to lose before I tell the producer to stop. If you don't pick a number, the runtime picks one for you, and the number is whatever fits in RAM right before the kernel kills the process.

Channels in Go look like message-passing because the syntax was deliberately borrowed from CSP, a model where independent processes communicate by passing values across an isolation boundary. In Go there is no isolation boundary. The channel is a struct in shared memory, the goroutines are coroutines on the same scheduler, and the entire setup is synchronization plumbing in CSP clothing.

Once you see the hchan struct, you can't un-see it. Every channel decision after that is a synchronization decision, not a transport decision. And synchronization decisions always have a capacity bound — you just have to choose whether to pick it explicitly or have the OOM-killer pick it for you.

Keep going

Code: harrison001/channels-oom-demo — reproduce both versions, capture your own pprof
Next piece: Goroutines Are Cheap — Until Backpressure Is Missing — coming next. The producer side of the same mistake: why "just spawn a goroutine" is the second half of the bug.
Subscribe: I write one of these monthly on runtime mechanics, distributed systems postmortems, and the security implications of getting them wrong. Newsletter · SecurityLab track

If you've hit this bug — or its cousin in a different runtime — I'd genuinely like to hear about it. The Erlang and Node.js shapes especially: I have hunches but not enough scars. Reply to the newsletter or open an issue on the demo repo.

How I Improved an AI Agent from 40% to 60% — With A/B Test Data

Harrison Guo — Tue, 12 May 2026 15:49:19 +0000

The Setup

I was optimizing an AI agent for a production system — a creator agent that handles user requests like "make this character fiercer" or "rename this entity." The agent runs a 5-layer pipeline: Perceive → Cognate → Decide → Act → Express, with real LLM calls at each step.

Quality was bad. Not "it doesn't work" bad — "it works 40% of the time" bad. The remaining 60% were wrong entity targeting, infinite reasoning loops, and silent failures.

I ran 5 standardized test cases, each repeated 5 times (LLMs are non-deterministic), measuring pass rate:

Test	What It Does	Baseline
QL-001	Create 4 entities + 1 relationship in one message	0%
QL-002	Classify user intent correctly	80%
QL-003	Update the right entity in a world with 6 characters + 4 locations	40%
QL-004	Maintain context across long conversation	100%
QL-005	Simple rename ("Ember" → "Infernia")	20%

Overall: 40% pass rate. The model (equivalent to GPT-4 class) was plenty capable. Something else was wrong.

The Diagnosis: Context Was the Problem

QL-003: Why the Agent Confused Entities (40% → 80%)

The user says: "Make Ember more fierce and give her fire breath."

The world has 10 entities: 6 characters (Ember, Luna, Grak, Roland, Mira, Pip) and 4 locations. The agent's BuildChatCompletionMessages function dumped ALL entity data into the prompt — every character's backstory, every location's description.

The LLM had to find Ember in a wall of irrelevant text. Sometimes it picked Luna. Sometimes it referenced the wrong character's traits. Not because the model was stupid — because the context was noisy.

QL-005: Why Simple Rename Failed (20% → 80%)

"Rename Ember to Infernia." One entity, one operation. Should be trivial.

Two problems:

No round limit — the agent sometimes looped 15+ times on a rename, reasoning tools firing endlessly
When a tool failed, the LLM got: {"error": true, "message": "This tool is temporarily unavailable."} — no context on what to do next

The model gave up or produced responses that didn't contain "Infernia."

QL-001: Why Multi-Step Creation Was Impossible (0% → 0%)

"Create a dragon named Ember who lives in Crystal Caves. Ember has a rivalry with Sir Roland who guards the village gate."

This requires creating 4 entities + 1 relationship. The 5-layer pipeline processes entities sequentially, each in isolation. The relationship creation doesn't know the knight was just created — there's no shared state between action steps.

Both baseline and improved scored 0%. This is an architectural problem, not a context problem.

The Fixes: 8 Changes, 7 Pure Code

Fix 1: PlanExecution (the only LLM call)

One API call before the main loop. The LLM generates a plan:

Goal: Update Ember's properties
Steps: 1. Identify Ember entity  2. Apply personality changes
Tools needed: updateCharacter

This plan gets injected into the cognition layer's context. The intent classifier now sees a roadmap, not just raw entity data.

Cost: ~$0.003 per request, 3-5s latency. The only fix that uses an LLM call.

Fix 2: PrioritizeContext (pure code)

Sort context items by salience score. Higher-relevance items go first. Low-relevance items dropped when the token budget is exceeded.

When the user says "Make Ember fiercer," Ember's data gets priority. Luna's backstory gets dropped. The LLM sees signal, not noise.

sort.Slice(items, func(i, j int) bool {
    return items[i].Salience > items[j].Salience
})
items = items[:tokenBudget]

Cost: Zero. Pure sort + filter.

Fix 3: CompressContext (pure code)

Old conversation rounds get summarized extractively — find tool names, find CONCLUSION markers, truncate the rest. No LLM needed for this level of compression.

Cost: Zero. String operations.

Fix 4: Preserve Conclusions (pure code)

When reasoning text is truncated at 4,000 characters, the truncation used to cut wherever it landed. If the LLM decided "I need to rename Ember to Infernia" in round 1 but that conclusion was at character 4,100, round 2 forgot the decision.

Fix: truncateReasoningPreservingConclusions() finds CONCLUSION/DECISION markers and keeps them even when truncating.

Cost: Zero. String search.

Fix 5: Max Rounds Cap (pure code)

const DefaultMaxRounds = 10
if roundCount > DefaultMaxRounds { break }

Previously unlimited. The agent sometimes looped 15+ rounds on a trivial task. Now it stops at 10 and produces its best result.

Cost: Zero. One if-statement.

Fix 6: Structured Tool Errors (pure code)

Before:

{"error": true, "tool_name": "updateCharacter", "message": "This tool is temporarily unavailable."}

After:

{"error": true, "tool_name": "updateCharacter", "message": "This tool is temporarily unavailable.",
 "error_type": "timeout", "retryable": true}

With retryable: true, the LLM knows to try again instead of giving up. With error_type: "timeout", it knows the issue is transient.

Cost: Zero. String classification.

Fix 7: Circuit Breaker (pure code)

Count failures per LLM provider. After 3 consecutive failures, skip that provider and try the fallback. Prevents the agent from burning through 120 seconds of timeout on a dead provider.

Cost: Zero. Counter + threshold.

Fix 8: HTTP Client Reuse (pure code)

Store *http.Client on the provider struct, reuse across calls. Previously each call created a new client, a new TCP connection, a new TLS handshake.

Cost: Zero. Struct field.

The Results

Test	Baseline	After Fix	Delta	What Fixed It
QL-001	0%	0%	=	Needs pipeline architecture change
QL-002	80%	80%	=	Already working
QL-003	40%	80%	+40%	PrioritizeContext + PlanExecution
QL-004	100%	100%	=	Already working
QL-005	20%	80%	+60%	Max rounds + structured errors + conclusion preservation

Overall: 40% → 60%. Same model. Better input.

Latency went from 26s to 43s due to the PlanExecution LLM call (~3-5s per test). The HTTP reuse and circuit breaker savings show up under concurrent load, not in a 5-test sequential run.

What Didn't Improve — And Why

QL-001 (multi-step creation) stayed at 0%. This isn't a context problem — it's a pipeline architecture problem. Each entity is created in isolation, and the IDs returned by each step are discarded before the next step runs:

Fixing this requires collapsing the 5-layer pipeline into a unified agent with cross-step state — a larger architectural change, not a context fix.

The lesson: Context optimization has a ceiling. Past that ceiling, you need architecture changes. But the ceiling is higher than most people think — we still had 20% improvement available before hitting it.

What's Still Missing

Three pieces of infrastructure were built but not wired:

Component	Status	Gap
VerifyOutput	Logs quality issues	Doesn't retry on failure
ScoreMemoryUsage	Computes relevance scores	Scores never applied to future retrieval
PlanExecution	Generates plan before loop	Plan not tracked during execution

All three are open loops. The infrastructure detects problems but doesn't act on them. Closing these loops is the next 20% — getting from 60% to 80%+.

The Takeaway

Better input → better output. The LLM is the same.

If your agent is underperforming, check the context before blaming the model. In our case:

7 out of 8 fixes were pure code
Zero additional LLM cost (except one planning call at $0.003)
20% quality improvement without changing the model
The model was always capable — the context was holding it back

The highest-ROI investment in any agent system is context management. It's not glamorous. It's sort, filter, compress, truncate, prioritize. But it's the difference between 40% and 60% — and the foundation for everything else.

Part of the AI Agent Architecture series. See also: The 90% Problem for the broader framework, and Claude Code Deep Dive Part 3 for how Anthropic solves context at scale.

Consistency in Distributed Systems: Scenarios, Trade-offs, and What Actually Works

Harrison Guo — Wed, 06 May 2026 15:50:16 +0000

There's an impulse, when someone first learns about consistency models in distributed systems, to want to classify the taxonomy into neat drawers. Strong here. Eventual there. Linearizable above it. Read-your-writes below. Study the diagram, pass the interview.

That taxonomy is real, but it's not useful the way people think. Production systems don't pick a consistency model and run with it. They pick a different model per feature, often per type of operation within a feature, and spend most of their engineering effort on the gaps between what the model provides and what users actually expect. The taxonomy is the menu. The interesting question is which dish each scenario needs.

This is a working engineer's walk through ten real consistency scenarios — from the obvious ones (money transfers need strong) to the less obvious (collaborative editing, notification feeds, analytic dashboards) — with the specific engineering that makes each one work.

tl;dr — Consistency is not a global system property; it's a per-operation property. A well-designed distributed system picks different consistency levels for different operations based on what users actually notice, what the business actually requires, and what latency budget each operation has. The CAP-theorem framing ("pick 2 of 3") is a caricature; real systems use PACELC (which adds the latency trade-off during normal operation) and pick per-feature.

The Frames That Matter

Before scenarios, three frames you actually use in practice.

CAP (Consistency, Availability, Partition tolerance, pick 2). Useful as a first-week mental model. Misleading if taken literally, because (a) you can't give up partition tolerance in a real network, and (b) the choice isn't binary — you can tune per operation.

PACELC: if there's a Partition, pick A (availability) or C (consistency). Else, pick L (latency) or C (consistency). Adds the latency trade-off you pay during normal operation, which is where 99% of design decisions actually live. A system that's "consistent when no partition" but pays 50ms of cross-region round-trip for every write has made a latency-vs-consistency call, not a CAP call.

Consistency models, from strongest to weakest:

Linearizable: operations appear to happen instantaneously, in a total order consistent with real time. The strongest practical model. Expensive.
Sequential: operations appear in a total order, but not necessarily aligned with real time. Slightly weaker, slightly cheaper.
Causal: if event A causally precedes event B, every observer sees A before B. Preserves the "this reply should appear after the comment it replied to" property.
Read-your-writes: you see the effects of your own operations, even if other users don't yet.
Monotonic read: once you see a value, you won't see an older value later.
Eventual: if writes stop, replicas eventually converge. No ordering guarantees during the transient.

You don't need to memorize these. You need to recognize which one each feature actually needs.

Moving left to right: cheaper, faster, less coordinated — and more work you do in application code to close the gap between what the model gives you and what users expect.

Ten Scenarios

Quick reference — each row is expanded into its own section below.

#	Scenario	Consistency Model	Key Technique
1	Money transfer between accounts	Linearizable	Synchronous quorum + idempotency keys
2	Inventory decrement, hot key	Strong w/ sharding	Reserved-inventory buckets
3	User profile update	Read-your-writes	Session timestamp + sticky read
4	Social media feed	Causal	Version vectors / Lamport timestamps
5	Collaborative document editing	Eventual + CRDT/OT	Conflict-free operations
6	Ad click counter	Eventual	Local shard + async aggregation
7	Multi-region primary/secondary	Eventual + RYW on demand	Primary routing per write
8	Distributed lock / leader election	Linearizable	Raft/Paxos consensus
9	Analytics dashboard	Append-only / none	Stream → warehouse ETL
10	Cross-service orchestration	Saga	Local txns + compensations

1. Money Transfer Between Accounts

Needs: strict linearizability. No double-spend. No lost updates.

Approach: transactional database with serializable isolation, or a strongly-consistent coordination layer (Paxos/Raft quorum). Typical implementation: single-region primary Postgres with synchronous replication, or a distributed SQL (Spanner, CockroachDB, YugabyteDB) with linearizable reads.

What you give up: latency (especially cross-region), availability during partitions. This is the right trade — a bank doesn't tolerate double-spend to save 30ms.

Key engineering: idempotency keys on every request, deduplication at the persistence layer, well-audited transaction boundaries. Strong consistency at the DB isn't enough if your retry logic double-writes.

2. Inventory Decrement with High Contention

Needs: "no overselling" without blocking every request.

Approach: the classic "hot key" problem. Options in ascending sophistication:

Pessimistic locking — SELECT ... FOR UPDATE on the inventory row. Works; serializes hot items. Under peak traffic on Black Friday, this queues up and tail latencies explode.
Optimistic concurrency — read version, decrement, compare-and-swap. Retries on conflict. Better tail latency at moderate contention, worse at very high contention (retry storms).
Reserved-inventory buckets — maintain N "shards" of available inventory, route requests to a random shard, only one shard hits contention at a time. Sacrifices a small amount of overselling risk (if shard A has 5 left but shard B has 0, a user might get told "out of stock" while 5 remain total) for huge throughput wins.
Best-effort with async reconciliation — accept orders optimistically, reconcile at a background worker, cancel overbooks with apology emails. Used by event-ticketing sites for popular drops.

The right choice depends on business rules. If overselling by 1% is unacceptable, pessimistic. If overselling by 0.1% is tolerable and user-experience matters, shard or async reconcile.

3. User Profile Update

Needs: read-your-writes. After I save my display name, I see it on next page load.

Approach: sticky reads. Either the session pins to the write replica for a short window, or the application tracks a "last write timestamp" per user and refuses to serve reads from a replica that hasn't caught up.

The naive alternative — "eventually consistent, just retry" — breaks user expectations immediately. "I updated my name and it didn't save" is one of the most expensive support tickets on a per-incident basis, because the user has no way to distinguish "didn't save" from "saved but replication is lagging."

The engineering is not glamorous. A session cookie that carries last_write_ts, a read path that asserts replica.latest_ts >= last_write_ts, and a fallback to the primary if the assertion fails. Most frameworks don't give you this for free; you build it.

4. Social Media Feed

Needs: causal consistency for comments and replies. Eventual consistency everywhere else.

Approach: two-tier. Posts and likes are written to a local region with async replication. Replies are linked to their parent post with an explicit cause-precedes relationship — the reply's store won't surface the reply until the parent has propagated.

The CRDT-adjacent pattern (version vectors, Lamport timestamps) sits underneath, but you don't usually expose it to the application. What you expose is "here's the list of replies, in causally-consistent order." What the user sees: "I replied to a comment, and my reply appears under it" — which is exactly the mental model they expect.

What you save by not using strong consistency everywhere: low write latency (local region only), high availability during partitions, and the ability to handle massive fan-out (a celebrity's post propagating to 40M followers doesn't need to wait on a single coordinator).

5. Collaborative Document Editing

Needs: offline-first, multi-user concurrent edits, always-eventually-converge, no lost updates.

Approach: CRDT (conflict-free replicated data type) or OT (operational transformation). This is one of the few spots where CRDTs genuinely shine. The underlying math guarantees that any two replicas will converge to the same state, regardless of the order operations arrive in, as long as all operations eventually reach all replicas.

Google Docs uses a version of OT. Figma uses multivalue registers and CRDT-adjacent primitives. Notion uses a mix. The common property: any user can edit while offline, sync when reconnected, and the final document reflects all edits.

What you give up: simplicity. CRDT implementations are subtle, and naive "last-write-wins" semantics are almost never what the user wants (their previous sentence vanished, not merged).

What you gain: offline support without an ugly "you've been offline, your changes may conflict" modal.

6. Ad Click Counter

Needs: eventual consistency, very high write throughput, lossy-okay for a tiny fraction.

Approach: local counter per shard, periodic aggregation to central store. Writes are fire-and-forget to a stream (Kafka, Kinesis). Reads come from a precomputed aggregate that's a few seconds stale.

Why this works: no advertiser is going to detect the difference between "47,312 clicks" and "47,318 clicks" in their dashboard. Counting-with-precision across a global distributed system is ten times harder than counting approximately. Do the latter.

What's non-obvious: the system should be designed for approximate counts, with explicit tolerance in the SLA ("counts are accurate to within 0.01% and updated every 30 seconds"). If you don't say that upfront, someone will eventually ask "why don't our counts match the backend logs exactly" and you'll be in a two-week project to eliminate errors that never mattered.

7. Multi-Region Primary / Secondary

Needs: fast reads in every region, writes can live in one region.

Approach: primary-in-region-A, async replication to regions B/C/D. Reads in B/C/D may lag the primary by milliseconds to seconds. Writes always route to A.

Consistency model you're serving: eventual, with read-your-writes available on demand (see scenario 3). Reads from the primary are strongly consistent; reads from secondaries are lagged but fast.

Key engineering: the client SDK should know which operations need primary reads (after a recent write, for "show me the thing I just wrote" operations) and which can hit secondaries (dashboards, history views, anything time-insensitive).

This is where most backend systems actually live. The bulk of reads go to secondaries — cheap, fast. A small percentage route to primary for freshness. Latency and availability both win.

8. Distributed Lock / Leader Election

Needs: exactly one leader, no split-brain, sometimes-unavailable-is-okay.

Approach: a consensus system (Zookeeper, etcd, Consul — all Raft or Paxos variants). Acquire the lock or lease, renew it, do the work, release. If you lose the network partition, the other side knows you lost it because it couldn't renew.

The classic failure: leader election on top of Redis. Redis is not a consensus system. RedLock has well-documented failure modes — it is not safe for correctness-critical locking. Use etcd. Use Zookeeper. Use a real consensus system. The tempting shortcut will, eventually, bite.

What consensus buys you: guaranteed linearizability for operations on the lock/lease. What it costs: every operation is a quorum round-trip. That's fine for leader election (infrequent). It's not fine for a hot write path (use a different mechanism).

9. Analytics Dashboard

Needs: all the data, eventually, in a queryable form. No urgency on freshness.

Approach: stream writes to a durable log (Kafka), have an ETL job populate a columnar warehouse (BigQuery, ClickHouse, Snowflake) on a schedule. Dashboards query the warehouse. Data is minutes to hours stale.

Consistency model: none, in the traditional sense. You have an append-only log and a materialized view. The view is eventually consistent with the log, and that's the whole contract.

This is simple but worth calling out because people sometimes try to do analytics against the operational database directly ("we'll run these queries on the primary, it'll be fine"). It will not be fine. Analytic queries are different workload shapes — they want columnar storage, aggressive parallelism, no transactional overhead. Put them in a warehouse.

10. Cross-Service Orchestration (Saga)

Needs: multi-step business flow across services — create order, reserve inventory, charge payment, schedule shipment. Each step might fail. The system should end up in a consistent state either way.

Approach: Saga. Each step is a local transaction in its own service. For each step, you also define a compensating step that undoes it. If a step fails partway through, you run compensations for the earlier steps in reverse:

Not all compensations are symmetric — you can't un-send an email, you can't un-refund a payment. But for most business flows, you can design compensations that leave the system in a consistent-enough state.

The alternative — 2PC (two-phase commit) across all services — is real but rarely used. 2PC requires every participant to support the protocol, holds locks while waiting, and blocks the whole transaction if any participant is slow or down. For services owned by different teams on different storage engines, 2PC doesn't scale.

Saga engineering concerns: saga orchestrators (a coordinator service that runs the state machine) vs saga choreography (each service emits events that trigger the next). Orchestrators are simpler to reason about. Choreography scales further but can produce spaghetti.

The Meta-Rule

Walking through those ten: the choice isn't really "which consistency model is best for my system." It's "which consistency model does this specific operation need, given what users expect to see."

Most production systems use all of the following, in different places:

Strong/linearizable consistency for anything money-related.
Read-your-writes for user-visible writes that users need to see immediately.
Causal consistency for feed-like data where ordering matters.
Eventual consistency for counters, analytics, and anything where approximate-and-fast beats exact-and-slow.
CRDTs (narrowly) for collaborative editing and specific offline-first features.
Saga for cross-service business flows.
Consensus (Zookeeper/etcd) for the very few things that actually need leader election or distributed locks.

The engineering decision is not "pick a consistency level for the whole system." It's "for this specific feature, what consistency level does the user need to experience, what trade-offs does the stronger version cost, and can we engineer the weaker version to feel as good?"

That last clause matters. A read-your-writes layer on top of eventual consistency often feels strongly consistent to users while actually being cheap to operate. Users don't experience consistency models; they experience whether their updates show up, whether their comments appear in order, whether their refund matches what they expected. Engineering consistency is about closing the gap between the model you can afford and the experience the user requires.

Common Anti-Patterns

A few shapes that show up repeatedly in code reviews:

"Retry until consistent." Seen in code that does a write, then reads from a secondary and loops until it sees the write. Works on the happy path, deadlocks on partition, creates unbounded retry storms under load. Use read-your-writes through a session token instead.

"We'll use eventual consistency for speed." Used as a justification for skipping engineering. Yes, eventual is faster. The engineering to make it feel correct (causal ordering, conflict resolution, read-your-writes fallback) is what you're skipping — and users will notice.

"Just use Redis for leader election." Already mentioned. Redlock is not safe. If you're doing anything correctness-critical with leader election, use a real consensus system.

"Saga with no compensations." "What happens if step 3 fails?" "Oh, we'll fix it manually." That's a saga you haven't designed. It's a half-finished state machine waiting to corrupt data. Design the compensations before you ship.

"Strong consistency everywhere, for safety." Default-safe sounds responsible. It also means your read latency is 50ms minimum, you can't serve a region during a partition, and the cost per query is high. Users rarely need strong consistency everywhere. They need it in a few specific places.

The Senior Move

Consistency is a user-experience feature, not a system property. The right question at design time isn't "what consistency model does our database provide" — it's "what does the user need to see, in what order, with what freshness, with what tolerance for partial failure."

Most of the work in a well-designed distributed system is engineering around the consistency model the storage layer provides: sticky reads, session tokens, version vectors, compensating actions, explicit ordering, user-visible "your change is saved" confirmations. The model is the floor; the engineering lifts the experience to what users actually expect.

The difference between senior and junior distributed-systems work often shows up here. Junior picks a model and fights everything else to conform. Senior picks the model per-feature, builds the engineering scaffolding that closes the gap, and ships something that feels right to users — even though underneath, ten different operations run on five different consistency levels.

Why Your "Fail-Fast" Strategy is Killing Your Distributed System — another "the system property is not the user experience" essay.
RPC vs NATS: It's Not About Sync vs Async — It's About Who Owns Completion — who owns completion is one of the things consistency models don't address.
From Locks to Actors: The Four Pillars of Modern Concurrency — the concurrency side of the same general question.
NATS vs Kafka vs MQTT: Same Category, Very Different Jobs — the durability choice that enables some of the consistency patterns here.

Don't Pick One AI. Run Three Against Each Other.

Harrison Guo — Sun, 03 May 2026 19:18:57 +0000

The Problem Nobody Talks About

AI can write code, generate content, analyze data, design systems, and manage projects. It's getting better every month. The natural question: what's left for humans?

The wrong answer: "AI will replace us."
The other wrong answer: "AI is just a tool, nothing changes."

The right answer is uncomfortable: stop picking the best AI. Run multiple AIs in competition, and become the judge.

The Tournament Model

Three rules, learned the hard way:

Multiple advisors, competing opinions. Don't bind to one AI — its bias becomes yours. Three models running the same task surface blind spots no single model catches.
You decide. After the AIs argue, you make the call. Not the smartest model — you. The one with context they don't have.
Results judge everyone. Did the call work? Keep it. Did it fail? Learn and move on. Never blame the AI — you chose to follow that advice.

That's the operating system for the AI age.

In Practice: Three AIs in One Window

Theory is cheap. The reason most people don't run multiple AIs is friction — opening three terminals, signing in to three CLIs, and pasting the same prompt three times kills the loop after one day.

So I wrote two small scripts and one tmux config to remove the friction. They live in this repo: harrison001/ai-tournament.

script	what it does
`prj`	one command opens a tmux window with codex, claude, and gemini running side-by-side, plus a shell pane
`tmx`	fzf picker to switch between tournament sessions
`tmux.conf`	binds `<prefix> b` to broadcast — type once, all three AIs receive it

The workflow:

$ prj ai-tournament       # opens tmux: codex | claude | gemini | shell
$ <prefix> b              # broadcast ON — status bar turns red
$ <type the question>     # all three receive the same prompt
$ <prefix> b              # broadcast OFF, read each answer, pick the winner

A real run

To test the loop I asked all three the same dumb question I had asked myself five minutes earlier:

"I want to wash my car. The car wash is 30 meters from my house. Should I drive or walk?"

Here is what came back, unedited:

codex: "Walk. For 30 meters, driving adds unnecessary startup, parking, and maneuvering, and it is barely faster unless you need to move supplies or the car itself into a wash bay."
claude: "Walk. At 30 meters, driving takes longer once you factor in starting the car, and you'd just be sitting in a freshly washed car driving it home through whatever's on the road. Walk over, drive back clean."
gemini: "You should drive. Although 30 meters is a very short distance to walk, your car needs to be physically present at the facility to be cleaned."

Two out of three confidently gave a wrong answer. They optimized the surface question — short distance, walking is fine — and missed the function of a car wash. If I had asked only the popular one, I would have walked over to wait in line for a service that requires a car.

Only gemini caught the obvious thing: the car has to be there.

This is what the tournament model is for. It is not "three AIs are smarter than one." Two of them were less smart than one. The point is the divergence becomes visible, and the human is the one who picks. With a single AI, you never see the disagreement — you just inherit whichever bias that model happened to have.

The car wash is a toy example. Replace it with "should we go gRPC, NATS, or HTTP for service-to-service?" and the same pattern holds — except the cost of picking the confident-but-wrong answer is no longer a wasted afternoon.

The Five Principles

1. Use Multiple AIs — Don't Bind to One

Claude, Gemini, GPT, Codex — they're all advisors. Each has strengths. Each has blind spots. Using only one AI is like having only one advisor: you inherit all their biases.

One AI:     The model's bias becomes your bias
Three AIs:  Biases cancel out, blind spots get covered

I write content using three AI models simultaneously. Same task, three outputs. I don't ask them to divide the work — I ask them to compete. The best output wins. The others get discarded.

This is not "AI-assisted writing." This is a tournament where AI models compete and the human judges.

2. Compete, Don't Divide

Most people who use multiple AIs assign each one a role: "Claude for writing, GPT for coding, Gemini for research." That's division of labor. It's a planned economy.

The tournament model is a market economy: same task to all, let results determine who's best.

Why competition beats division:

Division relies on your judgment of which AI is better at what — and that judgment is constantly wrong as models update
Competition is self-correcting — if GPT suddenly gets better at writing, it starts winning writing tasks. No reconfiguration needed
You don't need to solve the impossible problem of "which AI is best" — let them prove it through results

3. The Human Decides — Judgment Is Not Outsourceable

AI can analyze. AI can generate options. AI can evaluate tradeoffs. What AI cannot do: decide which tradeoff matters in this specific context for this specific person with these specific constraints.

Three capabilities make human judgment irreplaceable:

Insight — Knowing what question to ask. AI can answer any question, but it can't know which question matters right now. Insight comes from understanding the problem deeply enough to ask the question that unlocks everything else.

Critical Thinking — Knowing when AI is wrong. AI gives confident, articulate answers regardless of accuracy. The human must evaluate: does this make sense? Is this consistent with what I know? Is there a blind spot?

Result Evaluation — Knowing if the outcome is good enough. AI can generate a technically correct solution that's wrong for your context. Only the human who understands the full picture — users, business constraints, team dynamics, market timing — can judge whether the output actually serves the goal.

These three form a loop:

Insight → Ask the right question
  ↓
AI gives analysis
  ↓
Critical Thinking → Is this analysis trustworthy?
  ↓
Choose and execute
  ↓
Result Evaluation → Did it work?
  ↓
Insight → Why did it work / not work? → Better questions next time

4. No Blind Faith, No Emotions — Results Are the Only Standard

Two temptations:

AI agrees with you → "See, I was right." (Confirmation bias)
AI disagrees with you → "AI doesn't understand my situation." (Emotional rejection)

The tournament model rejects both:

AI agrees with me    → Good, but does the result confirm it?
AI disagrees with me → Interesting. Let me verify before judging.
Made a choice        → Own the outcome. Right? Improve. Wrong? Learn. Never blame the AI.

Practice as the sole test of truth. Not who said it. Not how confident it sounded. Did it work?

5. Human Drives AI, Not the Other Way Around

AI is an amplifier. The question is: amplifying what?

No insight + good AI tools = efficiently producing mediocrity
Good insight + no AI tools = good ideas, slow execution
Good insight + tournament model = insight amplified 10x

The human provides:

Direction — what to work on (insight)
Quality standard — what "good" looks like (evaluation)
Context — the constraints AI doesn't see (judgment)
Accountability — willingness to own the outcome (leadership)

AI provides:

Speed — generate options fast
Breadth — consider more possibilities than a human can
Consistency — apply the same standard across large volumes
Knowledge — access more information than any person can hold

The human's role isn't to do AI's job slowly. It's to do the job AI can't do at all.

Applied to Real Work

Content Creation

The temptation: let AI generate content and publish automatically. Maximum output, minimum effort.

The result: a flood of mediocre, AI-flavored content. No differentiation. No personal perspective. Platforms and audiences both learn to ignore it.

The tournament approach:

Three AI models generate competing drafts on the same topic
The human evaluates: which captured the insight? Which missed the point?
The winning draft gets refined — the human adds what AI can't: personal experience, controversial opinion, industry context
Publication decision: is this good enough to attach my name to?

The output isn't "AI content." It's human content, produced at AI speed.

Technical Decisions

The temptation: ask one AI "should I use vector databases for agent memory?" and follow its recommendation.

The result: you inherit that model's training bias. Claude might favor simplicity (it was trained by Anthropic, who chose Markdown files). GPT might favor complexity (it's aligned with enterprise patterns).

The tournament approach:

Ask all three: "What are the tradeoffs between Markdown files, SQLite + vectors, and self-evolving skills for agent memory?"
Each gives a different analysis weighted by its own biases
The human evaluates against the actual constraints: deployment model, team size, user count, latency requirements
The decision accounts for context that no AI has — your specific situation

Career Strategy

The temptation: "AI will replace developers, I need to switch careers."

The reality: AI replaces tasks, not roles. The question is which tasks become your competitive advantage.

For employees:  Agent engineering skills (the 90% problem) — because companies 
                have data and scenarios, but need people who can build reliable agents

For founders:   Data + scenario moats — because agent engineering can be hired,
                but proprietary data and deep domain knowledge can't

In both cases, the competitive advantage is insight — understanding what matters in your specific domain well enough to direct AI effectively.

The Anti-Patterns

Anti-Pattern	Problem	Tournament Alternative
Only use one AI	Single advisor's bias = your bias	Multiple AIs competing
Follow AI blindly	Lose judgment over time	AI advises, human decides
Reject AI when it disagrees	Miss good ideas out of ego	No emotions, evaluate by results
Automate everything	No quality control, garbage output	Human at quality gates
Treat AI as just a tool	Waste AI's analytical capability	Treat AIs as competing advisors

The Test

Here's how to know if you're using AI well:

Bad sign: You can't explain why you chose AI's suggestion over the alternatives.
Good sign: You can articulate the tradeoff — what you gained and what you gave up.

Bad sign: You use the same AI for everything.
Good sign: You use different AIs for the same task and pick the best output.

Bad sign: You haven't disagreed with AI in the past week.
Good sign: You regularly override AI when your insight says otherwise — and you're right more than you're wrong.

Bad sign: You can't tell the difference between AI output and human output.
Good sign: You use AI for speed and breadth, then add what only you can: context, judgment, and accountability.

One Sentence

In the AI age, run AIs like a tournament: many compete, you decide, results judge everyone. Your insight is the one thing that scales with AI instead of being replaced by it.

Part of the AI Agent Architecture series. For the technical deep dive behind these ideas: The 90% Problem and Claude Code Deep Dive.

Node Turns Waiting Into Events. Go Moves Context Switching Into User Space.

Harrison Guo — Tue, 28 Apr 2026 18:01:12 +0000

Most discussions of TypeScript/Node vs Go concurrency stop at the surface: Node is async, Go is threaded. That framing isn't wrong — it just isn't deep enough to be useful when you're picking a runtime, debugging a tail-latency problem, or explaining to your team why one of the services keeps falling over under CPU load.

The real difference is not async vs threaded. It's a question about where, in the system, suspended work lives — and what shape it takes when it's resumed.

tl;dr — Both Node and Go refuse to let the CPU sit idle while a request waits on I/O. They disagree on the unit of scheduling. Node's unit is the continuation — the tail of an async function captured as a heap closure. Go's unit is the goroutine — a full call stack the runtime can suspend and resume in user space. That single decision cascades into every other property of each runtime.

The Wrong Question

"Async vs threaded" is the wrong frame because it makes you think the choice is between paradigms. It isn't. Both runtimes have already made the same fundamental decision: do not block an OS thread waiting for slow external work. The interesting choice is how they implement that.

The actually useful question is:

When a request is waiting for I/O — for a database, an HTTP call, a Redis round-trip, a file read — what does the CPU do, and where does the suspended state of that request live?

Once you frame it that way, Node and Go aren't opposites. They're two answers to the same question — and each answer cascades into a different language shape, a different library style, and a different failure mode under load.

The naive blocking model answers the question with "an OS thread waits for the syscall to return." That model collapses around a few thousand concurrent connections — memory per thread, scheduler overhead, kernel context-switch cost. By 40,000 connections you're out of RAM, not CPU. Node and Go both refuse to do this. They diverge on which resource gets freed up and how the suspended work is captured for later resumption.

Node's Answer: Turn Waiting Into an Event

Node's model can be summarized in one line: the JS main thread only executes code that's already ready to run.

Look at this:

const user = await db.getUser(id);
return user;

It reads as if the function is paused, blocking on the database. It isn't. Here's what V8 actually does at the bytecode level when it compiles an async function: it rewrites the body into a state machine, with each await becoming a state transition.

The function above gets transformed into something equivalent to:

function asyncFn() {
  const promise = new Promise((resolve) => {
    let state = 0;
    const closure = {};                  // heap object holding locals

    function step(value) {
      switch (state) {
        case 0:
          state = 1;
          db.getUser(id).then(step);     // await → register continuation
          return;                         // ← function POPS here
        case 1:
          closure.user = value;           // resume: locals live in closure
          resolve(closure.user);
          return;
      }
    }
    step();
  });
  return promise;
}

Three things to notice:

await is not a pause. It's the point at which V8 returns from the function and pops the JS stack frame. The "rest of the function" is captured as a continuation registered on the awaited Promise via .then.
Local variables move to the heap. Because the stack frame is gone, locals (user here) live in a heap closure, accessible only when the state machine resumes.
Each await slices the function into another state. A function with two awaits runs in three event-loop turns, with three independently-pushed JS frames, with all live state stored in heap closures between them.

That third point is the most non-obvious. A single async function is not one unit of execution — it's a sequence of fresh frames separated by event-loop turns:

There is no "paused" function. There are only captured continuations and fresh frames that resume them. The event loop is the dispatcher: it watches for I/O readiness via libuv, for resolved Promises (via V8's microtask queue), for timers — and pulls the corresponding continuation onto the JS thread when it's ready to run. One thread can manage tens of thousands of concurrent connections, because at any moment only a handful of them have work that's actually ready.

This is event-driven concurrency in its precise sense — the runtime turns "waiting" into a registered event, and only resumes the captured continuation when the event fires.

The Visible Side Effect: Function Color

Because the suspension point has to be marked at compile time, async-ness becomes part of the function's type. A function that does I/O returns Promise<T>. Its callers must await it. Once they await, they themselves return Promise<T>. The "color" propagates up the call stack until you hit an async-aware entry point — typically the top of an HTTP handler or the event loop itself.

Bob Nystrom named this the function color problem in 2015. It's not a notation choice — it's a logical consequence of the stackless coroutine model. V8 cannot save and restore arbitrary JS call stacks. The only way to express suspension is "return a Promise and be marked async," and once one function does that, every function on the way up has to do the same.

The Hard Limit

The model fails the moment your code stops waiting. A single CPU-bound operation:

while (true) { /* heavy work */ }

…holds the JS main thread, and every other request on this process is dead until it returns. The event loop has nowhere else to go. Worker threads, child processes, or splitting CPU work into a separate service are real fixes, but they're escape hatches — they exist because the core model has only one main thread executing JS, and there is exactly one of it.

Go's Answer: Move Context Switching Into User Space

Go writes synchronous code:

user := db.GetUser(id)
sendResponse(user)

There is no await. There is no callback. The function looks like it blocks on the database. And yet the program scales to hundreds of thousands of concurrent operations on modest hardware.

The trick is that the scheduling boundary has been moved. Where Node has the programmer mark the suspension point with await and the runtime captures a continuation, Go lets the programmer write straight-line code and has the runtime suspend the entire goroutine when it hits a blocking I/O call.

This is the central insight, and the cleanest one-line statement of Go's concurrency model:

Go's essence is the user-space-ification of context switching.

A goroutine isn't an OS thread. It's a small (initially 2 KB) growable stack and a register snapshot, managed by the Go runtime. The runtime maps a large number of goroutines (G) onto a small number of OS threads (M) using scheduling contexts (P). This is the GMP model:

G — a goroutine. The unit of scheduling. Cheap to create, cheap to suspend.
M — an OS thread. Usually only GOMAXPROCS of them.
P — a scheduling context. Decides which G runs on which M.

many G  →  Go scheduler  →  few M  →  CPU cores

When a goroutine hits a blocking syscall or a channel wait, the Go runtime suspends the goroutine — saves its stack and registers — detaches it from the current M, and schedules another runnable goroutine onto that M. When the original goroutine's wait completes, it's marked runnable again, and some M eventually picks it up and resumes execution from the suspension point. None of this enters the kernel. No clone(2), no kernel-mediated thread switch, no kernel scheduler queue. The bookkeeping is all in user space.

That's the user-space-ification. The CPU still has to switch contexts when work shifts between goroutines, but the cost is roughly a function call plus a stack swap — not a kernel-mediated thread switch.

The key contrast with Node's model is in where the suspended state lives:

In Node, the JS call stack is shared and almost always near-empty — every async function in flight has already popped, with its state sitting in a heap closure. In Go, every goroutine owns its full call chain on its own heap-allocated stack; suspended goroutines look like frozen frames waiting for the runtime to resume them on some OS thread.

This is also why neither language can simply borrow the other's model. Node runs on V8, which was designed in 2008 for browser JS — single call stack, synchronous semantics, no concept of saving stacks across yields. Adding stackful coroutines would mean rewriting the engine, which is roughly what Java's Project Loom did to the JVM at huge cost. Go was designed from scratch with a runtime that owns stacks, can grow them, and can save them. The choice is locked in by runtime architecture, not language taste.

What "User-Space" Actually Buys You

The slogan only matters if user-space context switching is meaningfully cheaper than the kernel-mediated kind. It is — by more than an order of magnitude.

Two goroutines pinned to one OS thread (GOMAXPROCS=1), ping-ponging via runtime.Gosched() and via an unbuffered channel. Two pthreads pinned to one core (taskset -c 0), ping-ponging via pthread_mutex + pthread_cond. (Reproduction code at the end of the post.)

Measured on Intel N100, Ubuntu 24.04 (kernel 6.8.0), Go 1.23.4, gcc 13.3:

Operation	ns / switch
Goroutine yield (`runtime.Gosched`, GOMAXPROCS=1)	~102 ns
Goroutine round-trip via unbuffered channel	~436 ns (≈218 ns per G-switch + channel coordination)
pthread switch (mutex+cond ping-pong, single core)	~2,900 ns (range 2,818–3,611 across 5 runs of 2M iterations)

Ratio: roughly 28× cheaper for the bare scheduler yield, ~13× cheaper for the apples-to-apples synchronized round-trip.

Where the gap comes from:

Mode switch. The user → kernel → user round-trip alone is ~100 ns of entry/exit and ABI-mandated register save/restore. A goroutine switch never crosses that line.
Scheduler work in kernel space. Linux CFS maintains a red-black tree of runnable threads with locked, cross-CPU runqueues. The Go scheduler does the same job in user space with per-P local runqueues and lock-free fast paths — and skips the kernel locks entirely.
Cache and TLB effects. A kernel scheduler may migrate a thread to a different core, costing you cold L1/L2 and an instruction-cache reload. Goroutines normally stay on the same M, so the cache stays warm.

What the model does not buy you: a goroutine that makes a real blocking syscall still pays for a real OS thread switch — the runtime detaches the G from its M and may spin up another M so the rest of the goroutines keep running. Async preemption (Go 1.14+, signal-based) is the runtime's answer to tight loops that never yield, and it has its own cost. Once you saturate GOMAXPROCS, the user-space runqueue itself starts to show up in profiles.

The "user-space-ification" buys you cheap G-to-G switching on a hot M. That's where the order-of-magnitude lives. The syscalls, the M-to-M handoffs, the actual kernel work — those are still as expensive as they always were. The model wins by making the common case — many concurrent goroutines, mostly waiting, occasionally running — almost free.

(N100 is a low-power Alder Lake-N E-core; absolute numbers will be smaller on a server-class Xeon or EPYC, but the ratio is expected to hold.)

The Unit of Scheduling

The cleanest comparison is to ask what each runtime actually schedules:

	Node / TypeScript	Go
Unit of scheduling	callback / Promise continuation	goroutine
What's captured at suspension	tail of an async function as a heap closure	full call stack + registers
How code looks	explicit `async`/`await`	straight-line synchronous
Suspension marked by	the programmer (`await`)	the runtime (any blocking op)
Suspended state lives in	V8 microtask queue + heap closure	goroutine stack on the user-space heap
Kernel involvement	epoll/kqueue/IOCP via libuv	epoll/kqueue/IOCP via netpoller
CPU parallelism	one main JS thread; needs workers/cluster for cores	M:N scheduler runs goroutines across cores natively
Function color	yes (Promise infects up the call stack)	no (any function may block)
What breaks under CPU load	the entire event loop	nothing — scheduler runs another G on another M

The two columns describe deeply different mental models, but they belong to the same family. They are both user-space concurrency runtimes that avoid kernel thread-per-request. They differ in where the suspension is captured (the language vs. the call stack) and how broad the scheduler's mandate is.

Where the Boundaries Diverge: CPU-Bound Work

Node and Go look interchangeable on I/O-bound workloads. They diverge sharply the moment CPU work enters the picture.

Node's event loop has one job: dispatch ready callbacks onto a single JS thread. If a callback runs for 200 ms doing JSON parsing or hashing, the loop is frozen for those 200 ms. Every other suspended continuation has to wait. Throughput collapses.

Go's runtime has a different mandate. It doesn't only manage waiting — it also manages execution. If you spawn:

go task1()
go task2()
go task3()

…the scheduler is happy to put each goroutine on a different M, run them on different cores in true parallel, and preempt long-running goroutines so they don't starve the rest of the runtime. CPU-bound goroutines aren't a special case to work around. They're just goroutines.

That's why Go's concurrency model covers more ground:

Node's model mainly solves non-CPU-bound concurrency — network I/O, database waits, downstream API calls. Go's model solves I/O waiting and CPU parallelism with the same primitive.

This isn't a knock on Node. The event loop is brilliant at what it's designed for: lots of slow waits, light per-request CPU. It's the natural shape of API gateways, BFFs, websocket hubs, real-time aggregation, and most of the JSON-shuffling that makes up modern web backends. But sustained CPU work, mixed CPU + I/O pipelines, long-lived infrastructure services — those are workloads where Go's scheduler-driven model has more headroom built in.

Two Answers to the Same Question

Strip away the implementation details and the two runtimes are answering the same question with different abstractions:

Concurrency at scale is the problem of what to do with the CPU while a request waits on I/O.

Node's answer: turn the wait into an event, capture the rest of the function as a continuation, resume the continuation when the event fires. One thread cycling through ready continuations.

Go's answer: run the request on a goroutine, suspend the goroutine in user space when it blocks, schedule another runnable goroutine onto the OS thread, resume the original when its wait completes.

Two ways of solving the same waste. One state-machines it. The other lowers the cost of context switching far enough that you can afford to keep one execution flow per request.

Two answers to one question: one is events, implemented as a state machine. The other is low-cost user-space context switching.

But there's a deeper layer worth surfacing. The two answers also disagree about whether suspension should be visible in the type system. Node says yes — Promise<T> is part of the signature, async is part of the contract, function color propagates. Go says no — any function may block, and the type doesn't carry that information.

This visibility-vs-uniformity trade-off shows up far beyond Node and Go. It's the same shape as monadic IO vs implicit IO in Haskell, checked vs unchecked exceptions in Java, capability-based security vs ambient authority. Each pair makes the same trade: composable static reasoning vs ergonomic uniform code. Node and Go are picking sides of a much bigger question.

You see the consequence in the libraries. Node libraries publish fs.readFile and fs.readFileSync, two retry helpers (one for sync ops, one for async), p-limit-style bounded-concurrency wrappers around Promise.all. Go libraries publish os.ReadFile (one function), one Retry(op func() error, n int) error, twenty lines of chan + WaitGroup for bounded concurrency. The Go versions aren't simpler because Go developers are smarter — they're simpler because the runtime hides the same complexity that Node's type system insists on exposing.

The Closing Line

If you remember one thing from this:

Node turns waiting into events. Go turns execution flows into schedulable units. Both refuse to let the CPU sit idle while I/O blocks — they just disagree on what the unit of scheduling should be.

Or, if you want the deeper layer:

Node makes "this function might suspend" visible at the type level. Go makes it invisible.

That's the whole story. Everything else — await vs go, libuv vs the netpoller, V8's microtask queue vs GMP, single-thread bottleneck vs CPU-bound resilience, libraries that look complicated vs libraries that look simple — falls out of that one disagreement.

Appendix: Reproduce the Benchmark

goroutine_switch_test.go — GOMAXPROCS=1 go test -bench=. -benchtime=5s -count=5:

package bench

import (
    "runtime"
    "sync"
    "testing"
)

// Channel ping-pong: each iter is a full round-trip = 2 G-switches.
func BenchmarkGoroutineSwitchChannel(b *testing.B) {
    ch := make(chan struct{})
    done := make(chan struct{})
    go func() {
        for {
            select {
            case <-done:
                return
            case <-ch:
                ch <- struct{}{}
            }
        }
    }()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        ch <- struct{}{}
        <-ch
    }
    b.StopTimer()
    close(done)
}

// Bare scheduler yield. Each iter ≈ 1 G-switch.
func BenchmarkGoroutineSwitchGosched(b *testing.B) {
    var wg sync.WaitGroup
    wg.Add(1)
    half := b.N / 2
    go func() {
        for i := 0; i < half; i++ {
            runtime.Gosched()
        }
        wg.Done()
    }()
    b.ResetTimer()
    for i := 0; i < half; i++ {
        runtime.Gosched()
    }
    wg.Wait()
}

pthread_switch.c — gcc -O2 -o pthread_switch pthread_switch.c -lpthread && taskset -c 0 ./pthread_switch 2000000:

#define _GNU_SOURCE
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <stdint.h>

static pthread_mutex_t mu  = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t  cv  = PTHREAD_COND_INITIALIZER;
static volatile int    turn = 0;
static long            iters;

static void *worker(void *arg) {
    int my_turn = (int)(intptr_t)arg;
    pthread_mutex_lock(&mu);
    for (long i = 0; i < iters; i++) {
        while (turn != my_turn) pthread_cond_wait(&cv, &mu);
        turn = 1 - my_turn;
        pthread_cond_broadcast(&cv);
    }
    pthread_mutex_unlock(&mu);
    return NULL;
}

static double now_ns(void) {
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return (double)ts.tv_sec * 1e9 + (double)ts.tv_nsec;
}

int main(int argc, char **argv) {
    iters = (argc > 1) ? atol(argv[1]) : 1000000L;
    pthread_t t0, t1;
    double start = now_ns();
    pthread_create(&t0, NULL, worker, (void *)(intptr_t)0);
    pthread_create(&t1, NULL, worker, (void *)(intptr_t)1);
    pthread_join(t0, NULL); pthread_join(t1, NULL);
    double end = now_ns();
    printf("ns / switch: %.1f\n", (end - start) / (2.0 * iters));
    return 0;
}

GOMAXPROCS=1 forces both goroutines onto the same M so we measure pure G-to-G switching, not cross-core migration. taskset -c 0 pins both pthreads to one CPU so they actually have to context-switch (otherwise they run in parallel on two cores and there is nothing to measure). Both benches do the simplest possible synchronized hand-off — no I/O, no real work — so what is left is the cost of the switch itself.

gRPC Interceptors in Production: Design Patterns That Survive Real Load

Harrison Guo — Mon, 20 Apr 2026 17:02:20 +0000

gRPC interceptors are the middleware pattern, specialized for gRPC. If you've written HTTP middleware before, the shape is familiar — a function that wraps a call, can observe or modify the request, pass to the next handler, then observe or modify the response. The difference: gRPC's type system makes the flavors (unary, server-stream, client-stream, bidi) explicit, and chain ordering matters more than most people realize.

Most online examples show a single toy interceptor. Production systems stack five to ten of them per service. Getting the composition right — ordering, concern separation, testability — is half of running a gRPC-based microservice well.

tl;dr — gRPC interceptors are middleware with more explicit types. Chain them outside-in: observability wraps everything, then throttling, then auth, then retry, then the actual service. Keep each interceptor focused on one concern; the moment an interceptor does two things you're writing coupled middleware. Stream interceptors are trickier than unary — don't copy-paste unary logic into stream without thinking. Test the chain composition with bufconn, not just each interceptor in isolation.

The Four Interceptor Types

gRPC has four interceptor signatures, two for client, two for server:

Unary server interceptor: wraps a single request → single response call.
Stream server interceptor: wraps streaming RPCs (server-stream, client-stream, bidi).
Unary client interceptor: wraps the client side of a unary call.
Stream client interceptor: wraps the client side of a streaming call.

Unary interceptors are easy. Stream interceptors are harder because you're wrapping a bidirectional wire, not a single call.

Example unary server interceptor:

func loggingInterceptor(ctx context.Context, req interface{},
    info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    start := time.Now()
    resp, err := handler(ctx, req)
    log.Printf("method=%s duration=%s err=%v", info.FullMethod, time.Since(start), err)
    return resp, err
}

s := grpc.NewServer(grpc.UnaryInterceptor(loggingInterceptor))

Straightforward. Now stack five of them.

Chaining and Order

Real services need multiple interceptors. gRPC's standard library gives you grpc.ChainUnaryInterceptor(...) (since 1.25), or you can use google.golang.org/grpc/interceptor helpers:

s := grpc.NewServer(
    grpc.ChainUnaryInterceptor(
        observabilityInterceptor,  // outermost
        rateLimitInterceptor,
        authInterceptor,
        validationInterceptor,
        businessLogicContextInterceptor, // innermost
    ),
)

Chain order matters enormously. Interceptors execute outside-in on the way to the handler, inside-out on the way back. Put the wrong interceptor outside the wrong one and you get bugs that are hard to debug.

Canonical order I use:

I1" width="1784" height="94">

Outside-in on the way to the handler, inside-out on the way back. Observability must wrap everything — so it sees every rejection, every rate-limit hit, every failed auth — otherwise you have operational blind spots. Details:

Observability (tracing + metrics + logging) — outermost. You want to see every request, including the ones that get rejected by later interceptors. If observability is inside auth, unauth'd attempts are invisible — a security-relevant blind spot.
Rate limiting / quota — before auth. Why? Because auth involves token verification (DB lookup, JWT parsing, external identity service), and you don't want unauthenticated requests to cost you CPU. Rate-limit first, authenticate second.
Auth (authentication + authorization) — before business logic. Reject unauthenticated/unauthorized requests early.
Validation (request shape, basic sanity) — before business logic. Catches malformed requests before they hit service code.
Retry / idempotency handling — closer to business. Only retry what actually made it through auth.
Request context enrichment (trace IDs, user metadata) — innermost. Populate context with validated data for the service to use.

Inverted order produces real bugs. I've seen auth outside observability (auth failures weren't logged). Retry outside rate limiter (a retry storm blew through the rate limit). Validation outside observability (validation failures invisible in metrics). Each one a real incident.

Keeping Interceptors Focused

The rule: one concern per interceptor. The moment you have an "auth-and-logging" interceptor, you're coupling concerns that should evolve separately.

Concretely:

Don't: single "observability" interceptor that does tracing, metrics, and logging in one function.
Do: three interceptors (tracingInterceptor, metricsInterceptor, loggingInterceptor), chained.

Cost: three function-call overheads instead of one. Marginal.

Benefit: you can swap tracing backends without touching logging. You can disable metrics in tests without disabling tracing. Each interceptor is testable in isolation.

This is the same argument for Unix pipes over monolithic commands. Composition beats monoliths.

Common Interceptor Recipes

Real interceptors I've written variants of many times:

Tracing (OpenTelemetry)

Use the otelgrpc integration from go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc. Don't write your own — the ecosystem is mature. Current idiomatic setup uses a StatsHandler, which hooks deeper than the interceptor chain and captures stream events correctly:

import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

s := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
    grpc.ChainUnaryInterceptor( /* your app interceptors */ ),
)

Older codebases still use otelgrpc.UnaryServerInterceptor() and otelgrpc.StreamServerInterceptor() — those are deprecated but still work. Migrate when convenient; don't rewrite in a panic.

Metrics

Prometheus histogram of request duration per method:

var (
    reqDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "grpc_server_request_duration_seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "code"},
    )
)

func metricsInterceptor(ctx context.Context, req interface{},
    info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    start := time.Now()
    resp, err := handler(ctx, req)
    code := status.Code(err).String()
    reqDuration.WithLabelValues(info.FullMethod, code).Observe(time.Since(start).Seconds())
    return resp, err
}

Note: cardinality of method is bounded (you know your service's methods). Cardinality of code is bounded (gRPC codes are a fixed enum). Don't add user-id or request-id as labels — that's cardinality-explosion territory.

Auth

Extract bearer token from metadata, verify, inject user context:

func authInterceptor(ctx context.Context, req interface{},
    info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    md, ok := metadata.FromIncomingContext(ctx)
    if !ok {
        return nil, status.Error(codes.Unauthenticated, "no metadata")
    }
    tokens := md.Get("authorization")
    if len(tokens) == 0 {
        return nil, status.Error(codes.Unauthenticated, "no auth token")
    }

    claims, err := verifyToken(tokens[0])
    if err != nil {
        return nil, status.Error(codes.Unauthenticated, "invalid token")
    }

    // Skip certain public methods
    if isPublic(info.FullMethod) {
        return handler(ctx, req)
    }

    ctx = context.WithValue(ctx, userCtxKey{}, claims)
    return handler(ctx, req)
}

Key detail: add the user context here, near the boundary. Service code reads it from context. You don't pass claims as argument through every service method.

Rate limiting

Token bucket per caller or per method:

func rateLimitInterceptor(limiter *rate.Limiter) grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req interface{},
        info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
        if !limiter.Allow() {
            return nil, status.Error(codes.ResourceExhausted, "rate limited")
        }
        return handler(ctx, req)
    }
}

Production rate limiting is fancier — per-tenant, distributed state in Redis, burst capacity — but the shape is the same. Reject with ResourceExhausted before doing work.

Retry (client-side)

Client interceptor that retries on transient errors:

func retryClientInterceptor(attempts int) grpc.UnaryClientInterceptor {
    return func(ctx context.Context, method string, req, reply interface{},
        cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
        var err error
        for i := 0; i < attempts; i++ {
            err = invoker(ctx, method, req, reply, cc, opts...)
            if err == nil {
                return nil
            }
            if !isRetryable(err) {
                return err
            }
            backoff := time.Duration(1<<uint(i)) * 100 * time.Millisecond
            select {
            case <-time.After(backoff):
            case <-ctx.Done():
                return ctx.Err()
            }
        }
        return err
    }
}

Retry is one of the most dangerous interceptors. Get it wrong (no idempotency keys, retry non-idempotent operations, retry storm during outage) and it causes more production incidents than it prevents. Pair with grpc-middleware/retry if you can; it's battle-tested.

The Stream Interceptor Trap

Stream interceptors are harder. The interceptor signature gives you a grpc.ServerStream, which is a bidirectional channel. Logging becomes:

func loggingStreamInterceptor(srv interface{}, ss grpc.ServerStream,
    info *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
    start := time.Now()
    err := handler(srv, ss)
    log.Printf("stream=%s duration=%s err=%v", info.FullMethod, time.Since(start), err)
    return err
}

This only logs at stream-end, not per message. If you want per-message observability, you need to wrap the ServerStream itself:

type observedStream struct {
    grpc.ServerStream
    sent, recv int64
}

func (s *observedStream) SendMsg(m interface{}) error {
    atomic.AddInt64(&s.sent, 1)
    return s.ServerStream.SendMsg(m)
}

func (s *observedStream) RecvMsg(m interface{}) error {
    err := s.ServerStream.RecvMsg(m)
    if err == nil {
        atomic.AddInt64(&s.recv, 1)
    }
    return err
}

Then pass the wrapper to the handler. This is the pattern for any stream interceptor that needs per-message visibility.

Common mistakes:

Forgetting to propagate context to the wrapper. The wrapped stream's Context() should be the enriched context.
Per-message overhead blows up long streams. A message-level log line is fine at 100 msgs/sec. At 100K msgs/sec, it's your dominant cost.
State in the wrapper not thread-safe. Streams can be concurrent on the Send and Recv sides. Protect counters.

Testing Interceptor Chains

Unit test each interceptor in isolation:

func TestAuthInterceptor_NoToken(t *testing.T) {
    ctx := context.Background() // no metadata
    info := &grpc.UnaryServerInfo{FullMethod: "/my.Service/Method"}
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        t.Fatal("handler should not be called")
        return nil, nil
    }

    _, err := authInterceptor(ctx, nil, info, handler)
    require.Equal(t, codes.Unauthenticated, status.Code(err))
}

Integration-test the chain end-to-end using bufconn (in-memory connection):

func TestChain_Ordering(t *testing.T) {
    lis := bufconn.Listen(1024 * 1024)
    defer lis.Close()

    s := grpc.NewServer(grpc.ChainUnaryInterceptor(observability, auth, business))
    pb.RegisterMyServer(s, &realImpl{})
    go s.Serve(lis)
    defer s.Stop()

    conn, _ := grpc.Dial("bufnet",
        grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
            return lis.DialContext(ctx)
        }),
        grpc.WithTransportCredentials(insecure.NewCredentials()),
    )
    defer conn.Close()

    client := pb.NewMyClient(conn)
    resp, err := client.Method(ctx, req)
    // assert on behavior end-to-end
}

Integration tests catch bugs that unit tests don't: metadata propagation, interceptor ordering, context enrichment visible to the handler. Don't skip them.

Patterns That Save Time

Use grpc-middleware/v2 (github.com/grpc-ecosystem/go-grpc-middleware/v2) for chain helpers, recovery, and batteries-included interceptors. Don't reinvent every wheel.
Keep error semantics consistent. Every interceptor should return status.Error(code, msg) for failures. Don't return raw Go errors — clients can't parse them properly.
Skip-list for public methods. Auth and rate limit often need to skip health check and reflection endpoints. Keep the skip list in one place.
Per-service vs global interceptors. Most interceptors are global (tracing, metrics, auth). A few might be per-service (e.g., a bespoke rate limiter for a specific hot endpoint). Compose accordingly.
Panic recovery at the outermost layer. A panic in a handler shouldn't kill the server. Use the recovery middleware from grpc-middleware or write your own, and put it first in the chain.

The Discipline That Makes This Work

Interceptors are the right tool for cross-cutting concerns — the things every RPC needs but the service code shouldn't have to think about. The discipline is: one concern per interceptor, careful ordering, consistent error semantics, tested end-to-end.

The services I've seen do this well have clean business logic (because the cross-cutting stuff is outside it) and reliable operational behavior (because the interceptor chain is tested as a unit, not just piece-by-piece). The services that do it poorly have auth logic sprinkled through their handlers, tracing that randomly misses requests, and rate limiters that let certain code paths bypass.

Interceptor order is one of those details that looks tactical and turns out to be architectural. Get it right once; the service's behavior improves every release.

Go Context in Distributed Systems: What Actually Works in Production — the context that flows through every interceptor.
RPC vs NATS: It's Not About Sync vs Async — It's About Who Owns Completion — the shape of gRPC calls as one side of the bigger messaging picture.
Observability and Cost Attribution: Why One Pipeline Isn't Enough — why tracing interceptors alone aren't enough for business attribution.

Go Generics, One Year In: Which Promises Held, Which Didn't

Harrison Guo — Mon, 20 Apr 2026 16:30:05 +0000

Go 1.18 shipped generics in March 2022. The two years before that were dominated by hopeful blog posts ("finally, a real type system!") and the two years after by the predictable backlash ("why did we even bother, Go was simpler"). I've written production Go before and after. The honest answer is somewhere in the middle and closer to "useful for a narrower set of problems than we expected."

This is a look back from someone who has shipped generic code in anger and reviewed a lot more of it. What held up. What didn't. What habits to adopt and which to avoid.

tl;dr — Go generics are genuinely valuable for parametric operations on container-shaped types — slices, maps, channels, any-key lookup tables, min/max/sum utilities. Less valuable for "clever abstractions" that dress up control flow as type magic. The clearest gains are in the standard library itself (slices, maps) and in domain-specific utility packages. Most application code didn't need generics before and doesn't need them after. The mistake is not using generics; it's using them for things interfaces already handled fine.

What Generics Actually Are

Go generics are type parameters on functions and types. A function like slices.Contains can be written once, work for any slice element type, and still be type-checked at compile time:

func Contains[S ~[]E, E comparable](s S, v E) bool {
    for _, x := range s {
        if x == v {
            return true
        }
    }
    return false
}

Three features you should know:

Type parameters: the [E any] or [E comparable] in brackets.
Constraints: tell the compiler what operations the type parameter supports. any, comparable, or custom interfaces like constraints.Ordered.
Approximate constraints: ~[]E means "any type whose underlying type is []E" — lets you be flexible about named slice types.

What they aren't: Java-style wildcards, C++ SFINAE, or anything that mimics variance. The design is deliberately narrower than most prior languages. It's more like Rust's generics, minus the trait system's complexity.

Where Generics Clearly Win

Standard-library style container and utility functions

The slices and maps packages in the standard library are the canonical example:

slices.Contains(users, "alice")
slices.Sort(numbers)
maps.Keys(config)
maps.Values(settings)

Before generics, these were either hand-written per-type (tedious, error-prone), done via interface{} (type-unsafe, slow), or done via reflect (slow and error-prone). Generics are strictly better for these.

The same pattern shows up in third-party libraries: samber/lo (JS-style utilities), thoas/go-funk (functional helpers), and many domain-specific ones. If you reach for lodash-style helpers in JavaScript, you'll want similar in Go, and generics made that workable.

Concurrency helpers

Generic worker pools, futures, result types — these all benefit from generics:

type Future[T any] struct {
    done chan struct{}
    val  T
    err  error
}

func (f *Future[T]) Get() (T, error) {
    <-f.done
    return f.val, f.err
}

Before generics, you'd have had an interface{} return and a type assertion at the call site. Now you can express "this future produces a T" in the type. Cleaner at the boundary, safer at the call site.

Typed collections

If your system has a genuinely typed container use case — say, an ordered map keyed by a domain ID — generics let you write it once:

type OrderedMap[K comparable, V any] struct {
    order []K
    data  map[K]V
}

This is a rare case where "custom generic container" is the right tool. The majority of code doesn't need this. But when you do need it, the generics version is much better than the interface{} alternative.

Numerical / algorithmic code

constraints.Ordered (or its post-1.21 replacement cmp.Ordered) is the key constraint for "works for any numeric or ordered type":

func Max[T cmp.Ordered](a, b T) T {
    if a > b { return a }
    return b
}

Math helpers, min/max, sum, average — all cleanly generic. Readable, type-safe, performant.

Where Generics Don't Help, Or Hurt

"Generic services" and similar framework-y code

I've seen codebases where someone wrote a generic "repository" type:

type Repository[T any] struct { /* ... */ }
func (r *Repository[T]) FindByID(id string) (T, error) { /* ... */ }
func (r *Repository[T]) Save(t T) error { /* ... */ }

The instinct — "all repositories do the same thing" — is mostly wrong. Real repositories differ in query shape, error cases, caching rules, transaction boundaries. Forcing them behind a generic interface either (a) produces a lowest-common-denominator API that doesn't fit any actual use, or (b) gets so many type parameters that readability collapses.

The Go idiom is usually better: one non-generic UserRepository, one OrderRepository, etc. Each concrete, each tuned to its domain.

Over-constrained helpers

If your "generic" function has five type parameters with custom constraints each, readability dies:

func Complicated[
    T comparable,
    K Hashable,
    V any,
    F func(T, K) (V, error),
    M map[K]V,
](items []T, f F, cache M) error { /* ... */ }

This is technically legal. Reading it, you realize it's a glorified map-with-cache-and-error. Interfaces or function types would have been clearer. Generics don't make complex APIs simple; they just let you make them complex in a type-checked way.

Behavioral polymorphism

Interfaces are still the right tool when different types have different behavior. A generic Process[T any](x T) error doesn't help if you actually want different logic per type. You want an interface with a method.

// Good use of interface
type Processor interface {
    Process(ctx context.Context) error
}

// Bad use of generics
func ProcessGeneric[T any](x T) error {
    // can't actually differentiate behavior
}

The separation: generics for parametric operations (same logic, any type), interfaces for polymorphic behavior (different logic per type).

Performance: Usually a Wash

The performance story is more nuanced than either "generics are slow" or "generics are free."

Go's current generic implementation uses GCShape stenciling — one compiled version per "GC shape" (roughly, per memory layout). This is between full monomorphization (one version per type, like Rust) and type-erased dispatch (one version total, like Java's reified-erased hybrid).

Practical implications:

Small primitive types (int, int64) often get specialized versions. Competitive with hand-written.
Pointer-sized types (most structs, interfaces) share code. Slightly slower than hand-written but usually faster than interface-based dispatch.
Call overhead is similar to function calls, not interface dispatch. No devirtualization issue.
Compile times increase, especially for libraries with many instantiations. This is the real cost.

Benchmarks I've seen: generic versions are within 5% of hand-written equivalents, and consistently faster than interface{}-based alternatives. Performance is almost never the deciding factor — readability and design fit matter more.

Idioms That Emerged

Over the years since 1.18, a few conventions have stuck:

Prefer `any` to `interface{}`

any is a type alias for interface{} added in 1.18. Shorter, clearer. Use it everywhere.

Single-letter type parameters for simple cases, descriptive for complex

T, K, V for the obvious cases. More descriptive when the role is specific:

func Reduce[In any, Out any](items []In, f func(Out, In) Out, initial Out) Out

Put constraints in a dedicated package

If you have several custom constraints, group them:

package constraints

type Ordered interface { ~int | ~int64 | ~string | ~float64 }
type Numeric interface { ~int | ~int64 | ~float64 }

The standard golang.org/x/exp/constraints (and later cmp.Ordered in 1.21) set the pattern.

Use `~T` approximations for flexibility

~[]E includes named slice types. ~int includes type MyInt int. Almost always the right choice for generic parametric code; refuses arbitrary extension.

Never overload generic helpers to do too much

Each generic function should do one parametric thing. Generic helpers that try to be many things at once collapse under type-parameter weight.

The Standard Library Won

The clearest vindication of Go generics is what happened to the standard library. slices, maps, cmp.Ordered — these additions are uncontroversially better than the pre-1.18 alternatives. A lot of code that used to be hand-rolled or based on sort.Interface has cleaner replacements.

The user-land picture is more mixed. Libraries that benefit from generics genuinely use them well (samber/lo, kelindar/column, many others). Libraries that don't need them mostly haven't been retrofitted with them.

What I Do Now

A few simple rules I apply:

Prefer standard library generic helpers over hand-rolled. slices.Contains, slices.Sort, maps.Keys — use them.
Write a generic helper only when I have at least two concrete use cases for it. One use case is a pattern waiting to be born, not necessarily a generic.
Prefer functions to methods on generic types when possible. Generic methods have more friction (can't overload by type, can't add methods outside the defining package).
Keep constraints simple. any, comparable, cmp.Ordered, and domain-specific single-type-union constraints cover 95% of cases. More complex constraints usually mean the abstraction is wrong.
Never turn interfaces into generics just because you can. If the types have genuinely different behavior, an interface is right.

Where Generics Actually Sit Now

Generics were oversold before they landed ("Go finally becomes a real language!") and oversampled in the aftermath ("generics everywhere!"). The truth is narrower and more boring: they're a useful addition for a specific class of problems, mostly centered on parametric operations over containers and numerics. They improved the standard library. They haven't changed the shape of most Go code.

If you've been writing Go and wondering whether you're missing out by not using generics, the answer is almost certainly no. Code without them is still idiomatic. Code with them, when the use case fits, is cleaner. Neither is dominant. Both are fine.

The one concrete thing I'd say: learn the generic parts of the standard library. slices, maps, cmp.Ordered. Use them reflexively. Stop hand-rolling indexOf and contains. Everything else can wait until you have a real problem that generics solve.

Go Profiling in Anger: pprof, Escape Analysis, and Inlining Without Magic — the performance toolchain that tells you whether your generic code actually matches the hand-written version.
sync.Pool in Go: When It Actually Helps, and When It Quietly Hurts — another feature most commonly misapplied.
Scale-Up vs Scale-Out: Why Every Language Wins Somewhere — the meta-question behind every language-feature debate.

Go Profiling in Anger: pprof, Escape Analysis, and Inlining Without Magic

Harrison Guo — Mon, 20 Apr 2026 16:29:23 +0000

Go's performance culture has a ritual quality. "Use sync.Pool." "Avoid interface boxing." "Preallocate slices." Copy-pasted from blog posts and applied without measurement. Sometimes helpful. Often hollow.

The honest answer is that Go performance work is mostly just profiling. Good profiling tells you what's actually slow. Bad profiling — or no profiling — leaves you guessing. The toolchain that Go ships with is genuinely excellent; more engineers should use it, and fewer should follow checklist optimizations they haven't measured.

This is a practical, end-to-end guide to pprof, escape analysis, and inlining — the three Go-specific tools that answer most performance questions.

tl;dr — Start every Go perf investigation with a CPU pprof of the hot path under realistic load. 80% of issues are obvious in the flame graph. For the remaining 20%, add a heap profile and look for allocation pressure driving GC. Only after you've localized the problem with real data should you reach for micro-optimizations: escape analysis via -gcflags='-m', inlining hints, and targeted benchmark-driven rewrites. Skip the profile step, and you are optimizing the wrong thing.

The Investigation Flow

CPU[Take CPU profile
-http pprof · 30s under load]" width="803" height="1086">

CPU Profiling: The First Thing, Always

Every Go binary can expose a pprof HTTP endpoint in two lines:

import _ "net/http/pprof"
// later
go http.ListenAndServe("localhost:6060", nil)

Under load, grab a CPU profile:

$ go tool pprof -http=:9999 http://localhost:6060/debug/pprof/profile?seconds=30

This opens a flame graph in your browser. The wide blocks are where CPU time is spent. Usually the answer is immediate — "oh, JSON encoding is 40% of my CPU; let me switch to a faster encoder." Or "regex compilation is in the hot path because someone forgot to pre-compile."

A few things that look surprising on first profile but shouldn't:

runtime.mallocgc taking 10%+ is GC pressure. You're allocating a lot. Look at heap profile next.
runtime.schedule or runtime.findrunnable taking 5%+ means you have too many goroutines churning. Check if you're spawning per-request.
syscall.Syscall high means you're system-call-heavy — usually I/O. Either buffer/batch, or consider epoll-direct if it's in your hot path.
mutex.Lock visible means contention. Either shrink the lock hold time or shard the lock.

Don't guess your way through these. Click into each, read the stack, find the user code that caused it.

Heap Profiling: When CPU Points to GC

If runtime.mallocgc shows up in your CPU profile as a non-trivial chunk, heap profile tells you why:

$ go tool pprof -http=:9999 http://localhost:6060/debug/pprof/heap
$ go tool pprof -http=:9999 http://localhost:6060/debug/pprof/allocs

heap shows current memory usage. allocs shows cumulative allocations since program start — this is usually what you want to optimize.

In the flame graph, look for:

Specific allocation sites taking disproportionate share. A single line of code creating 50% of allocations is an obvious target.
Calls to makeslice, makemap, newobject with known-size inputs. If you know the size, preallocate.
Interface boxing in hot paths. Every time you pass a concrete type through an interface{} argument in a tight loop, the runtime may heap-allocate the boxed value.
String concatenation with +. This is the textbook preventable allocation — use strings.Builder.

The goal isn't "zero allocations" — that's usually not practical. The goal is "allocations per operation in a tight, repeated path are bounded and understood."

Escape Analysis: The Compiler's Story

Go's compiler decides at compile time whether a variable lives on the stack (free, garbage-collected with the function) or the heap (allocated, GC-tracked). This is called escape analysis.

To see the analysis for your code:

$ go build -gcflags='-m' ./...

Output looks like:

./foo.go:12:6: can inline hotFunction
./foo.go:15:10: &Thing{} escapes to heap
./foo.go:18:14: make([]int, 100) does not escape
./foo.go:22:6: parameter "x" escapes to heap

Key things to read for:

escapes to heap — this allocation is heap-allocated. If it's in a hot path, investigate.
does not escape — stack-allocated, free. You want most short-lived locals to do this.
parameter escapes to heap — the caller's passed value escapes because this function keeps a reference to it. Often fixable by taking a copy or not storing a reference.

The most common surprise: passing a value to a function that eventually hands it to interface{} causes the value to escape. A pattern like:

func log(msg string, args ...interface{}) {...}
func handleRequest(req *Request) {
    log("got request", req.ID) // req.ID boxes to interface{} and may escape
}

req.ID escapes because of the ...interface{} argument. In a tight path, this is measurable. Fix: use a typed logger that takes concrete types, or accept the cost because logging on the hot path is usually not the hot path.

Escape analysis is one of those things where reading the output a few times is worth it. You start seeing your code differently.

Inlining: When the Compiler Eliminates the Call

Go's compiler inlines small functions to avoid call overhead. Seeing what got inlined:

$ go build -gcflags='-m' ./... 2>&1 | grep -E 'can inline|cannot inline'

./foo.go:12:6: can inline hotFunction
./foo.go:18:6: cannot inline bigFunction: function too complex: cost 117 exceeds budget 80
./foo.go:22:6: cannot inline interfacingFunction: call to unknown method

Default budget is 80 AST nodes. Hard blockers:

Calls through interfaces. The compiler doesn't know what concrete method gets called. No inlining.
Calls to functions that contain loops with for range over a channel. Historically blocked, though the mid-stack inliner has improved this.
Recursive functions. Obvious.
Functions over the budget. Refactor smaller if the call is hot.

When to care:

Never in normal code. Go inlines what it can; your code runs.
Sometimes in tight hot loops where the call overhead is 10%+ of the total work. Benchmark shows it.
Occasionally when you control an interface boundary and can replace it with a concrete type on a hot path.

Don't structure your code around inlining. Code readability beats hypothetical call-overhead wins in nearly every case.

Benchmarks: The Ground Truth

Every perf claim should be backed by a benchmark. testing.B is the tool:

func BenchmarkEncodeResponse(b *testing.B) {
    resp := newResponse()
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _, _ = encode(resp)
    }
}

Run:

$ go test -bench=BenchmarkEncode -benchmem -count=5

-count=5 runs each bench 5 times, so you can compare variance. Don't trust a single run. Hardware, OS scheduling, thermals — all add noise.

For comparing two implementations:

$ go test -bench=BenchmarkEncodeResponse -benchmem -count=10 ./... > old.txt
# (change code)
$ go test -bench=BenchmarkEncodeResponse -benchmem -count=10 ./... > new.txt
$ benchstat old.txt new.txt

benchstat (golang.org/x/perf/cmd/benchstat) gives you statistical significance. If the difference isn't statistically meaningful, you didn't actually improve anything — you just rolled the dice differently.

The 80/20 of Go Performance

After enough of this work, a few patterns dominate the real wins:

Query shape, not language. A slow endpoint is usually doing 10 DB queries when it could do 1. Go is almost never the bottleneck; the data layer is.
Network hop count. Every inter-service call adds latency. Merging two small services or co-locating tight integrations beats any language-level optimization.
Caching at the right layer. A well-placed LRU cache saves more than micro-optimizing the uncached path.
Preallocating known-size slices/maps. make([]int, 0, n) when you know n is almost free. The default make([]int, 0) reallocates as you append.
Avoiding interface boxing in loops. This is the one micro-optimization that regularly shows up in real profiles.

Everything else — sync.Pool, escape analysis hand-tuning, loop unrolling — is a long-tail optimization. Worth it when profiling tells you it is. Premature otherwise.

A Habit I Recommend

Before adding any optimization, do exactly three things:

Take a profile with the optimization off. Save it.
Apply the optimization.
Take a profile with the optimization on. Compare.

If the comparison doesn't show clear improvement on the metric you cared about, revert. Do not add complexity without evidence.

This sounds obvious. Almost nobody does it. Most perf work in Go codebases accumulates dead optimizations that add nothing or actively hurt — but nobody knows which, because nobody benchmarked.

The Habit That Compounds

Go's performance tooling is better than Go's performance culture gives it credit for. pprof, escape analysis, inlining diagnostics, and benchmarks are built in. They're precise. They tell you the truth.

The reason most Go code isn't as fast as it could be isn't that Go is slow (it isn't). It's that engineers copy-paste optimizations they haven't measured, call the work done, and move on. The few engineers who profile first and optimize second write code that's actually fast — and usually simpler than the ritual-heavy version.

Profile first. Everything else follows.

sync.Pool in Go: When It Actually Helps, and When It Quietly Hurts — the one Go optimization most likely to be misapplied.
Why Go Handles Millions of Connections: User-Space Context Switching, Explained — understanding the runtime is the prerequisite to understanding profiles.
Testing Real-World Go Backends Isn't What Many People Think — benchmarking is the last mile of testing.

DEV Community: Harrison Guo

Agent Retrieval Above the Crossover: A First-Principles Read of CodeGraph

Why this is a first-principles question, not a tool review

Recap: the six conditions, in 30 seconds

Section 1 — The AST extraction boundary: an information-theoretic case

The information-theoretic case

The boundary is a literal table

Why this matters beyond CodeGraph

Section 2 — SQLite + FTS5 vs vector DB: the cost curve, recursed

The cost-curve frame, recursed

The recursion as a design principle

Section 3 — Where CodeGraph's abstraction leaks

The honest part: the provenance column

The part the schema can't tell you about

The empirical check, and the null result that sharpens it

Why mapping the leaks matters

Mapping CodeGraph to the six conditions

What this says about LLM retrieval as a discipline

Closing — the mini-series arc

I Tested CodeGraph on Hono. The Tool-Call Savings Reproduce — the Cost Savings Don't.

What "tool calls down, cost flat" actually means

Setup — install CodeGraph, ~10 minutes

Verifying the tool actually ran (this is not optional)

The 5 questions

The data

Cost / tokens

Tool calls / wall latency

The variance story — CodeGraph bounds the worst case

Per-question narrative

Cross-validation with CodeGraph's published 7-repo benchmark

Decision matrix — install CodeGraph when

What I'd want to test next

One-line verdict

Appendix: Benchmark Questions

Q1 — Route resolution

Q2 — Middleware chain trace

Q3 — Cross-file abstraction navigation

Q4 — Refactor impact

Q5 — Text search (control)

Scoring & environment

Appendix: Why there's no third arm (knowing)

Agent Memory Is a Cache Coherence Problem

The Two Axes (Don't Collapse Them Into One)

The Baseline: Lossless + Exact (CLAUDE.md + built-in memory)

The Contender: Lossy + Approximate (claude-mem)

The Test (Real Numbers, 2026-05-20)

The Sharpest Failure: Compression Flattens Modality

Six Measured Findings, Versioned to v13.2.0

Why This Is a Cache Problem, Precisely

When Each Wins

A Decision Framework

What a Coherent Agent Memory System Would Need

Closing — "Persistent Memory" Is a Trade, Not a Feature

Appendix: How I Measured This

Agent Retrieval Is a Cost Curve Problem: Why Claude Code Doesn't Use RAG

The Frame That Matters: Total Cost Over Time

The Popular Answer, Charitably

So Why Hasn't Symbol-Graph Become the LLM-Companion Either?

The Three Primitives, Audited

Grep — ripgrep, with structured output and a "don't shell out" enforcement clause

Glob — filename patterns with a recency heuristic and a hard cap

Read — bounded, fresh, stat-checked

The composition

The Loop, Sketched Honestly

The Subagent Twist Nobody Quotes Correctly

The Economics of Explore, in Detail

Fork — The Architecture Under Test

When RAG Still Wins

1. Mega-monorepos where any kind of index is amortized across millions of queries

2. Pure semantic queries where there's no symbol to grep for

3. When LLM context is the scarce resource — cheap, short-context model regime

A Decision Framework

The Companion Question

Channels Aren't Message Passing — How Parked Goroutines OOM-Killed a Pod

The Mental Model That Killed The Pod

What A Channel Actually Is

The Incident, Mechanism By Mechanism

1. Sustained baseline: rendezvous works

2. Replay: the producer detaches from the consumer's pace

3. The actual heap growth: parked sender goroutines

The Baseline: Lossless + Exact (`CLAUDE.md` + built-in memory)

The Contender: Lossy + Approximate (`claude-mem`)