DEV Community: J. Gravelle

The real cost is not "AI"

J. Gravelle — Thu, 30 Jul 2026 13:16:10 +0000

Stop Paying for Tools Your AI Isn't Using

Three recent results point at the same costly habit:

Anthropic reports an 85% token reduction from its Tool Search Tool, which discovers tools on demand while keeping the full library reachable.

An arXiv paper shows 8.9x faster tool selection and 23.8x lower token usage by loading tools on demand instead of dumping every schema into context, at a scale past 3,000 tools.

A second paper reports 31 to 42% fewer GPU-hours simply by routing on estimated total token budget.

Different layers. Same diagnosis: you are still paying a flat rate for capacity the current turn does not need.

That diagnosis is now consensus. What is still rare, and far more useful to a manager, is a number you can generate yourself, on your own install, with one command.

The real cost is not "AI." It is idle schema.

An MCP host that serializes every tool's schema into context on every turn creates two separate bills:

A fixed per-turn token tax for tools you did not call.
Dispatch dilution: the model must choose correctly from a larger and larger menu.

Retrieval-based loading attacks both. Keep a tiny resident surface. Describe the catalog only when asked. Dispatch by name.

What we actually shipped

21 June 2026 (v1.108.66)

A three-tool front door in front of the entire catalog:

order(action, args): dispatch any catalog action by name
menu(query?): search and browse the catalog as compact rows
route(task, ...): map natural language to the best action (and optionally execute it)

No capability removed. Every action remains one call away.

Safety boundary: order is read-only at the edge. State-changing or file-write actions require an explicit flag or are refused outright.

9 July 2026 (v1.108.115)

The collapsed surface became the default for brand-new installs. Existing installs keep their previous surface, enforced by construction rather than by hope.

21 July 2026 (v1.108.154)

The part that actually matters for cost conversations:

jcodemunch-mcp surface

Read-only. Instant. No session required. It reports:

visible tools vs. full catalog
estimated schema tokens for the active surface
tokens avoided
the heaviest schemas

One real install (v1.108.199, measured today):
6 of 94 tools resident, 1,038 of 25,801 estimated schema tokens, 24,763 avoided.

That measures resident schema cost only, so it is not comparable to the 85% or the 23.8x above, which measure total token usage. Compare it to your own full run instead.

That is not a paper number. It is the number that appears on your machine when you run the command.

Two different meters, deliberately

surface is an advisory receipt. It uses a bytes/4 estimate so it stays instant and dependency-free.

The CI schema-budget gate is a different instrument: tiktoken on a frozen baseline with a 5% drift tolerance. Its job is to stop schema bloat from shipping, not to inform a human.

They will never match digit-for-digit. That is intentional. A receipt that pretends to be a tokenizer count is worse than one that labels its own estimator and stays cheap.

Why this is the interesting number

An 85% figure in a blog post is something you accept.

A figure generated on your own repo, your own config, and your own tool surface is something you can take into a budget meeting.

The general discipline is simple: if you measure one side of a comparison live and leave the other side as a constant in documentation, the comparison rots, usually in your favor. Nobody audits a flattering constant.

We fixed our own README the same way. Absolute numbers went stale as the catalog grew; the multiple happened to survive. The line now names the version, states that bytes/4 is an estimate, and points at the command instead of a bare digit.

A constant in prose has no way to notice it is wrong.

A command re-derives itself every time someone runs it.

The broader pattern

Retrieval shrinks what you load to answer a question.

The front door shrinks what you pay before you have asked one.

The field is converging on the second lever. That is progress.

What should converge next is shipping the meter alongside the optimization.

If your tool-loading change saves 85%, put a command in the package that tells the next person what it actually saved on their install.

They will believe that number, because they generated it.

Run it:

jcodemunch-mcp surface

The interesting number is no longer ours...

-jjg

The Harness Effect...

J. Gravelle — Wed, 22 Jul 2026 13:44:38 +0000

...a controlled study just showed 41% of agent cost is design, not model

Thesis: arXiv 2607.06906 held models and tasks constant, varied only the harness, and found 38% fewer tokens and 41% lower blended cost. The post walks the three harness levers the paper implicates: retrieval strategy, tool-surface size, prefix stability, and shows each is independently measurable, and demonstrates per-session instrumentation for measuring your own.

The experiment: 22 tasks, six models, harness-only variation, 38%/41% at level quality. Why "held constant" matters; this isn't a vendor benchmark.
Lever 1, retrieval: full-file reads vs grounded symbolic retrieval — the 99.6% aggregate (5.12M → 19.4K, 15 task-runs, tiktoken cl100k_base, CI replay-gated), and what varies by repo shape.
Lever 2, tool surface: companion paper arXiv 2607.15593 on tool-surface bloat; our measurement of 93 resident schemas ≈21.6K tokens/turn vs a 6-tool on-demand front door ≈0.9K (dated CHANGELOG receipt).
Lever 3, prefix stability: arXiv 2607.15516 (51.7% cost reduction at three reuses on a 94K-token prefix); our deterministic-emission work and the honest part — we measured 0.139 against a 0.30 threshold and held a feature because the problem was small.
Measuring yourself: token yield rate (credited to FinOps Foundation), the session yield block, advisory budgets, the tool-surface receipt — shipped July 19 and 21, all three servers.
Close: agent cost is a distribution (per McKinsey) and the harness sets the distribution. Instrument first, optimize second.

They're preaching to our well-established choir of over 60,000 developers worldwide. And I'd strongly urge everybody else to pick up a hymnal...

-jjg
https://jcodemunch.com

How a retrieval tool can know when it's wrong

J. Gravelle — Sat, 18 Jul 2026 19:14:12 +0000

Most code-retrieval tools have exactly one voice: confident. You ask, they return their top-k, and the agent on the other end has to guess whether to trust it. That guess fails worst in one specific case: the empty result.

An empty search result is ambiguous. It can mean the thing doesn't exist, or it can mean the index couldn't see it. Those are wildly different facts, and an agent that treats every empty result as "doesn't exist" will confidently hallucinate the negative. "There's no retry logic in this codebase" is a strong claim to make off a search that ran against a half-built index.

We think retrieval responses should carry their own trust signals, machine-readable, so the agent can gate its next action on data instead of vibes. Here's the contract jCodeMunch ships, offered as a shape we'd like to see more of the ecosystem adopt.

The four-state verdict

Every retrieval response carries a verdict:

ok: confident matches returned.
low_confidence: matches exist but score below the confidence threshold. Verify before relying on them.
absent: the corpus was genuinely scanned and the answer isn't there. The verdict carries the scan counts (how many symbols and files were actually examined), so "absent" is a claim with evidence, not a shrug.
degraded: the index itself is impaired (stale, partial, or mid-rebuild). A missing or thin result may be truncation, not absence. Don't conclude anything from silence in this state.

The absent-versus-degraded distinction is the whole point. "Absent" is a positive claim: we looked, here's how much we looked at, it isn't there. "Degraded" is the index admitting it can't be trusted right now. Collapsing those two into one empty list is lying by omission, and it's what most retrieval surfaces do today.

For an agent, the practical difference is enormous. On absent, stop re-querying; the code provably isn't indexed there. On degraded, don't conclude anything; refresh the index or fall back to a raw read. Without the distinction, agents burn tokens re-asking questions the index already answered, or worse, they build plans on a negative that was never proven.

Signal two: calibrated confidence

Verdicts cover the empty case. For non-empty results, every response carries a 0 to 1 confidence score composed from measurable components: the top-1 versus top-2 score gap (is the winner dominant or a coin flip?), absolute top-1 strength, whether an exact identity match is present, and index freshness.

One number, gateable. An agent can decide "above 0.8, use it; below, fetch the source and check" without parsing prose.

Signal three: per-symbol freshness

Every result entry is stamped fresh, edited_uncommitted, or stale_index, derived from comparing the index's recorded git HEAD against the working tree's actual HEAD plus per-file mtime checks. The index doesn't pretend the world stopped at indexing time. It tells you, per symbol, whether what it's showing you still matches the bytes on disk.

The contract is machine-checkable

None of this is prose in a README. The verdict contract is published as a JSON Schema (schemas/retrieval-verdict.schema.json in the repo), alongside schemas for the confidence provenance and the ranked-context response shape. A CI pipeline, yours and not just ours, can validate live responses against it mechanically. If a response ever ships without the honesty fields, that's a schema violation, not a style regression.

Why bother

Retrieval quality gets all the benchmark attention. Retrieval honesty gets almost none, and it's the thing agents actually break on. A tool that's right 95% of the time but can't tell you which 5% it's unsure about forces the agent to treat every answer with the same suspicion, which means re-verifying everything, which means the retrieval tool saved you nothing.

Three honest signals (verdict, confidence, freshness) let the agent spend verification effort only where the tool itself flagged doubt. That's the trade we think the whole ecosystem should be making, and the schema is there for anyone who wants to make it.

The full mechanics are in Chapter 1 of the technical manual: https://github.com/jgravelle/jcodemunch-mcp/blob/main/UNDER_THE_HOOD.md

If a claim in this post doesn't match the source, that's a bug in the post. File an issue and I'll fix it.

-jjg

About that 'your 997 says rejected but not why' problem...

J. Gravelle — Tue, 14 Jul 2026 18:35:52 +0000

Somebody on Reddit posted about 997s that just say AK5*R*5 — one or more segments in error — no AK3, no AK4.

Preach.

That's the problem this free doohickey* is for: rejectdecoder.com

^{*If you'd prefer a "gizmo", I can make that happen.}

What it does

Paste the rejection (997, 999, 824, TA1) plus the original bounced document. It parses both locally in your browser and cross-audits them:

control number agreement
segment counts
envelope consistency
code validity
required segments

It then quotes the exact segment byte-for-byte and ranks the likely causes for anything it finds. If it finds nothing, it says the answer isn't in the docs and tells you to escalate to your partner with your control numbers — which beats pulling a diagnosis out of my... AIs.

Where the AI does (and doesn't) fit

I know how and appreciate WHY "AI-powered EDI" is sneered at. So the audits here are deterministic parser code, not a model. The AI only writes the plain-English narration of facts the parser already verified, every card says so, and if the narration fails you still get the full audit results. No hallucinations or guesswork.

Privacy

Parsing runs entirely in-browser (the real Python parser, compiled to WebAssembly via Pyodide) and even works with the WiFi off. If you use narration, only a masked summary you preview first ever leaves the page. Don't take my word for it — check your network tab.

Free. No signup for the examples or the deterministic audits; narration is a handful of decodes a month with just an email.

Built it solo from an in-house tool of mine, so it's young AND kinda old. Please tell me where it's wrong. Walmart's rejection quirks are encoded so far. Whose partner nonsense should be next...?

-jjg

You Don't Need an LLM to Route Agent Context: Regex Beats Classifiers by 45 Points

J. Gravelle — Wed, 08 Jul 2026 18:35:19 +0000

LLM agents burn a ridiculous number of tokens on redundancy: opening the same files again and again, trying a patch, failing, then wandering back through the repo like they’ve never seen it before.

A July 2026 paper, ContextSniper: AntTrail's Token-Efficient Code Memory for Repository-Level Program Repair, puts real numbers behind that waste. In repository-level repair, agents keep dragging in irrelevant code and logs. ContextSniper tackles that with a context layer built around tiered memory and an intention-aware context gate that filters low-value regions before they ever reach the model.

That gate alone cut tokens by 51.5% on one host agent and 38.9% on Claude Code, while submitted-resolution rates stayed basically in the same neighborhood.

The gate is the interesting part, because it is not tied to that paper’s exact system. It is a more general idea, and it is starting to show up across agent architectures.

At heart, the gate is just a classifier. Given a request, it has to decide what kind of retrieval will answer the question cheapest: symbol lookup, semantic search, graph impact, mutation prep, or something else.

That leads to the practical question the paper does not really answer:

Do you need another LLM call just to decide what context to retrieve?

We tested that directly.

Five ways agents get code into context

Before you can gate anything, you need a retrieval strategy. Most current systems fall into one of five rough families:

Grounded read-only retrieval: parse the code and return exact symbol source by name. Byte-precise, no synthesis.
Graph code intelligence: model calls, imports, entities, and dependencies as a graph, then traverse it.
Embedding / RAG search: use vector similarity over chunks.
Whole-repo packers: compress or dump the repo into the context window.
Mutate / execute runtimes: retrieve context, then modify or run code.

None of these is magic. Graphs are great for relationships, but they can drift away from source. RAG is useful, but fuzzy by design. Packers are simple, but expensive. Mutation runtimes are powerful, but they widen the blast radius.

The important point is that every approach still has to answer the same question:

What should I fetch for this request?

That decision is the gate.

The gate is where the real leverage is

An intention-aware gate looks at a request like:

“where is parse_config defined”
“how does caching work here”
“what breaks if I rename this”
“change the timeout to 60s”

Then it chooses the cheapest retrieval path that is likely to work.

That might be a symbol lookup. It might be semantic search. It might be a graph-impact query. It might be mutation prep.

ContextSniper uses a traceback / behavioral / stateful intent split, which is a useful framing. But it leaves an obvious engineering question for anyone building this kind of system:

How heavy does the gate actually need to be?

The default instinct is to reach for an LLM router, because “intent” sounds like a language-understanding problem.

We wanted to know whether that was true.

The setup

We used 140 hand-authored requests, balanced across seven dispatch classes:

symbol lookup
text search
semantic
graph impact
structure
stateful
mutate

The label space borrows from ContextSniper’s intent split. We ran five-fold stratified cross-validation and compared three cheap routing tiers:

heuristic: about 40 lines of regex over the request text
centroid: TF-IDF with nearest class mean
logreg: TF-IDF with a linear classifier

The result

Tier	What	Accuracy	Macro-F1
heuristic	~40 lines of regex	94.3%	0.945
centroid	TF-IDF nearest-class	47.9%	0.474
logreg	TF-IDF linear	48.6%	0.484

The regex tier won by about 45 points.

The two learned models were not terrible for a seven-class problem, but they were nowhere close. Same corpus, same folds, same labels, and the dumb little ruleset walked off with the trophy.

That was surprising enough that we dug into why.

Intent lives in shape, not word frequency

Bag-of-words models care about which words appear and how often. But for these requests, intent usually lives somewhere else. It lives in the shape of the request.

“Where is X defined” and “how does caching work” do not share much useful vocabulary, but both are almost embarrassingly easy to classify from structure alone.

A camelCase or snake_case token usually means symbol lookup.

A leading “how does” or “why” usually means behavioral or semantic exploration.

“What breaks if I” usually means impact analysis.

“Change,” “rename,” or “set ... to” usually means mutation.

A quoted literal or bare number often means text search.

Regex sees those shapes directly.

TF-IDF mostly throws them away, then gets punished again by the tiny dataset. With roughly 16 examples per class, a sparse linear model does not have much signal left to recover.

In this experiment, the cheap learned tier was not merely unnecessary. It was worse than the rules.

The 3% that fights back

Four of the 140 requests resisted both cheap tiers:

Example Request	Why it resists cheap classification
what does the parse method look like	symbol lookup, but no definitional keyword
find where the timeout value 30000 appears	text search phrased like a location query
show me how caching is implemented	semantic vs symbol, genuinely ambiguous
what modules depend on the core package	graph question that reads structural

These are not really rule failures. They are ambiguity.

And the fix for ambiguity is not always “ask a bigger model.” Often, the fix is to probe reality.

Try the cheapest grounded lookup first. If it misses, fall back to semantic search. That resolves the uncertainty by checking the code, not by paying a language model to make a better guess.

What to build

For an intention-aware gate, the order should be:

Start with regex. It handles most traffic, costs nothing, behaves deterministically, and never rate-limits.
Add probe-and-fallback for the ambiguous cases. When a request could mean two things, cheaply test the first interpretation before escalating.
Use a model call only for what survives both. In our corpus, that was about 3% of requests, and even those had cheaper ways out.

The expensive tier should be the last resort, not the front door.

That is the opposite of the usual reflex, which is to reach for an LLM router because the problem sounds like “understanding.” In practice, a lot of the signal is sitting right there in the punctuation, casing, verbs, and shape of the request.

Honest caveats

Two points that really matter:

First, our corpus and rules share an author. So 94.3% is an in-distribution number. It proves that hand-written rules can separate hand-written requests, which is not exactly a thunderclap from Mount Science.

The number is less important than the shape of the result. Real validation needs real request logs. I would happily rerun this on a public trace.

Second, ContextSniper is convergent evidence, not an endorsement of any specific tool. It is a separate group arriving at the same architecture from the repair-agent direction: tiered memory plus an intent gate, large token savings, and resolution held roughly flat.

To be precise about “roughly,” their official validation reports 24.0% of issues resolved with the gate versus 26.0% for the baseline, which they describe as comparable while also noting a validation-error imbalance.

The important signal is that two independent lines of work are landing in the same place: context gates matter. The exact accuracy of one regex tier is not the point.

Where this leaves the taxonomy

The gate sits above whatever retrieval family you use.

In our stack, it routes into a grounded read-only retriever, jCodeMunch, because byte-exact symbol source is often the cheapest correct answer to the most common kind of request.

But the routing lesson is not specific to jCodeMunch. It applies whether you retrieve by symbol, graph, vector, or whole-repo packing.

The thing deciding what to retrieve should be cheap first, deterministic second, shape-aware third, and “smart” only when it has no cheaper option left.

For the full map of the five families and where different tools fit, there is a running field guide to code-context tools.

Links: ContextSniper (arXiv 2607.01916) · field guide to code-context tools · jCodeMunch

They Spent $81,267 By Accident...

J. Gravelle — Mon, 29 Jun 2026 15:43:43 +0000

...and I Spent $1.39 On Purpose

Last week, fintech startup Slash told its whole company to lean into AI coding. One employee took that note personally, sat down with Claude, and built a video game.

Then the bill showed up: $81,267. In one week. On the company card.

The game is called Brainrot Shooter. It's a bare-bones, blocky, Minecraft-looking first-person shooter where you run around blasting characters named after viral memes (Skibidi Toilet, Tung Tung Tung Sahur, the whole brainrot pantheon). Slash handled the receipt the only sane way: they posted it to the internet and begged people to play the game so they could write it off as a marketing expense. It went viral. The dumb little game actually found an audience.

The employee summed it up himself: "This is actually insane, am I going to become a case study for how AI spend can get out of control."

Yes. You are. And this is the case study. But not the one you think.

Everybody blamed the wrong thing

The internet's takeaway was "see, AI coding is a money pit." That's the lazy read, and it's wrong. The model didn't cost him eighty grand. The way he used it did.

Here is the part nobody on X bothered to explain.

A coding agent is stateless. The model has no memory between turns. So every single time you hit send, the harness re-sends the context it needs: your system prompt, the conversation so far, and the file contents the model is supposed to look at. The model reads all of it, fresh, every turn. You pay for all of it, fresh, every turn.

Now watch what happens over a full day of active development. Your codebase grows. Your conversation grows. The agent keeps pulling large files into context so it can "look at the whole project and change this one thing." Every one of those turns re-bills everything you already showed it five minutes ago. Do that across hundreds of turns on a swelling codebase and the meter spins like a slot machine that never pays out.

That is not the AI being expensive. That is you paying to make it re-read the same code a few hundred times.

The tell is in the token split

Here's where it gets concrete, and here's the part a developer can actually use.

Claude Opus 4.8 bills input and output tokens at different rates. As of this writing, straight from Anthropic's own announcement:

Input: $5 per million tokens
Output: $25 per million tokens

Output is five times pricier per token. So your gut says output is where the money goes. In agentic coding, that's backwards. Output tokens are what the model writes, and a model can only write so fast. Input tokens are what the model reads, and there is no ceiling on how many times you can make it re-read the same files. Input is cheaper per token and ruinous in volume, because volume is the thing that compounds turn over turn.

So when a coding bill goes nuclear, it's almost always input-dominated. I don't have the Slash employee's dashboard, so I won't pretend to quote his exact split. But a five-figure bill from a single day of iterative development is the unmistakable signature of context reloading: the same big files, read again and again, hundreds of turns deep. That is the mechanism. That is what burned the card.

Which gave me an idea for a test.

Same game. One shot. On purpose.

I rebuilt Brainrot Shooter from scratch (mine's called jBrainRot, naturally). Same premise, same blocky world, same meme enemies waddling at you while a combo counter climbs. The difference was in how I asked.

Instead of a day-long conversation where the model re-reads a growing codebase on every turn, I wrote one complete, self-contained prompt. A full spec: the tech constraints, the controls, the enemies, the game feel, all of it, up front. One message. One shot. No "now look at the file again and tweak this" loop, because the loop is the leak.

I also ran jCodeMunch in the middle, an MCP tool I built that trims the dead weight out of context before it ever reaches the model. (Full disclosure: that's my product. Use it, don't use it, the principle stands either way.)

The result was a playable browser game. And here's the receipt:

Input: 11,900 tokens = $0.0595
Output: 53,400 tokens = $1.335
Total: $1.39

Look at that split. It's output-heavy. The model spent its tokens writing the game, not re-reading the game. That is the exact inverse of the reloading trap, and it's why the number has a decimal point in front of it instead of five digits.

His $81,267 versus my $1.39. That's roughly 58,000 to one, for the same dumb game.

Don't Nick-Up Your Budget

You don't need my tool to avoid this (but it'd really help). You need to stop paying to make the robot re-read. A few rules that actually move the number:

Spec hard, then let it write. The single biggest lever is turning a hundred small "look again and tweak" turns into one well-specified turn. Re-reads are the cost. Front-load the thinking so the model writes instead of re-reads. An output-heavy bill is a healthy bill.
Watch your context window, not just your prompt. The expensive token is the one you send 300 times without noticing. If your agent is loading huge files every turn, that's your leak. Trim it.
Turn on prompt caching. Anthropic's prompt caching bills repeated context at cache-read rates, up to 90% cheaper than fresh input. If you're running long sessions without it, you're paying full freight to re-read static files. This alone would have gutted a bill like Nick's.
Cap your spend, then watch the meter. Set a hard budget limit in your harness before you start, not after the invoice. And put eyes on the live number: something like our free jmunch-console that not only tracks token usage, savings, and throughput, but also fires threshold alerts so a rogue leak trips a wire instead of surfacing as a five-figure surprise. (Disclosure: also mine. The category is the point.) "I underestimated my own ability" is a beautiful sentence and a terrible budgeting strategy.
Mind the defaults. Opus 4.8 defaults to high effort and bills its thinking tokens at output rates. Great for hard problems, wasteful for trivial ones. Match the effort to the task.

None of this is exotic. It's just the stuff nobody tells you until your CFO takes your nameplate off the wall.

The actual lesson

The viral version of this story is "AI made a guy spend $81,000." The true version is "a guy paid to make a model re-read his code a few hundred times, and nobody had set a limit."

The model isn't the money pit. Your context window is. Watch that, and the same work that torched a five-figure bill costs you next to nothing...

*The full one-shot prompt I used (and jBrainRot, the game it built) are free on the repo. More token-efficiency tooling at jcodemunch.com

Your AI's memory is quietly making it worse

J. Gravelle — Thu, 11 Jun 2026 11:16:16 +0000

...and your CLAUDE.md is next

On June 10, TechCrunch ran a piece called "How memory tools can make AI models worse." I read it the night it dropped, and the next morning I shipped a fix to one of my MCP servers. So this one's personal.

Memory systems built to personalize your AI, the ones that remember your preferences so the model feels like it knows you, can make that model less accurate and more sycophantic as the stored context piles up. The research came out of Writer, led by Dan Bikel, and it tested real systems (Mem0 and Zep) under peer review. The feature sold as making your assistant smarter was, in their tests, making it dumber.

Here's the part most coverage skipped: This isn't a chatbot problem. If you run coding agents with a CLAUDE.md file, a memory MCP, learned routing weights, or any config that accumulates over time, you're exposed to the exact same failure mode. The mechanism doesn't care whether the persistent state lives in a vector store or a markdown file in your repo root.

The common thread

In every version of this failure, the cause is the same. You have persistent state with no notion of relevance and no notion of expiry. The system stores something true at the time you stored it, then keeps applying it forever, to everything, whether it fits or not.

The research has a clean example. Tell the model a user's favorite book is "Station Eleven." Later, ask a question that has nothing to do with reading preferences, and the model reaches for that stored fact anyway and names the book as a bestselling dystopian novel. The anchor leaked into a query it didn't belong in. In their financial-analysis tests it got worse: models with memory enabled started adopting the user's misconceptions instead of running the numbers independently. The memory didn't add knowledge. It added a bias the model felt obligated to honor.

Once you see the pattern, you start seeing it in your own tooling. Four places, specifically. Here's each one and the shape of the fix.

Problem 1: Stale anchors

Learned state goes stale and keeps steering anyway. Tuned weights, cached rankings, accumulated usage stats: if any of them trained on your project as it looked six months ago, they're still nudging behavior toward a codebase that no longer exists. The model isn't wrong about what it learned. It's wrong about when.

The fix is a recency window on anything that learns from a history log. Don't compute proposals from all of time. Compute them from the part of time that still resembles now.

from datetime import datetime, timedelta

def learn_weights(events, window_days=90, lifetime=False):
    if lifetime:                      # explicit escape hatch
        scoped = events               # for true lifetime reads
    else:
        cutoff = datetime.now() - timedelta(days=window_days)
        scoped = [e for e in events if e.timestamp >= cutoff]
    return compute_proposals(scoped)

The window is the whole point. The lifetime flag is there because sometimes you really do want the full ledger, and that should be a deliberate choice you can read in the call site, not the silent default. This is the fix I shipped the morning after the article. It was about nine lines.

Problem 2: Irrelevant anchors and preference leakage

This is the "Station Eleven" failure, generalized. A memory store retrieves candidates by similarity, similarity is loose, and a memory that's vaguely close to your query gets injected into the prompt for a query it has nothing to do with. The model treats it as relevant because you handed it over. So scope the injection.

def gather_memories(query, candidates, threshold=0.82):
    keep = []
    for m in candidates:
        if relevance(query, m) < threshold:
            continue                  # not close enough, drop it
        if not m.provenance_ok():     # where did this come from?
            continue
        keep.append(m)
    return keep

Two gates, not one. Relevance decides whether the memory belongs in this query, and a provenance check decides whether you trust where it came from in the first place. When a candidate is borderline, leave it out. A missing memory costs you one extra lookup. A wrong anchor costs you correctness, and you won't see the bill until the answer's already wrong.

Problem 3: Memory-file rot

Your CLAUDE.md, your AGENTS.md, your .cursorrules: these are memory too, just written in markdown instead of embeddings. And they rot. You rename a function and the file still references the old name. You delete a directory and the file still points at the path. You add a rule in March that contradicts a rule from January, and both are still in there. The agent reads all of it as gospel and obeys ghosts.

The fix is to audit the config against ground truth, meaning the actual code, not the config's memory of the code.

def audit_config(config, code_index):
    issues = []
    for symbol in config.referenced_symbols():
        if symbol not in code_index.symbols:
            issues.append(("stale_ref", symbol))
    for path in config.referenced_paths():
        if not code_index.exists(path):
            issues.append(("dead_path", path))
    issues += find_contradictions(config.rules)
    return issues   # flag for deletion, don't auto-delete

Every symbol the config names should exist in the index. Every path should resolve. Every rule should be consistent with its neighbors. Anything that fails gets flagged for a human to cut. You'd be surprised how much dead weight a six-month-old agent file is carrying.

Problem 4: Silent self-modification

This is the one that turns a small error into a spiral. Some systems write their own memory or rewrite their own config without a checkpoint. So the model makes a wrong inference, stores it, reads it back next turn as a fact it "already knows," and builds on it. That loop is how sycophancy snowballs. Nobody ever told it no, so it keeps agreeing with the wrong thing more confidently each round.

The rule is simple. Suggest, never write.

def propose_memory_update(current, proposed):
    diff = render_diff(current, proposed)
    print(diff)
    if not approval_step(diff):       # human or supervising agent
        return current                # rejected, nothing changes
    return apply(proposed)

The system can propose whatever it wants. It just can't commit. Show the diff, require an explicit yes, and the snowball never starts because there's always a point where someone can look at the change and say that's not right.

The actual fix isn't "turn memory off"

None of this argues for ripping memory out. Memory is useful. A model that remembers your stack and your conventions is a better collaborator than one that re-learns you every session.

The fix is memory with hygiene. Recency windows so old state expires. Relevance scoping so anchors only fire on the queries they fit. Ground-truth audits so your config can't drift away from your code. Human-in-the-loop writes so errors can't compound unsupervised.

And underneath all four guards there's one principle. Grounded retrieval beats accumulated recollection. Deriving your context from the source artifact at query time, the live code, the current files, the real index, will beat trusting what you wrote down about that artifact months ago. Recollection rots. The source doesn't.

You can wire up all four of these guards yourself. The sketches above are most of the shape, and none of them are long.

Or you can install jCodeMunch, a free MCP server (pip install jcodemunch-mcp) where this is already the architecture. Retrieval is grounded in a live index of your actual code instead of accumulated memory, weight learning is recency-windowed out of the box, and a built-in audit_agent_config tool finds the rot in your CLAUDE.md for you. It's here: https://github.com/jgravelle/jcodemunch-mcp

Look What GitLab Invented!

J. Gravelle — Sun, 31 May 2026 12:38:03 +0000

The industry just got slapped awake by two numbers it cannot unsee.

Uber torched its entire 2026 AI coding budget in four months. Its COO went on record saying he still cannot draw a straight line from all that Claude Code spend to shipping more useful features. "That link is not there yet."

At the same time, an unnamed company accidentally dropped half a billion dollars on Claude in a single month because nobody put a cap on employee licenses.

The panic narrative writes itself:

"AI IS TOO EXPENSIVE!"

The diagnosis everyone is missing: most of those tokens were spent forcing models to skim entire files just to find the three functions, classes, or references they actually needed.

You are not paying for intelligence. You are paying retail, per token, to make the robot read the whole damn book when it only needed one page.

GitLab just validated the fix in public

While the budget fires were burning, GitLab was building the same fix in the open. They call it Orbit, their Knowledge Graph. It turns repositories into a structured, queryable map of definitions and cross-file references, then hands agents only the precise context they ask for through what they literally call a unified context API. It even exposes an MCP endpoint.

Sound familiar?

It should. That is the entire premise behind the jMRI spec and jCodeMunch.

GitLab's own Knowledge Graph team wrote down exactly why they built it. In their internal engineering notes they describe treating AGENTS.md as a table of contents instead of an encyclopedia, keeping knowledge in structured docs, and enforcing architecture mechanically. Their stated design goal is agent legibility, on the logic that anything the agent cannot access in-context effectively does not exist.

A four-person team used that discipline to ship roughly 135,000 lines of Rust in about two weeks, around 95% of it agent-generated. They validated the thesis with their own velocity numbers.

I have been screaming that from the rooftop since March.

Here is where the big-vendor version gets... funny

GitLab's full SDLC graph, the one that actually maps merge requests, pipelines, work items, and code together, is Ultimate tier, GitLab.com only, runs in its own Kubernetes cluster, and is still sitting behind an experiment flag that is off by default. The local code piece writes to a private DuckDB file on your machine and is built to feed GitLab's own stack.

The insight is correct.

The delivery vehicle is "buy our most expensive seat, stand up a cluster, and put your source on our infrastructure."

The same idea, on your laptop, for the price of a tank of gas

jCodeMunch does the retrieval part locally. It is MCP-native, so any agent that speaks MCP can use it. It works against any repository on any host. Your code never leaves your machine. No recurring enterprise seat. No cluster to babysit.

This is not theory. Universal Plant Services, an industrial-services company, is already running the full jMunch platform across more than forty engineer seats. Their live dashboard, as of this week:

Nearly 48.7 billion tokens saved across the fleet in 72 days, an API-rate value of $177,256.07.

That is what jCodeMunch's savings counter put up, in a couple months, at a single company.

While other organizations are writing postmortems about leaderboards that rewarded waste, a real forty-plus-seat shop is quietly running the efficient version and letting the numbers speak.

Honest side-by-side

Aspect	GitLab Orbit	jCodeMunch
Where it runs	Their Kubernetes cluster, GitLab.com only	Your laptop
Price	Ultimate-tier seat, ongoing	One time, per seat or platform
Repositories	GitLab-hosted, default branch only	Any repo, any host
Model lock-in	Built for GitLab's own stack	Model-agnostic over MCP
Your source code	Lives on their infrastructure	Stays local
Status	Experiment, feature-flagged, off by default	Shipping, running at 40+ seats today

GitLab independently reached the same architectural conclusion at the exact moment the rest of the industry is waking up to the bill.

If your agent cannot retrieve it, it does not exist. GitLab's own team said it. The receipts for ignoring it are showing up in this week's headlines.

The fix is not "use less AI". It's "use AI less".

Stop buying the whole book when your robot only needs one page...

Saving the World From AI... with AI!

J. Gravelle — Sat, 16 May 2026 12:46:04 +0000

I applied SCI for AI to a coding-assistant tool, and the math actually works

The Green Software Foundation ratified SCI for AI five months ago. It's the AI extension to the Software Carbon Intensity standard, the same ISO/IEC 21031 methodology that's been around for software broadly, now with AI-specific boundaries and functional units bolted on.

If you've never heard of it, that's fine. It's new, the case studies published so far are all from Microsoft, UBS, Google, and Accenture, and the existing literature focuses on training and serving infrastructure rather than the application layer. None of which is where most of us work.

I wanted to know if the spec held up at the LLM-tooling layer, the part of the stack where MCP servers, retrieval-augmentation tools, context compressors, and developer-facing AI assistants actually live. I had a tool with three months of production telemetry and a number I could point at, so I applied the spec to it.

The short version: the spec works cleanly there, the math is more interesting than I expected, and there's a piece of intellectual machinery in it that I think is underappreciated and worth showing to other developers.

The setup

The tool is jCodeMunch, an MCP server I maintain. It sits between AI coding assistants and the codebase they're working on, and serves AST-level summaries and dependency graphs instead of the full-file reads the assistant would otherwise do. The pitch is that the assistant gets the same answers from a tenth of the input tokens.

Since March 3rd I've been collecting opt-in production telemetry on per-call usage.input_tokens deltas, what the assistant would have requested versus what the tool actually returned. As of writing, the counter sits at 225,266,057,553 input tokens across 24,645 reporting sessions. The endpoint is public if you want it: https://j.gravelle.us/APIs/savings/total.php. Returns JSON.

That's the raw observable. The interesting part is what SCI for AI lets you do with it.

The spec, briefly

SCI for AI uses the same base formula as classic SCI:

SCI = ((E × I) + M) per R

where E is energy, I is grid carbon intensity, M is amortized embodied emissions, and R is the functional unit you scale by. What SCI for AI adds is two persona-based boundaries (Consumer, which covers operation and monitoring, i.e. what deployers experience; and Producer, which covers training, fine-tuning, and the upstream lifecycle), plus standardized functional units that let you compare across AI system types.

The LLM-tooling layer sits squarely at the Consumer boundary. We're not training models. We're affecting what gets sent to inference. The relevant R for the buyer's experience is per-developer-task: one self-contained unit of work like "explain this function" or "find where authentication is handled." Each task corresponds to one or more LLM calls with measurable token counts.

That's the whole framework. The rest is putting numbers in the right places.

The thing I want to show other developers

When you go to compute per-task energy reduction, you immediately hit a problem: nobody actually knows what per-input-token energy is. The published estimates span an order of magnitude. The peer-reviewed numbers from Microsoft Research's recent Joule paper (median 0.31 Wh per query, IQR 0.16 to 0.60) are the most defensible, and even they're explicit that their precision is roughly an order of magnitude. ML.ENERGY, TokenPowerBench, and IEA all give different ranges within that uncertainty.

This is where the spec earns its keep. If a developer-task uses T input tokens under the baseline and T' under the with-tool case, and per-token energy is some unknown but bounded e:

E_baseline = T  × e
E_tool     = T' × e
Reduction  = (T - T') / T

The e cancels. The percentage reduction in inference energy equals the percentage reduction in tokens delivered, regardless of which per-token energy estimate you trust. As long as I and M are held constant across runs (same grid, same hardware), the per-task SCI for AI score drops by the same proportion.

This is one of those moments where a spec turns out to be smarter than you'd expect. The ratio is the auditable unit. The absolute conversion is downstream of an unsettled empirical question, and the spec doesn't require you to settle it. You commit to the relationship; the literature handles the magnitude as it improves.

For jCodeMunch's end-to-end production reduction rate (15 to 25 percent per task, established via a 50-iteration A/B test on a Vue 3 + Firebase codebase, archived in the repo), that means the per-task SCI for AI score drops by 15 to 25 percent, full stop. No methodology fight required.

What the absolute numbers look like

If you do want absolutes, here's the conversion against the published energy bounds:

Input-token energy	225B tokens × energy	CO₂ at U.S. grid avg (~400 gCO₂/kWh)
0.1 mWh/token (conservative)	22.5 MWh	9.0 tonnes
0.3 mWh/token (upper)	67.5 MWh	27.0 tonnes

For scale: that's between 2 and 6 U.S. household-years of electricity, or 2 to 6 passenger-cars-removed-for-a-year-equivalent on the EPA's standard conversion in our first 10 weeks online. The grid intensity is U.S. average; the actual number depends on where the LLM provider's inference runs, which they don't disclose at the per-query level. That's a regulatory gap SCI for AI's EU AI Act alignment is meant to close over time.

The reason I trust the absolutes despite the uncertainty: even the conservative bound is real elimination, not paperwork. SCI for AI explicitly rejects offsets, RECs, PPAs, and the rest of the score-reduction-via-financial-instrument toolkit. The only thing that counts under the standard is causing fewer GPU-seconds to be consumed in the first place. That's the intervention here.

The mechanism, if you want it

There are two measurements behind the 15 to 25 percent end-to-end number, and they're worth understanding separately because they measure different things.

Synthetic retrieval-layer benchmark. A back-to-back harness compared jCodeMunch's AST-based BM25 retrieval against dense-retrieval RAG at the optimal chunk size, across three open-source repos. Token-per-query reductions ranged 36 to 74 percent against optimized RAG and 99 percent or higher against a full-file-read baseline. These numbers explain why end-to-end savings are achievable. They are not themselves the SCI for AI claim.

End-to-end A/B test. A 50-iteration A/B test on a real Vue 3 + Firebase codebase, contributed by community member @Mharbulous and archived in the repo, ran the same naming-audit task alternating between native Read/Grep/Glob and jCodeMunch's MCP tools. Same model, same iteration count, controlled for session-order effects. The with-tool variant completed more tasks within the timeout (80 percent vs 72 percent), ran shorter on average (299 s vs 318 s), and showed equivalent finding quality. Tool-layer per-task savings landed in the 15 to 25 percent range, lower than the retrieval-layer figure because each iteration also includes variant-independent fixed overhead.

The end-to-end number is what production telemetry actually accumulates. The retrieval number explains why.

Three things I don't know

This is also a real practitioner contribution, which means flagging what I don't have answers to.

Whose I applies. When a developer in one grid region uses an LLM provider whose inference runs in another, the carbon intensity in the SCI for AI calculation should be the inference-side grid, not the developer's. Providers don't disclose this at the per-query level. The math currently uses U.S. average as an upper bound.

How M is allocated for multi-tenant accelerators. The spec allocates embodied emissions by time-share and resource-share. For multi-tenant inference hardware serving millions of inferences per hour across customers, that allocation is non-trivial and relies on disclosure that mostly doesn't exist yet.

What baseline aggregate industry comparisons should use. A reduction claim is only meaningful relative to a baseline. The most defensible baseline for LLM-tooling claims is the same workflow without the tool, but cross-vendor comparisons would benefit from a published reference workload for the LLM-tooling layer specifically. None exists.

I've filed these on the Green-Software-Foundation/sci-ai repo for the working group. If you're working on something adjacent (a context-compression tool, a retrieval optimizer, an MCP server with its own per-task savings story), these are the gaps you'll hit too, and the working group benefits from more practitioners poking at them.

The fuller version

The detailed case study with the full methodology, the per-repo benchmark tables, the A/B test data, and the failure-mode analysis is in the project wiki: Token Reduction as an Energy-Efficiency Action: A 225-Billion-Token Case Study Against SCI for AI. Same numbers, more of the math.

The point of writing this up

Two reasons, beyond the obvious "I built a thing and the numbers are interesting."

The first is that SCI for AI is a usable standard for application-layer work, and most developers haven't looked at it. The Consumer-boundary framing fits MCP servers, retrieval tools, and AI-augmented developer tooling more naturally than I expected. If you're working in this part of the stack, the spec gives you a defensible way to talk about per-task carbon impact that doesn't require you to commit to controversial absolute energy numbers. That's worth knowing.

The second is that the AI-energy conversation has gotten stuck on data-center construction and grid capacity, and the per-task denominator (which the IEA's most recent update explicitly identifies as the leverage point) gets less attention than it should. The software layer is where that denominator moves fastest, and it's where the lowest-capital interventions live. Worth more developer mindshare than it currently gets.

If you build something in this space, publish your numbers. Even rough numbers. The literature improves through case studies, and right now the case-study record is dominated by the largest organizations in the world. More practitioner data is good for everyone...

-jjg

Headless Claude, Done Right: Slice-Level Retrieval and the Subscription Trap

J. Gravelle — Wed, 06 May 2026 14:07:47 +0000

An open-source CLI that respects your Claude Pro auth, retrieves only what it needs, and stays inside the lines Anthropic drew in their TOS.

= = =

Spawn claude -p from a Python subprocess without the right precautions and you'll silently bill your Anthropic API account instead of using the Claude Pro subscription you already pay for. For a 799,000-token query, that's the difference between $0.00 and $11.99. Two environment variable strips later, my CLI does the right thing by default. And that's the smallest piece of what makes this work.

This is a write-up of jragmunch-cli, an open-source tool I built (Apache 2.0, on PyPI) that wraps the official claude -p binary with a few opinions: respect the user's auth, retrieve only what's needed, and stay inside the lines Anthropic actually drew in their own legal docs. The interesting parts are the ones that aren't obvious from the README.

Why your subscription quietly turns into an API bill

Anthropic's claude CLI binary (the one you install with npm install -g @anthropic-ai/claude-code) is auth-flexible by design. It will use whatever credentials it finds, in priority order:

ANTHROPIC_API_KEY if set
ANTHROPIC_AUTH_TOKEN if set
Your Claude Pro / Max OAuth login otherwise

That's reasonable behavior for the binary as a primitive. The footgun is what happens when you spawn it as a subprocess from Python.

By default, subprocess.Popen (and friends) pass the parent process's full environment to the child. If you have ANTHROPIC_API_KEY exported in your shell (because, say, you also use the API from other scripts), every claude -p invocation your tool makes will silently pick that up and bill your API account. You won't see an error. You won't see a warning. You'll just see a bill at the end of the month and a perfectly preserved subscription quota you never touched.

The fix is mechanically tiny:

import os
import subprocess

env = os.environ.copy()
env.pop("ANTHROPIC_API_KEY", None)
env.pop("ANTHROPIC_AUTH_TOKEN", None)

subprocess.run(
    ["claude", "-p", prompt, "--output-format", "stream-json"],
    env=env,
    check=True,
)

Three lines. But the discipline behind it matters: any tool that spawns claude -p from a parent process should default to subscription mode and require an opt-in flag to switch to API. jRAGmunch-CLI's --use-api flag is exactly that, and jragmunch doctor will tell you which mode you're in before you run anything expensive.

If you're wrapping claude -p in your own scripts, copy the pattern. It'll save someone a surprise bill.

The bigger problem: dumping your repo at the model

Even with auth handled correctly, the default pattern for "ask Claude about my repo" still wastes obscene amounts of tokens. The naive pattern looks like this:

Walk the repo
Concatenate every relevant file into one giant string
Stuff it into the prompt
Hope the model finds what it needs

This is what most "chat with your repo" wrappers do, and it's what burns through Claude Pro session limits in fifteen minutes flat. The 2.5GB Node.js source tree I demo with would need around 21 million tokens to fit in one prompt. Even if that fit (it doesn't), you'd be paying for the model to read 100% of the code to answer a question that touches 0.1% of it.

Here's a real run from AskClaude.py, the side-by-side demo script in the repo:

In its raw form, your request may have used as many as 799,037 tokens,
at a cost of $11.99.

Using jRAGmunch-CLI, our call to Opus 4.7 only used 24,771 tokens.

By using your subscription WITHIN THE TERMS OF ANTHROPIC'S TOS, you paid
$0.00 and used a nearly imperceptible fractional percentage of your quota.

799K tokens versus 24K. Same question, same answer quality. The difference isn't the model. The difference is what gets sent to it.

Slice, don't dump

The retrieval layer is where the real engineering happens. jRAGmunch-CLI delegates retrieval to jcodemunch-mcp, a separate MCP server I maintain that does AST-level symbol extraction across 70+ languages via tree-sitter.

Here's the conceptual difference between this and traditional RAG.

Traditional RAG. Chop the codebase into arbitrary text chunks. Embed each chunk. When a query comes in, embed the query, find the chunks with the highest cosine similarity, send those to the model. The retrieval is statistical and approximate. It can miss things. It can include things that look related but aren't. It treats your code as if it were prose.

Slice-level retrieval. Parse the codebase into an AST. When a query references a symbol (function name, class, identifier), look up that exact symbol in the index. Return the actual function body. Trace the actual import graph. The retrieval is structural and exact. If you ask for AuthMiddleware.verify, you get AuthMiddleware.verify, not the seven chunks that happened to contain the word "auth."

Surgical, not statistical.

The result is what shows up in jRAGmunch-CLI's _meta output on every call:

[tokens in=24 out=1273  cost actual=$0.0000 (notional=$0.5334, auth=subscription)  time=27549ms]

actual is what you really paid (zero, in subscription mode). notional is what the same work would have cost via the API at Opus 4.7's input rate. auth is which credential path the subprocess used. Every verb returns this. You always know what you actually spent and what you would have spent.

That transparency matters more than it sounds. Most LLM tooling hides cost behind the abstraction. jRAGmunch-CLI makes you look at it on every call. After a week of that, your intuition for "what's a reasonable token budget for this question" sharpens dramatically.

It's not just `ask`

The verb most people see first is jragmunch ask, because that's the obvious "chat with your repo" use case. But the more interesting verbs are downstream of that:

jragmunch index indexes a repo via jcodemunch (one-time, then incremental on subsequent calls).
jragmunch review does diff-aware PR review against a git range.
jragmunch changelog summarizes changes since a tag.
jragmunch refactor fans out batch refactors across matched symbols.
jragmunch tests generates tests for symbols that don't have them.
jragmunch sweep does pattern-driven cleanup across the repo.
jragmunch run is a power-user passthrough for direct prompts.
jragmunch doctor verifies your CLI + MCP wiring before you spend tokens.

The review and refactor verbs are where this stops looking like a Q&A wrapper and starts looking like an agentic CLI toolkit. review reads your diff, retrieves the surrounding symbol context that the diff actually touches (not the whole file, not the whole repo, just the symbols affected), and runs a structured review pass. refactor does fan-out work across multiple call sites in parallel, with each subprocess getting only the slice it needs.

That fan-out pattern is also where the TOS line gets interesting.

When subscription mode is the right answer (and when it isn't)

Anthropic's Claude Code Legal and Compliance docs draw a bright line that most wrappers ignore. Paraphrased:

Individual ordinary use of Claude Code on your own machine, with your own subscription, is permitted.
Business, always-on, multi-contributor, or high-throughput use should run against the API with an API key.

jRAGmunch-CLI's defaults are tuned to that line. Subscription mode by default for solo interactive work; explicit --use-api for anything that crosses into the second bucket. The README ships a decision table covering the typical cases:

You are…	Recommended mode
A solo developer running verbs interactively on your own machine	subscription (default)
A solo developer running `jragmunch review` in your own personal repo's CI	subscription (default), with `CLAUDE_CODE_OAUTH_TOKEN`
A team running CI bots on a shared / commercial repo	`--use-api`
Multi-developer or commercial automation	`--use-api`
Heavy parallel fan-out (`refactor --parallel 16`, etc.)	`--use-api`

This isn't a workaround. It isn't a loophole. Anthropic explicitly permits the first column and explicitly directs the second column to the API. jRAGmunch-CLI just makes the right default for each case the easy default.

The recent wave of "I got rate-limited on Claude Pro after two days" complaints comes mostly from tools that don't respect this line. They run on a personal subscription, fan out twenty parallel subprocesses doing CI-grade work, then act surprised when the throttle drops. If you respect the line Anthropic drew, your subscription stays healthy. If you don't, it doesn't. jRAGmunch-CLI is opinionated about which side of the line you're on.

Try it

If you have the claude CLI on your PATH and jCodemunch-MCP registered as an MCP server, getting started is two commands:

pip install jragmunch
jragmunch doctor

doctor will tell you whether your auth resolves to subscription or API, whether the MCP server is reachable, and whether anything is misconfigured before you spend tokens. From there:

jragmunch index --repo .
jragmunch ask "how does auth work in this repo"
jragmunch review --since main

If you want the side-by-side cost comparison I quoted earlier, clone the repo and run python AskClaude.py. It prompts for a repo path and a question, then prints the answer plus the token math. Use it as a sanity check on your own codebases or as a template for embedding jRAGmunch-CLI into other tools.

One last thing

The repo is brand new. Star it if it's useful. File issues if it isn't. Send a PR if you've got opinions about which verb should ship next.

The 2.5GB Node.js demo, the live cost math, and a fuller walkthrough are in the AI Tips With J video premiering today. Both links below.

Repo: github.com/jgravelle/jragmunch-cli
Video: https://www.youtube.com/watch?v=ZP0OPSq0jcQ
Comparison page (vs. RAG, vs. raw file reads): j.gravelle.us/jCodeMunch/versus.php

Slice, don't dump.

How we measured 99.6% token reduction across 15 task-runs

J. Gravelle — Thu, 30 Apr 2026 11:31:33 +0000

Two months after publishing the headline, here are the receipts.

Two months ago I published "Your AI Agent Is Dumpster Diving Through Your Code." The most common reply was some flavor of: "Cool numbers, but how did you actually measure them?"

Fair question. Here's the answer.

What we measured

The jCodeMunch benchmark measures retrieval token efficiency — how many LLM input tokens a code-exploration tool consumes compared to reading all source files. It does not measure answer quality, latency, or end-to-end task completion. Those are separate axes (we measure precision separately in jMunchWorkbench, but that's a different post).

Three repos, five queries, run on 2026-03-28:

Repository	Files	Symbols	Baseline tokens
expressjs/express	165	181	137,978
fastapi/fastapi	951	5,325	699,425
gin-gonic/gin	98	1,489	187,018

The five queries cover the most common code-exploration intents I see in the wild: router route handler, middleware, error exception, request response, context bind. Three repos × five queries = 15 task-runs.

Total baseline cost across all 15: 5,122,105 tokens.
Total jcodemunch cost across all 15: 19,406 tokens.

Reduction: 99.6% average.

How the runs work

For each query, the harness does two things and compares.

Baseline: concatenate every file in the repo, tokenize, count. This is the "read everything first" agent — the minimum cost to put the entire codebase in context.

jcodemunch: call search_symbols(query, max_results=5), then get_symbol_source() on the top 3 matching symbol IDs. Total tokens = search response tokens + 3 × symbol source tokens.

AI summaries are disabled during benchmarking (signature-only fallback). Without that, jcodemunch numbers would look even better, but it would conflate retrieval efficiency with summarization efficiency, and those are separable concerns.

Token counts come from the serialized JSON response strings, not raw source bytes — JSON field names and structure overhead are included. Slightly understates savings, but the count is deterministic and reproducible.

Tokenizer: tiktoken with cl100k_base encoding. Used by GPT-4 and compatible with Claude estimates within ~5%.

The "common misreadings"

The methodology doc has a section called Common Misreadings. It addresses the four pushbacks I get every time I publish a number.

"The claim is up to 99%." No — the primary claim is 99.6% average across all 15 task-runs. Individual queries reach 99.9% on large repos with tight symbol matches (error exception on fastapi/fastapi: 99.9%, 801× reduction). The 99.6% aggregate is the honest headline across the current index state.

"I tested it on a different repo and got 80%." Results vary by repo structure. Flat script collections — repositories of hundreds of unrelated standalone scripts — produce lower savings, because the symbol index can't distinguish which script is relevant to a given query, and the agent still has to scan broadly. The benchmark repos (Express, FastAPI, Gin) are structured application codebases where symbol-based navigation is most effective. Testing a flat script collection and comparing to our benchmark is apples-to-oranges. I say so out loud because pretending otherwise would be dishonest.

"The benchmark is cherry-picked." The three repos were chosen to represent common backend frameworks across different languages — JavaScript, Python, Go. No file filtering beyond standard skip patterns. The harness (benchmarks/harness/run_benchmark.py) and query corpus (benchmarks/tasks.json) are in the repo. Run them yourself. Publish your numbers. If they're better than mine, I'll cite you.

"The baseline is unrealistic." It is — but it's intentionally a lower bound. Real agents re-read files, branch across sessions, and load documentation. Actual production baseline costs are higher than what I report. That makes the 99.6% number a conservative floor, not a ceiling.

What others have measured independently

Two reviewers ran their own numbers and published them.

Artur Skowroński (VirtusLab GitHub All-Stars #15): "roughly 80% fewer tokens, or 5× more efficient — index once, query cheaply forever."

Julian Horsey (Geeky Gadgets): "3,850 tokens reduced to just 700 — a 5.5× improvement."

These don't contradict the 99.6%. They're testing different workloads at different scales. Smaller per-query savings × hundreds of queries per session = the same compounding effect at the larger scale. Multiple methodologies converging in the same direction is exactly what you want from a benchmark you'd actually rely on.

How to run it yourself

pip install jcodemunch-mcp tiktoken

jcodemunch index_repo expressjs/express
jcodemunch index_repo fastapi/fastapi
jcodemunch index_repo gin-gonic/gin

python benchmarks/harness/run_benchmark.py

Every release in v1.76.0+ runs the benchmark in CI with a regression gate at 0.02 — if any aggregate metric drops by more than 2% versus the saved baseline, the build fails. I don't ship without the receipts.

What this means for you

If you're building agents on Claude Max, hitting context-window pain, or paying API bills that scale with token count: don't take my word for it. Run the calculator on your own stack.

Token Tax Calculator

It tells you what you're spending today, what jcodemunch saves, and what that maps to annually. Takes 60 seconds. Costs nothing.

The benchmark is a number. Your own stack is the number that matters.

—-jjg
github.com/jgravelle/mcp-retrieval-spec

I didn't set out to build a sustainability tool...

J. Gravelle — Tue, 21 Apr 2026 15:43:12 +0000

...but only seven weeks in, we've accidentally saved enough electricity to power roughly 65 American households for a year, and enough avoided CO₂ to take 64 gasoline cars off the road.

I set out to fix a dumb problem: LLM coding agents load entire files into their context window to answer questions about single functions. That's expensive in dollars, it's slow, and it pollutes the context with junk the model doesn't need. So I wrote jCodeMunch, an MCP server that returns only the symbol, slice, or bundle the agent actually asked about.

Since I shipped the telemetry on March 3, 2026, opted-in installs have collectively avoided 172,000,000,000 tokens of LLM inference.

def estimate_savings(raw_bytes: int, response_bytes: int) -> int:
    return max(0, (raw_bytes - response_bytes) // 4)

That's the whole thing.

raw_bytes = the bytes the agent would have loaded under "cat the file into context". Whole-file reads are the default retrieval primitive for every major coding agent today.
response_bytes = the bytes jCodeMunch actually returned.
// 4 = OpenAI's published bytes-per-token approximation. Within 5% of tiktoken on real code.
max(0, ...) = if my response is somehow bigger than the baseline, I count zero. No anti-savings allowed.

Every single API call emits a _meta block with the delta. You can audit any call. The accumulator flushes to disk every three calls and ships anonymous deltas (opt-out with one flag) to a public endpoint. 172B is the sum.

Why the number is a floor, not a ceiling

I made four choices that all bias the number down:

File-level dedup. A call that touches 5 symbols in one file counts the file once, not five times.
max(0, ...) clamp. Never negative.
Opt-in telemetry only. Users who never enabled reporting don't count.
Single-file baseline. I compare against "read one whole file." A real agent doing grep-and-cat across a repo would've loaded way more.

Telemetry participation is opt-in. The real savings are higher than 172B. I just can't prove numbers I didn't measure. A valid criticism of this number might be: "Dude. It's way higher than THAT."

Tokens → kilowatt-hours

Here's where it stops being cute and starts mattering.

Peer-reviewed estimates of LLM inference energy cluster around 0.004 Wh per token on a modern H100 stack (Epoch AI, Altman's own disclosure, Google's median text query, and the Surfshark meta-analysis all triangulate here).

172B tokens × 0.004 Wh/token = 688,000 kWh avoided.

That's:

The annual electricity use of ~65 average US homes
~292 metric tons of CO₂ not emitted (US grid avg)
~64 gasoline cars off the road for a year
~14,600 gallons of gasoline not burned

From one MCP server. That's free to the general public, and costs $79 for commercial users. Once.

The part nobody in the sustainability conversation is talking about

AWS has publicly stated inference accounts for more than 90% of an LLM's lifecycle energy. Training is a one-time capital expense. Inference is a forever bill. And the inference bill is bloated with tokens that shouldn't exist — context stuffed with entire files when the model needed one function, whole repos ingested when a symbol lookup would do.

Every AI-sustainability paper I've read treats this as an unsolvable infrastructure problem: build bigger data centers, run them on renewables, hope for the best. Nobody asks the obvious question: what if we just sent the model less garbage?

The answer turns out to be worth 64 cars' worth of CO₂, per small indie product, in eight weeks.

Could this sort of frugality allow AI to continue to thrive exponentially with only 1/8th of the proposed new power plants? 1/10th?

I'm not a climate researcher. I'm a developer in Wisconsin who got tired of watching my token bill. But I think there's a bigger story here, and I don't think it's mine alone to tell. Context-size discipline might be the single highest-leverage thing the AI tooling community can do for energy. Every retrieval-augmented anything is, in aggregate, a carbon-reduction tool. Most of us just don't measure it.

jCodeMunch does. The counter is live at [jcodemunch.com]. The formula is on GitHub. If you want to challenge the math, please do — the methodology doc lists every source citation, every conversion constant, and every assumption.

Until somebody does, the number stands: 172 billion tokens, 688,000 kWh, 64 cars. And counting...

DEV Community: J. Gravelle

The real cost is not "AI"

Stop Paying for Tools Your AI Isn't Using

The real cost is not "AI." It is idle schema.

What we actually shipped

Two different meters, deliberately

Why this is the interesting number

The broader pattern

The Harness Effect...

...a controlled study just showed 41% of agent cost is design, not model

How a retrieval tool can know when it's wrong

The four-state verdict

Signal two: calibrated confidence

Signal three: per-symbol freshness

The contract is machine-checkable

Why bother

About that 'your 997 says rejected but not why' problem...

What it does

Where the AI does (and doesn't) fit

Privacy

You Don't Need an LLM to Route Agent Context: Regex Beats Classifiers by 45 Points

Five ways agents get code into context

The gate is where the real leverage is

The setup

The result

Intent lives in shape, not word frequency

The 3% that fights back

What to build

Honest caveats

Where this leaves the taxonomy

They Spent $81,267 By Accident...

...and I Spent $1.39 On Purpose

Everybody blamed the wrong thing

The tell is in the token split

Same game. One shot. On purpose.

Don't Nick-Up Your Budget

The actual lesson

Your AI's memory is quietly making it worse

...and your CLAUDE.md is next

The common thread

Problem 1: Stale anchors

Problem 2: Irrelevant anchors and preference leakage

Problem 3: Memory-file rot

Problem 4: Silent self-modification

The actual fix isn't "turn memory off"

Look What GitLab Invented!

GitLab just validated the fix in public

Here is where the big-vendor version gets... funny

The same idea, on your laptop, for the price of a tank of gas

Honest side-by-side

Saving the World From AI... with AI!

I applied SCI for AI to a coding-assistant tool, and the math actually works

The setup

The spec, briefly

The thing I want to show other developers

What the absolute numbers look like

The mechanism, if you want it

Three things I don't know

The fuller version

The point of writing this up

Headless Claude, Done Right: Slice-Level Retrieval and the Subscription Trap

An open-source CLI that respects your Claude Pro auth, retrieves only what it needs, and stays inside the lines Anthropic drew in their TOS.

Why your subscription quietly turns into an API bill

The bigger problem: dumping your repo at the model

Slice, don't dump

It's not just ask

When subscription mode is the right answer (and when it isn't)

Try it

One last thing

How we measured 99.6% token reduction across 15 task-runs

Two months after publishing the headline, here are the receipts.

What we measured

How the runs work

The "common misreadings"

What others have measured independently

How to run it yourself

What this means for you

I didn't set out to build a sustainability tool...

Why the number is a floor, not a ceiling

Tokens → kilowatt-hours

It's not just `ask`