I spent 3.8 million tokens finding which service to change

Alex Punnen — Fri, 04 Sep 2026 14:36:00 +0000

Code: github.com/alexcpn/catalogify · uv tool install catalogify · MIT

Take a monorepo with ninety microservices in it. A spec lands that will touch maybe three of them. Before anyone writes code, somebody has to work out which three, and on a repo that size that is most of the job.

I wanted to know what it costs to let an agent answer that. So I did the obvious thing and gave it a map.

There are good tools for this now. I used Graphify, which parses your code with tree-sitter and builds a queryable graph of every symbol and call edge. It runs locally, needs no API key, and tags every edge with whether it was extracted or inferred. I pointed it at one service and it chewed through 706 files in twenty seconds.

Then I looked at what it produced.

graph.json, 14.5 MB. About 3.8 million tokens for one service.

Ten candidate services would be 38 million. Ninety would be 343 million. And routing is a comparison problem, so you can't just load the winner. You have to load everything you're choosing between.

That was the moment I realised I'd been solving the wrong problem.

Two questions, opposite shapes

There are two questions people lump together as "give the agent context on the codebase."

Which service does this touch? Wide and shallow. You need a little about everything.

Inside that service, what changes? Narrow and deep. You need everything about a little.

Graphify is superb at the second one. graphify affected "UpdateApiUsageCount" --depth 2 gives you a precise impact radius in a few hundred tokens. Nothing I write by hand will ever beat that, and I'm not going to pretend otherwise.

But the artifact backing those queries has to be complete, and completeness is exactly the wrong property for the first question. You're not asking about symbols. You're asking which service owns pod eviction, or image pulls, or volume mounting, and a call graph doesn't know that, because ownership isn't a topological fact.

You can see it in the filenames. Graphify will export a markdown wiki, which is the right instinct. For one service it wrote 446 articles called things like basicWorkQueue.md and allPrimitiveFieldPaths.md. Those are symbols, clustered by how tightly the code couples. Useful for tracing. Useless for picking a service.

What the opposite costs

So I built the other thing. One small markdown file per service, in the language the spec is written in, with typed frontmatter and links to its neighbours.

For the same Kubernetes component, same 108,648 lines of Go:

	Size	Tokens
Structural graph	14.5 MB	3,813,486
Its markdown wiki	1,004 KB	256,968
One service entry	2.6 KB	676

Ninety of those is 61,000 tokens. That fits in one cheap call with the spec still sitting next to it.

The part that surprised me

I wrote the generator to mine git history, on a hunch that the interesting stuff lives in reverts. Then I ran it against the kubelet's container manager and got this back:

508709d007e  cpumanager: improve V4 checkpoint corruption error
bc752cdfdb0  Fix V3 checkpoint checksum for rollback compatibility
76100602564  Revert "Fix CPU checkpoint migration V2 to V3"

Chase that revert to the commit it reverted and the story comes out. When a V3 checkpoint has a bad checksum, restore falls back to V2, but the V3 fields it already read stay in the struct. You get a hybrid. Someone fixed that, and the fix got reverted, so it may still be sitting there.

Nobody wrote that in a comment. There's no ADR. It exists in the commit log and in the head of whoever was on call that week.

That's the stuff I actually want in a routing catalog, and it's the stuff no parser can reach, because the engineer fixing production at 2am was not writing documentation.

Peter Naur got here in 1985. His argument in Programming as Theory Building is that the real product of programming is the theory of the system carried in the developers' heads, not the code, and that no amount of program text or documentation carries it for them. A program whose original team has dispersed is, in his word, dead. A new team patching a dead program makes fixes that look fine locally and quietly wreck the design.

That is the position an AI agent is in on its first commit to your repository. It arrives after the team dispersed, holding the artifacts and none of the theory, and it patches with total confidence.

Writing a better spec does not get you out of it. Brooks settled that one: the complexity of software is essential rather than accidental, so descriptions that abstract it away abstract away the essence. The ceiling is the same whether a human or a model wrote the spec.

A revert is not the theory. It is a fossil of one place where the theory and the code collided hard enough that somebody had to act at speed. Fossils are cheap to collect, and they are worth more than anything else in the catalog.

What I'd do instead

Two layers, and let each tool do the thing it's good at.

Route with markdown. Small files, domain language, one per service, cheap enough to read all of them.

Reach with the graph. Run it inside the one service you picked, call it as a CLI, keep graph.json in .gitignore and out of your context window entirely.

The composition costs less than either half does alone once you're past a few dozen services.

One more thing, since I'd rather say it than have it found: the first time I ran my own tool against Kubernetes it crashed. Exit 141. A head -N closing a pipe under set -o pipefail, which never fires on a small repo because the producer finishes writing before head walks away. Every test I'd written passed. If you only test on small repos, you're only testing the easy case.

Fixed, measured, and the numbers above are all reproducible:

uv tool install catalogify
catalogify install

Then ask your agent for a knowledge bundle. On a monorepo, start coarse.

Source and issues: github.com/alexcpn/catalogify.

Package: pypi.org/project/catalogify.

For Spec Kit projects the same workflows ship as slash commands in speckit_okf.

It is a few days old and I would rather hear where it breaks than where it shines.

MIT licensed. Measurements against kubernetes/kubernetes at d5ccf7968e5, tokens estimated at bytes ÷ 4. The comparison tool is Graphify, run AST-only without an API key.

DEV Community: Alex Punnen

I spent 3.8 million tokens finding which service to change

Two questions, opposite shapes

What the opposite costs

The part that surprised me

What I'd do instead