TL;DR: Swapping grep for MCP retrieval tools cost my coding agent 4.1x more tokens on a 33-file repo, and saved 86% on a 249-file one. Same model, same tasks, opposite sign.
I spent a weekend replicating the agent experiment from a paper called CodeNib, on a Windows laptop with no GPU, against two repositories I wrote myself. Eight behavioral questions, four arms, 32 agent runs, every token read straight off the provider's response.
Two rules before I wrote any code. Measure, never estimate. And a result that contradicts the paper is the interesting result, so do not massage it.
The second rule protected me from the paper. It did not protect me from myself, which is the last third of this post.
What CodeNib claims about agent tokens
CodeNib (arXiv:2607.25431) builds three materialized views over a single repo commit: a lexical index, a dense embedding index, and a symbol graph. One line in the abstract is the reason I read it. Use its context policies and your coding agent spends 50 to 87% fewer trajectory tokens than a paired grep/read agent.
I wanted to know whether that survived contact with hardware I actually have.
| mine | the paper | |
|---|---|---|
| CPU | Ryzen 5 7530U | 2x Xeon Gold 5416S |
| GPU | none | H100 80GB |
| RAM | 15.3 GB | plenty |
| Agent model | gemma-4-31b |
5 models, incl. Gemma 4-12B |
Wall-clock timings from a CPU-only laptop are not comparable to an H100 run, so I do not report them as if they were. Token counts are hardware independent, and tokens are what I measured.
Both repos are smaller than they look
| repo | files on disk | actual source files |
|---|---|---|
| SalesRabbit (TypeScript) | 9,524 | 33 |
| Leadpipe (Python + TS) | 20,301 | 249 |
6,032 of SalesRabbit's TS/JS files are node_modules. 8,558 of Leadpipe's 8,693 "backend Python files" are backend/.venv.
That gap turned out to be the whole story. On 33 real files, discovery is cheap, and a system that makes discovery cheaper has nothing to sell you.
Three context policies that share one tool set
Here is the thing the abstract does not tell you. The paper's three arms all use the same tools. Only the prompt history differs.
-
grep/read: history starts as[S, Q]. The agent discovers everything itself. -
eager: history starts as[S, Q, C10], whereC10is the frozen top-10 embedding-ranked callable-level blocks, computed once before the run. -
eager + compact: same start, then one deterministic history rewrite after the first successful read. It keeps the deduplicated read paths, the latest read result in full, and the first 600 characters of the latest assistant message. Everything else is discarded, and no summarizer runs.
So "50 to 87%" is not one number and not one mechanism. It is the per-model best arm. Gemma 4-12B gives the 87%. Claude Haiku 4.5 gives the 50%, and for Haiku compaction actually made things worse, at 123.3% of eager tokens, which is why the paper's own selection rule picks plain eager for Haiku. That negative result is in the body and nowhere near the abstract.
What I set out to run was different. Hold the prompt strategy fixed and swap the tool set, replacing grep and read_file with the tools from codenib mcp. That is the question an engineer actually asks. It is also not the paper's experiment, and I wrote the distinction into a file before running anything so I could not fool myself later.
Two agents that differ by exactly one thing
The harness is named after coffee, because I was going to be staring at it for two days and the names may as well amuse me. brew_kit.py holds the shared loop. filter_menu.py is the control tool set, grep plus read_file and nothing else. espresso_menu.py pulls its tools live from codenib mcp over stdio. cafe.py runs the whole thing.
The loop knows nothing about which menu it is holding. Swapping one for the other changes the tool set and nothing else. Same system prompt, same 16-turn cap, same temperature 0, same model.
Token accounting is provider-reported, per invocation, summed across the trajectory, which is the paper's own definition. Full transcripts dump to JSON per run, so every count is auditable rather than trusted. That decision paid for itself twice, and you will see both times below.
MCP tools cost 4.1x more tokens than grep
First paired result on the 33-file repo.
| arm | input tokens | turns | found the file |
|---|---|---|---|
control (grep + read_file) |
3,531 | 3 | yes |
treatment (codenib mcp) |
14,480 | 3 | yes |
Both agents found the right file in the same number of turns. Here is the gap taken apart.
| turn | control | treatment | what happened |
|---|---|---|---|
| 1 | 307 | 1,182 | byte-identical user prompt |
| 2 | 1,503 | 4,518 |
grep returned one line, search_semantic returned whole function bodies |
| 3 | 1,721 | 8,780 | control read 6 targeted lines, treatment ran another search |
At turn 1 the user message is identical across arms and the treatment prompt is still 875 tokens larger. That is pure tool-schema overhead. Nine MCP tools with paragraph-long descriptions against two compact ones, charged on every single turn before any code is retrieved. Over 16 turns that is roughly 14,000 tokens of nothing.
If you are wiring an MCP server into an agent loop, that number is the one to take away. Tool descriptions are prompt, and you pay for them every turn.
Nothing in the paper's accounting can surface this, because its arms share a tool set and schema cost cancels by construction. It cannot cancel when the tool set is the variable.
The second cost is subtler. Every search_semantic result comes back with the full function body inline. Genuinely better evidence, and it lands in history and gets re-charged every later turn. The control agent's grep returned a single 60-character line and then read exactly 6 lines.
Across the three tasks on the small repo, the treatment arm cost a median of 410% of control, at identical 3/3 correctness.
One task reversed the sign
Then I moved to the 249-file repo.
| task | control | turns | codenib | turns | ratio |
|---|---|---|---|---|---|
lp-phone |
8,337 | 4 | 5,525 | 2 | 66% |
lp-city |
7,777 | 4 | 6,623 | 2 | 85% |
lp-dberror |
47,276 | 16 (capped) | 6,569 | 2 | 14% |
lp-dberror asks where a violated database constraint gets turned into a readable sentence instead of a raw Postgres error string.
The control agent burned all 16 turns. Fourteen greps, two reads, no FILES: line at all. 47,276 input tokens to find nothing. Every failed grep stayed in history and was billed again on every subsequent turn.
I did not construct that task to produce that outcome. I wrote all eight questions from the source files before running anything, phrased behaviorally so they never name the file or the function. I also checked each target symbol is defined in exactly one place, so there is no ambiguity about the right answer. The agent simply could not guess that "readable message for a constraint violation" lives in db_errors.py behind SQLSTATE codes like 23514.
The treatment agent ran one semantic search and answered in two turns for 6,569 tokens. That is 13.9% of control. The paper's figure for the same model family is 12.9%.
Four arms, eight tasks, 32 agent runs
At that point I still had not run the paper's experiment, so I built it. eager and compact use the control's two tools, so they pay no schema tax at all. CodeNib appears only offline, as the source of the frozen candidates.
| arm | what varies | total tokens | vs control | median | mean turns | found |
|---|---|---|---|---|---|---|
control (grep + read_file) |
nothing | 98,153 | 100% | 100% | 5.4 | 7/8 |
| codenib mcp (tool swap) | tool set | 97,659 | 99.5% | 114.3% | 2.6 | 8/8 |
| eager | prompt history | 70,366 | 71.7% | 114.4% | 2.4 | 8/8 |
| eager + compact | prompt history | 42,172 | 43.0% | 70.8% | 2.6 | 8/8 |
The paper's mechanism reproduces. A 57% aggregate saving at no cost in correctness, and applying the paper's own selection rule picks compact, exactly as the paper picks compact for Gemma.
Same retrieval engine underneath every row. How the evidence reaches the model mattered more than which tools the model was holding.
Every context policy loses on 33 files
Then I split the same table by repository.
| control tokens | codenib | eager | compact | |
|---|---|---|---|---|
| Leadpipe (249 files), 5 tasks | 87,325 | 71.8% | 44.1% | 32.3% |
| SalesRabbit (33 files), 3 tasks | 10,828 | 323.0% | 294.4% | 129.3% |
On the bigger repo compact uses 32.3% of grep/read tokens, inside the paper's claimed band, with 5/5 correctness against the control's 4/5.
On the small one every arm loses, eager worst of all at 294%.
That last row partly exonerates my tool-swap result. My tidy explanation for the 4x penalty was the 875-token schema tax. But the paper's arms carry no schema tax whatsoever and they lose there too, by more. So the dominant variable is neither the tool set nor the delivery policy.
It is whether the control agent's grep was going to succeed anyway. On 33 files it always did, in three turns, for about 3.6k tokens. Every mechanism I tested was buying insurance against a spiral that never happens at that scale.
Eager and compact receive byte-identical candidates, so their contrast is the only clean way to isolate retention. It puts compact at 59.9% of eager. The paper reports 27.9% for Gemma 4-12B. Same direction, same arm selected, roughly half the magnitude. With 8 tasks against their 100 snapshots, agreement on sign is all I would claim.
Nine MCP tools made the model explore
I had just written a confident paragraph about how the treatment arm was flat and predictable. The next task demolished it. On lp-sms it used fewer turns than control and still cost 2.4x more.
turn 1: 1,179 tokens search_semantic
turn 2: 4,980 search_bm25
turn 3: 9,137 search_regex
turn 4: 9,203 search_bm25
turn 5: 13,296 (answer)
Four searches across four different tools. A broad tool surface changes the policy the model follows, not only the cost per call. Nine tools present nine plausible next actions, and a model that is not yet confident will try several. Two tools force convergence or death.
No experiment that holds the tool set fixed can observe that, which is why I think it is the most genuinely new thing in the run. I was glad the task broke my generalization, because I had already written it down.
Embedding batch size, not sequence length
Indexing the 33-file repo had not finished after 45 minutes. I read the vector store source while I waited, found that the embedding model's sequence cap defaults to 8192 tokens, and concluded that was obviously the problem.
Then I benchmarked it instead of fixing it.
| threads | max_seq_length | seconds/chunk |
|---|---|---|
| 4 | 512 | 1.54 |
| 4 | 8192 | 2.06 |
| 1 | 512 | 2.52 |
Sequence length barely mattered. The chunks are around 400 tokens and transformers pad to the longest item in the batch, so the 8192 ceiling was never being reached.
The real culprit was batch size. sentence-transformers defaults to 32 and sorts longest-first, so batch 0 packs the 32 largest chunks together. I watched the worker's working set peak at 6.9 GB on a 15.3 GB machine before it started thrashing. Dropping to batch 4 took the index from unfinished after 45 minutes to 212 seconds.
I nearly shipped the sequence-length fix. It would have degraded retrieval quality for no speedup, and I would have believed it worked, because the batch-size change I was making at the same time was doing all the real work.
Then the bigger repo flatly contradicted the conclusion. Its Python chunks hit the 300-line chunk ceiling, so there the sequence cap really does dominate, exactly as I had first guessed for the wrong repo. Both knobs matter, and which one wins depends on your chunk size distribution.
Errors you will hit installing CodeNib on Windows
Every one of these cost me real time, and none of them can happen in the paper's Linux environment.
-
Indexing segfaulted, exit code
-1073741819, an access violation, 248 seconds in. torch and faiss each ship their own OpenMP runtime DLL and both load into one process.KMP_DUPLICATE_LIB_OK=TRUEfixes it. -
pip install codenibgives you one of the three views. Lexical only. No dense, no graph, no MCP.codenib doctorcatches this immediately and is a genuinely well-built command. Had I trusted the default, I would have benchmarked a BM25 wrapper and called it CodeNib. With the extras the venv goes to 863 MB. -
The structural view cannot be built from the published package at all. It needs external SCIP binaries (
scip-typescript,scip-python) that the graph extra does not provide. It failed on both languages, so four of the nine MCP tools occupy schema tokens on every turn while being unable to return anything. That is the clearest way my numbers are unfair to CodeNib. -
The one parameter separating a 212-second index from one that never finishes is not reachable from the CLI.
register_default_builders()acceptsembedding_batch_sizeandembedding_max_seq_length.codenib indexdoes not pass them through. Two lines of argparse. -
HTTP 403, Cloudflare error 1010 on every POST, while GET worked fine. Cloudflare bans
User-Agent: Python-urllib/3.12. -
HTTP 401 with a correct API key, the one that fooled me longest. PowerShell 5.1's
Set-Content -Encoding utf8writes a byte-order mark, so Python read an invisible character in front ofcsk-...and sent it. Read the file withencoding="utf-8-sig". -
A run that failed with an empty error message.
[sr-hmac] espresso FAILED:and nothing after the colon. The empty string is the clue:concurrent.futures.TimeoutErrorstringifies to"". Printtype(error).__name__next to the message and this can never happen to you.
Auditing my own writeup
The last session ran no new experiments. I recomputed every published number straight from the receipts, using a script that shares no code with my aggregator, so a bug could not hide itself.
Every published number held. One came out better than I deserved. I had reported the schema tax as "approximately 875 tokens", and the first-turn delta is exactly 875 on all eight tasks, because the schemas are fixed and the system prompt is shared.
Then I went looking for places where the prose was doing work the data did not support. I found four, and none of them were wrong numbers.
A truncation I disclosed for one arm and not the other. My results file flags the 12,000-character cap on MCP observations and calls it a caveat that flatters the treatment arm. Good. But my candidate builder has MAX_BLOCK_CHARACTERS = 2000 and I never wrote that down. It binds on 5 of 10 blocks for sr-loop, 3 for lp-decision, 2 for lp-city. That shrinks C10, which lowers the token counts of eager and compact, the two arms producing my headline. I disclosed the caveat that made my negative result look worse and forgot the one that made my positive result look better.
Compaction bought its tokens by spending turns. I reported mean turns as 2.4 for eager and 2.6 for compact and left it there. Per task, compaction raised turn count on 4 of 8 tasks and lowered it on one. The rewrite discards C10, so when the first read was not the right file, the agent goes looking again. Elsewhere I had argued that halving model invocations halves wall-clock time under a rate-limited endpoint. If turns are a benefit when the tool-swap arm delivers them, they are a cost when compaction takes them away.
A run where found measures retrieval, not agent behaviour. On lp-decision, eager and compact both answered straight out of C10 with zero tool calls and produced byte-identical receipts. That is correct per the paper's equation, since no read ever succeeded so the rewrite never fired. It also means the system prompt's "do not emit FILES until you have inspected the code" was violated on 2 of 32 runs, which is a mild point against candidate injection sitting inside the section arguing for it.
A reproduce step that fails silently. tasting_notes.py results_*\receipts.json, the final aggregation command in my own REPRODUCE.md, writes an empty table and exits 0 under PowerShell. PowerShell does not glob-expand arguments to native executables, the pattern arrives as a literal string, is_file() returns False, and my collector had a bare continue. Anyone following the doc end to end gets an empty results file and no indication anything went wrong. That is the worst thing I found, because it is the part a reader actually runs.
A latent bug turned up too, which had touched nothing published. matches_expected was an unanchored bidirectional substring test, so policy.py would match an expected y.py. Every one of my 32 runs cited exact repo-relative paths, so the correctness column is sound, and re-scoring all 32 through the anchored version produces 0 flips. On a repository with less tidy filenames it would have inflated quietly.
The pattern is worth naming. None of these were wrong numbers. They were true statements placed where they did the least damage to the story. Auditing for wrong numbers would have found nothing.
When code retrieval is worth the tokens
- Small repo, conventional naming,
grepusually lands in two or three turns: this costs you tokens. Measured at 143 to 412% for the tool swap, and 129% even for the paper's best arm. Stop here. - Large repo, behavioral questions, domain vocabulary that does not appear in identifiers: it pays for itself several times over and converts failures into answers. Measured at 14% of control on exactly such a task.
- The variance reduction is the actual product. Control tokens ranged 3,531 to 47,276, a 13.4x spread. You are buying a tighter tail, not a lower mean.
One honesty note on my own numbers. These are single trajectories per condition. They reproduce exactly at temperature 0, but that rests on one repeated task, and 8 tasks cannot support a confidence interval. I claim effect size and direction, not significance.
What I would run next
Trim the nine-tool MCP surface down to three (search_semantic, search_bm25, get_manifest) and re-run. That cuts the fixed schema tax and narrows the action space, and I expect most of the small-repo penalty to disappear along with the lp-sms thrashing.
The repo has the full build log with every dead end in the order I hit them, all 32 receipts and transcripts, and a REPRODUCE.md with the exact commands and the checkpoint numbers you should see at each stage. Commit the raw receipts, not the summary tables. That is the only reason the audit found anything.
CodeNib-Replication
A partial, measured replication of CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents (Yu et al., arXiv:2607.25431v1), run on a CPU-only Windows laptop against two real private repositories.
Headline: the paper's eager + compact policy reproduces where discovery is expensive —
32.3% of grep/read tokens on a 249-file repository, inside the paper's claimed 50–87%
band, with 5/5 correctness against the control's 4/5. On a 33-file repository every arm
loses. The variable that decides the sign of the result is not the tool set and not the
delivery policy; it is whether the control agent's grep was going to succeed anyway.
Four arms, 8 behavioral tasks, 32 runs, gemma-4-31b at temperature 0:
| arm | what varies | total tokens | vs control | found |
|---|---|---|---|---|
control (grep + read_file) |
— | 98,153 | 100% | 7/8 |
| codenib mcp (tool swap — not a paper arm) | tool set | 97,659 | 99.5% | 8/8 |
| eager |
If you have run agent-context experiments where the baseline grep agent was strong, I would like to know your repo size. That one variable moved my result further than anything I built.
Top comments (0)