DEV Community: Pranav Raj

MCP retrieval cost 4x more tokens than grep, until repo size flipped it

Pranav Raj — Wed, 05 Aug 2026 15:06:27 +0000

TL;DR: Swapping grep for MCP retrieval tools cost my coding agent 4.1x more tokens on a 33-file repo, and saved 86% on a 249-file one. Same model, same tasks, opposite sign.

I spent a weekend replicating the agent experiment from a paper called CodeNib, on a Windows laptop with no GPU, against two repositories I wrote myself. Eight behavioral questions, four arms, 32 agent runs, every token read straight off the provider's response.

Two rules before I wrote any code. Measure, never estimate. And a result that contradicts the paper is the interesting result, so do not massage it.

The second rule protected me from the paper. It did not protect me from myself, which is the last third of this post.

What CodeNib claims about agent tokens

CodeNib (arXiv:2607.25431) builds three materialized views over a single repo commit: a lexical index, a dense embedding index, and a symbol graph. One line in the abstract is the reason I read it. Use its context policies and your coding agent spends 50 to 87% fewer trajectory tokens than a paired grep/read agent.

I wanted to know whether that survived contact with hardware I actually have.

	mine	the paper
CPU	Ryzen 5 7530U	2x Xeon Gold 5416S
GPU	none	H100 80GB
RAM	15.3 GB	plenty
Agent model	`gemma-4-31b`	5 models, incl. Gemma 4-12B

Wall-clock timings from a CPU-only laptop are not comparable to an H100 run, so I do not report them as if they were. Token counts are hardware independent, and tokens are what I measured.

Both repos are smaller than they look

repo	files on disk	actual source files
SalesRabbit (TypeScript)	9,524	33
Leadpipe (Python + TS)	20,301	249

6,032 of SalesRabbit's TS/JS files are node_modules. 8,558 of Leadpipe's 8,693 "backend Python files" are backend/.venv.

That gap turned out to be the whole story. On 33 real files, discovery is cheap, and a system that makes discovery cheaper has nothing to sell you.

Three context policies that share one tool set

Here is the thing the abstract does not tell you. The paper's three arms all use the same tools. Only the prompt history differs.

grep/read: history starts as [S, Q]. The agent discovers everything itself.
eager: history starts as [S, Q, C10], where C10 is the frozen top-10 embedding-ranked callable-level blocks, computed once before the run.
eager + compact: same start, then one deterministic history rewrite after the first successful read. It keeps the deduplicated read paths, the latest read result in full, and the first 600 characters of the latest assistant message. Everything else is discarded, and no summarizer runs.

So "50 to 87%" is not one number and not one mechanism. It is the per-model best arm. Gemma 4-12B gives the 87%. Claude Haiku 4.5 gives the 50%, and for Haiku compaction actually made things worse, at 123.3% of eager tokens, which is why the paper's own selection rule picks plain eager for Haiku. That negative result is in the body and nowhere near the abstract.

What I set out to run was different. Hold the prompt strategy fixed and swap the tool set, replacing grep and read_file with the tools from codenib mcp. That is the question an engineer actually asks. It is also not the paper's experiment, and I wrote the distinction into a file before running anything so I could not fool myself later.

Two agents that differ by exactly one thing

The harness is named after coffee, because I was going to be staring at it for two days and the names may as well amuse me. brew_kit.py holds the shared loop. filter_menu.py is the control tool set, grep plus read_file and nothing else. espresso_menu.py pulls its tools live from codenib mcp over stdio. cafe.py runs the whole thing.

The loop knows nothing about which menu it is holding. Swapping one for the other changes the tool set and nothing else. Same system prompt, same 16-turn cap, same temperature 0, same model.

Token accounting is provider-reported, per invocation, summed across the trajectory, which is the paper's own definition. Full transcripts dump to JSON per run, so every count is auditable rather than trusted. That decision paid for itself twice, and you will see both times below.

MCP tools cost 4.1x more tokens than grep

First paired result on the 33-file repo.

arm	input tokens	turns	found the file
control (`grep` + `read_file`)	3,531	3	yes
treatment (`codenib mcp`)	14,480	3	yes

Both agents found the right file in the same number of turns. Here is the gap taken apart.

turn	control	treatment	what happened
1	307	1,182	byte-identical user prompt
2	1,503	4,518	`grep` returned one line, `search_semantic` returned whole function bodies
3	1,721	8,780	control read 6 targeted lines, treatment ran another search

At turn 1 the user message is identical across arms and the treatment prompt is still 875 tokens larger. That is pure tool-schema overhead. Nine MCP tools with paragraph-long descriptions against two compact ones, charged on every single turn before any code is retrieved. Over 16 turns that is roughly 14,000 tokens of nothing.

If you are wiring an MCP server into an agent loop, that number is the one to take away. Tool descriptions are prompt, and you pay for them every turn.

Nothing in the paper's accounting can surface this, because its arms share a tool set and schema cost cancels by construction. It cannot cancel when the tool set is the variable.

The second cost is subtler. Every search_semantic result comes back with the full function body inline. Genuinely better evidence, and it lands in history and gets re-charged every later turn. The control agent's grep returned a single 60-character line and then read exactly 6 lines.

Across the three tasks on the small repo, the treatment arm cost a median of 410% of control, at identical 3/3 correctness.

One task reversed the sign

Then I moved to the 249-file repo.

task	control	turns	codenib	turns	ratio
`lp-phone`	8,337	4	5,525	2	66%
`lp-city`	7,777	4	6,623	2	85%
`lp-dberror`	47,276	16 (capped)	6,569	2	14%

lp-dberror asks where a violated database constraint gets turned into a readable sentence instead of a raw Postgres error string.

The control agent burned all 16 turns. Fourteen greps, two reads, no FILES: line at all. 47,276 input tokens to find nothing. Every failed grep stayed in history and was billed again on every subsequent turn.

I did not construct that task to produce that outcome. I wrote all eight questions from the source files before running anything, phrased behaviorally so they never name the file or the function. I also checked each target symbol is defined in exactly one place, so there is no ambiguity about the right answer. The agent simply could not guess that "readable message for a constraint violation" lives in db_errors.py behind SQLSTATE codes like 23514.

The treatment agent ran one semantic search and answered in two turns for 6,569 tokens. That is 13.9% of control. The paper's figure for the same model family is 12.9%.

Four arms, eight tasks, 32 agent runs

At that point I still had not run the paper's experiment, so I built it. eager and compact use the control's two tools, so they pay no schema tax at all. CodeNib appears only offline, as the source of the frozen candidates.

arm	what varies	total tokens	vs control	median	mean turns	found
control (`grep` + `read_file`)	nothing	98,153	100%	100%	5.4	7/8
codenib mcp (tool swap)	tool set	97,659	99.5%	114.3%	2.6	8/8
eager	prompt history	70,366	71.7%	114.4%	2.4	8/8
eager + compact	prompt history	42,172	43.0%	70.8%	2.6	8/8

The paper's mechanism reproduces. A 57% aggregate saving at no cost in correctness, and applying the paper's own selection rule picks compact, exactly as the paper picks compact for Gemma.

Same retrieval engine underneath every row. How the evidence reaches the model mattered more than which tools the model was holding.

Every context policy loses on 33 files

Then I split the same table by repository.

	control tokens	codenib	eager	compact
Leadpipe (249 files), 5 tasks	87,325	71.8%	44.1%	32.3%
SalesRabbit (33 files), 3 tasks	10,828	323.0%	294.4%	129.3%

On the bigger repo compact uses 32.3% of grep/read tokens, inside the paper's claimed band, with 5/5 correctness against the control's 4/5.

On the small one every arm loses, eager worst of all at 294%.

That last row partly exonerates my tool-swap result. My tidy explanation for the 4x penalty was the 875-token schema tax. But the paper's arms carry no schema tax whatsoever and they lose there too, by more. So the dominant variable is neither the tool set nor the delivery policy.

It is whether the control agent's grep was going to succeed anyway. On 33 files it always did, in three turns, for about 3.6k tokens. Every mechanism I tested was buying insurance against a spiral that never happens at that scale.

Eager and compact receive byte-identical candidates, so their contrast is the only clean way to isolate retention. It puts compact at 59.9% of eager. The paper reports 27.9% for Gemma 4-12B. Same direction, same arm selected, roughly half the magnitude. With 8 tasks against their 100 snapshots, agreement on sign is all I would claim.

Nine MCP tools made the model explore

I had just written a confident paragraph about how the treatment arm was flat and predictable. The next task demolished it. On lp-sms it used fewer turns than control and still cost 2.4x more.

turn 1:  1,179 tokens   search_semantic
turn 2:  4,980          search_bm25
turn 3:  9,137          search_regex
turn 4:  9,203          search_bm25
turn 5: 13,296          (answer)

Four searches across four different tools. A broad tool surface changes the policy the model follows, not only the cost per call. Nine tools present nine plausible next actions, and a model that is not yet confident will try several. Two tools force convergence or death.

No experiment that holds the tool set fixed can observe that, which is why I think it is the most genuinely new thing in the run. I was glad the task broke my generalization, because I had already written it down.

Embedding batch size, not sequence length

Indexing the 33-file repo had not finished after 45 minutes. I read the vector store source while I waited, found that the embedding model's sequence cap defaults to 8192 tokens, and concluded that was obviously the problem.

Then I benchmarked it instead of fixing it.

threads	max_seq_length	seconds/chunk
4	512	1.54
4	8192	2.06
1	512	2.52

Sequence length barely mattered. The chunks are around 400 tokens and transformers pad to the longest item in the batch, so the 8192 ceiling was never being reached.

The real culprit was batch size. sentence-transformers defaults to 32 and sorts longest-first, so batch 0 packs the 32 largest chunks together. I watched the worker's working set peak at 6.9 GB on a 15.3 GB machine before it started thrashing. Dropping to batch 4 took the index from unfinished after 45 minutes to 212 seconds.

I nearly shipped the sequence-length fix. It would have degraded retrieval quality for no speedup, and I would have believed it worked, because the batch-size change I was making at the same time was doing all the real work.

Then the bigger repo flatly contradicted the conclusion. Its Python chunks hit the 300-line chunk ceiling, so there the sequence cap really does dominate, exactly as I had first guessed for the wrong repo. Both knobs matter, and which one wins depends on your chunk size distribution.

Errors you will hit installing CodeNib on Windows

Every one of these cost me real time, and none of them can happen in the paper's Linux environment.

Indexing segfaulted, exit code -1073741819, an access violation, 248 seconds in. torch and faiss each ship their own OpenMP runtime DLL and both load into one process. KMP_DUPLICATE_LIB_OK=TRUE fixes it.
pip install codenib gives you one of the three views. Lexical only. No dense, no graph, no MCP. codenib doctor catches this immediately and is a genuinely well-built command. Had I trusted the default, I would have benchmarked a BM25 wrapper and called it CodeNib. With the extras the venv goes to 863 MB.
The structural view cannot be built from the published package at all. It needs external SCIP binaries (scip-typescript, scip-python) that the graph extra does not provide. It failed on both languages, so four of the nine MCP tools occupy schema tokens on every turn while being unable to return anything. That is the clearest way my numbers are unfair to CodeNib.
The one parameter separating a 212-second index from one that never finishes is not reachable from the CLI. register_default_builders() accepts embedding_batch_size and embedding_max_seq_length. codenib index does not pass them through. Two lines of argparse.
HTTP 403, Cloudflare error 1010 on every POST, while GET worked fine. Cloudflare bans User-Agent: Python-urllib/3.12.
HTTP 401 with a correct API key, the one that fooled me longest. PowerShell 5.1's Set-Content -Encoding utf8 writes a byte-order mark, so Python read an invisible character in front of csk-... and sent it. Read the file with encoding="utf-8-sig".
A run that failed with an empty error message. [sr-hmac] espresso FAILED: and nothing after the colon. The empty string is the clue: concurrent.futures.TimeoutError stringifies to "". Print type(error).__name__ next to the message and this can never happen to you.

Auditing my own writeup

The last session ran no new experiments. I recomputed every published number straight from the receipts, using a script that shares no code with my aggregator, so a bug could not hide itself.

Every published number held. One came out better than I deserved. I had reported the schema tax as "approximately 875 tokens", and the first-turn delta is exactly 875 on all eight tasks, because the schemas are fixed and the system prompt is shared.

Then I went looking for places where the prose was doing work the data did not support. I found four, and none of them were wrong numbers.

A truncation I disclosed for one arm and not the other. My results file flags the 12,000-character cap on MCP observations and calls it a caveat that flatters the treatment arm. Good. But my candidate builder has MAX_BLOCK_CHARACTERS = 2000 and I never wrote that down. It binds on 5 of 10 blocks for sr-loop, 3 for lp-decision, 2 for lp-city. That shrinks C10, which lowers the token counts of eager and compact, the two arms producing my headline. I disclosed the caveat that made my negative result look worse and forgot the one that made my positive result look better.

Compaction bought its tokens by spending turns. I reported mean turns as 2.4 for eager and 2.6 for compact and left it there. Per task, compaction raised turn count on 4 of 8 tasks and lowered it on one. The rewrite discards C10, so when the first read was not the right file, the agent goes looking again. Elsewhere I had argued that halving model invocations halves wall-clock time under a rate-limited endpoint. If turns are a benefit when the tool-swap arm delivers them, they are a cost when compaction takes them away.

A run where found measures retrieval, not agent behaviour. On lp-decision, eager and compact both answered straight out of C10 with zero tool calls and produced byte-identical receipts. That is correct per the paper's equation, since no read ever succeeded so the rewrite never fired. It also means the system prompt's "do not emit FILES until you have inspected the code" was violated on 2 of 32 runs, which is a mild point against candidate injection sitting inside the section arguing for it.

A reproduce step that fails silently. tasting_notes.py results_*\receipts.json, the final aggregation command in my own REPRODUCE.md, writes an empty table and exits 0 under PowerShell. PowerShell does not glob-expand arguments to native executables, the pattern arrives as a literal string, is_file() returns False, and my collector had a bare continue. Anyone following the doc end to end gets an empty results file and no indication anything went wrong. That is the worst thing I found, because it is the part a reader actually runs.

A latent bug turned up too, which had touched nothing published. matches_expected was an unanchored bidirectional substring test, so policy.py would match an expected y.py. Every one of my 32 runs cited exact repo-relative paths, so the correctness column is sound, and re-scoring all 32 through the anchored version produces 0 flips. On a repository with less tidy filenames it would have inflated quietly.

The pattern is worth naming. None of these were wrong numbers. They were true statements placed where they did the least damage to the story. Auditing for wrong numbers would have found nothing.

When code retrieval is worth the tokens

Small repo, conventional naming, grep usually lands in two or three turns: this costs you tokens. Measured at 143 to 412% for the tool swap, and 129% even for the paper's best arm. Stop here.
Large repo, behavioral questions, domain vocabulary that does not appear in identifiers: it pays for itself several times over and converts failures into answers. Measured at 14% of control on exactly such a task.
The variance reduction is the actual product. Control tokens ranged 3,531 to 47,276, a 13.4x spread. You are buying a tighter tail, not a lower mean.

One honesty note on my own numbers. These are single trajectories per condition. They reproduce exactly at temperature 0, but that rests on one repeated task, and 8 tasks cannot support a confidence interval. I claim effect size and direction, not significance.

What I would run next

Trim the nine-tool MCP surface down to three (search_semantic, search_bm25, get_manifest) and re-run. That cuts the fixed schema tax and narrows the action space, and I expect most of the small-repo penalty to disappear along with the lp-sms thrashing.

The repo has the full build log with every dead end in the order I hit them, all 32 receipts and transcripts, and a REPRODUCE.md with the exact commands and the checkpoint numbers you should see at each stage. Commit the raw receipts, not the summary tables. That is the only reason the audit found anything.

CoffeeAurCode / CodeNib-Replication

CodeNib-Replication

A partial, measured replication of CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents (Yu et al., arXiv:2607.25431v1), run on a CPU-only Windows laptop against two real private repositories.

Headline: the paper's eager + compact policy reproduces where discovery is expensive — 32.3% of grep/read tokens on a 249-file repository, inside the paper's claimed 50–87% band, with 5/5 correctness against the control's 4/5. On a 33-file repository every arm loses. The variable that decides the sign of the result is not the tool set and not the delivery policy; it is whether the control agent's grep was going to succeed anyway.

Four arms, 8 behavioral tasks, 32 runs, gemma-4-31b at temperature 0:

arm	what varies	total tokens	vs control	found
control (`grep` + `read_file`)	—	98,153	100%	7/8
codenib mcp (tool swap — not a paper arm)	tool set	97,659	99.5%	8/8
eager

…

View on GitHub

If you have run agent-context experiments where the baseline grep agent was strong, I would like to know your repo size. That one variable moved my result further than anything I built.

Kimi K3 without the maths

Pranav Raj — Wed, 29 Jul 2026 13:30:00 +0000

TL;DR: The biggest AI model ever released to the public came with a 1,500 line instruction manual. Almost none of the good parts are about how big it is. Here is that manual, with the equations taken out and analogies put in.
Twelve days ago I wrote about Kimi K3 on the day it was announced, and I finished that post with a question. Moonshot had promised to release the weights on July 27 along with a technical report. Would they, and what would the report say?

They did. The weights went up on Hugging Face on the 27th and were the most popular thing on the site within half an hour. The report came with them.

I have now read all of it. I have not run the model and I am not going to, because it needs a small server room, but the report turns out to be the more interesting artifact anyway.

Here is the thing I did not expect. The best section in it describes a problem Moonshot created for themselves in the first month of the project and did not discover until the model was in production. I will get to that one, but the setup takes a few steps.

No equations below. If you can follow a story about a restaurant kitchen, you can follow this.

What 2.8 trillion parameters actually means

Parameters are the model's adjustable settings. More of them means more room to store patterns. Kimi K3 has 2.8 trillion, which is the largest number anyone has published openly.

But it does not use them all at once, and that distinction is the whole game.

Picture a hospital with 896 specialists on staff. A patient walks in. They do not see all 896. A triage nurse reads the chart and sends them to 16. The hospital is enormous, but any single visit only involves a small team.

That is a Mixture of Experts, and Kimi K3 is one. 896 specialist sub-networks, 16 of them working on each word. The model is huge on disk and comparatively cheap to run, because 98% of it sits idle at any given moment.

Kimi K2, the previous version, had 384 specialists and used 8. So K3 more than doubled the staff and doubled the size of the team seeing each patient.

Which sounds straightforwardly better, and is where the problems start.

How Kimi K3 reads a million words at once

Kimi K3 can take a million tokens of input, roughly 700,000 words. An entire large codebase. Several novels.

Fitting that much text in was never the hard part. The hard part is that the standard way a model reads is quadratic: to write each new word, it re-reads everything before it. Imagine replying to an email by first re-reading every email you have ever received. Fine for twenty. Ruinous for a million.

The alternative is to keep a running summary instead. After each email, you update your notes and throw the email away. Your notes stay the same size forever, no matter how much arrives. Cheap. But you have lost the ability to go back and quote an old email exactly.

Kimi K3 does both, in a ratio. Three summary-keeping layers, then one full re-reading layer, over and over, 93 layers deep. The summary layers do the bulk of the work cheaply, and every fourth layer gets to look at the raw material properly.

The summary-keeping mechanism is what Moonshot call Kimi Delta Attention, and it comes with a bonus that sounds small and is not.

Most models need to be told explicitly where each word sits in the sequence. Word 3, word 40,000, word 900,000. Stretching a model from 128,000 words to a million means recalibrating that ruler, which is fiddly and lossy.

Kimi K3 has no ruler. The running summary naturally fades older information, so recency is built into the mechanism rather than stamped on top. There is nothing to recalibrate. The model stretches to a million tokens because there is no measuring device to break.

Why 896 experts and only 16 do the work

Back to the hospital, where there is a courier problem.

Normally every specialist you consult receives the patient's complete file. Send it to 8 and you have 8 copies moving around the building. Send it to 16 and you have doubled the courier traffic, the printing, and the delays. This is why models do not simply crank up the number of active experts. The bandwidth bill follows.

Kimi K3 sends a summary instead. Specialists get a compressed version, half the width. Only two general practitioners, who see every patient regardless, get the full file. So the routing can be far more generous without the courier traffic following it.

Now the subtler problem. Who decides which 16?

A router scores every specialist for every word and takes the top ones. Left alone this fails predictably: a few popular specialists get swamped, most idle, and some never see a patient at all and so never learn anything. Worse, when experts live on different machines, one overloaded machine makes every other machine wait.

The old fix was a nudge. Each round, bump the score of underused experts and dock the overused ones by a fixed amount. It is a thermostat, and thermostats have a known failure mode. Nudge too gently and the room never warms. Nudge too hard and it overshoots, then overshoots back, forever. With 896 rooms, no single setting works for all of them.

Moonshot threw out the thermostat. Their replacement is a bouncer who has stopped guessing the dress code: sort everyone by score, count down to exactly the number the venue holds, put the bar right there. One pass, no guessing, the target load every time.

There is a lovely practical wrinkle. To sort everyone you would need to gather every score in one place, and at training scale that is millions of numbers scattered across hundreds of machines. Gathering them is not affordable.

So they do not gather them. Each machine reports how many of its scores fell into each band, like a census returning "4,200 people between 170 and 175cm" rather than a list of names. Add the bands together and you have the national picture. Counts add up cleanly no matter how you split the population, which is why the answer is exact regardless of how the data happened to be distributed.

If you ever need a global statistic across shards and cannot afford to collect the raw data, that trick is worth remembering.

Keeping the numbers small enough to fit

Here is a pattern that shows up three separate times in the report, and once you notice it you cannot stop.

Computers store numbers with limited precision, and AI hardware deliberately uses very coarse precision because it is faster. So a calculation that produces an enormous number does not merely get slow. It breaks.

Kimi K3's summary mechanism has exactly this hazard. Old information fades a little at every step, and doing the maths correctly sometimes means undoing the fade, which means dividing by it. Fade something to almost nothing and you are dividing by almost nothing. It is the audio equivalent of reconstructing a whisper by amplifying it a trillion times. The amplifier does not cope.

The previous version handled this by detecting the dangerous cases and sending them down a slow, careful, special-purpose path.

Kimi K3 changed one setting instead. Nothing is allowed to fade below roughly 0.7% of its original strength, ever. That floor keeps the undo number inside what the hardware can hold, and with the hazard gone, the slow path was not optimised. It was deleted.

The same instinct shows up in the activation function, which had two parts that could each grow without limit. Multiply two unbounded things and eventually both spike together and blow the speakers. Kimi K3 put a limiter on both, shaped to behave identically at normal volumes and simply refuse to go past a ceiling. And again in how it was trained: most models are trained at full precision then compressed afterwards, losing a little quality. Kimi K3 trained in the compressed format from the start, so it never had to adapt to a haircut it did not see coming.

Three subsystems, one idea. At this scale, the shape of a curve is an infrastructure decision.

The proof that removed three bottlenecks

This is my favourite thing in the report, and it is about a kitchen.

Imagine 896 dishes on a menu, and a kitchen split into stations. Orders arrive unevenly, so some stations get slammed while others idle. The usual fix is floating cooks who can be sent wherever the pressure is. But how many floaters do you keep on shift?

Guess too few and one night the kitchen jams with no legal way to redistribute the work, and service stops. That is not a metaphor: in existing systems, training genuinely halts, and someone has to retune a number by hand.

Moonshot proved a floater count that mathematically cannot fail, for any distribution of orders. They also proved you cannot do better. So they staff exactly that many, and the jam becomes impossible.

Now watch what a guarantee buys you, which is the part I keep thinking about.

Because the load is always perfectly even, every station does an identical amount of work. Because the work is identical, the size of every task is known before the shift starts. Because the sizes are known in advance, nobody has to stop and ask each station "how many orders do you have right now?" before sending the next batch.

That question was itself a major delay. It happened between every single layer of the model, hundreds of times per step, and each time the whole pipeline stalled waiting for an answer.

One proof. Three separate slowdowns gone, two of them nowhere near the thing being proved.

The problem that showed up eight months late

Right. The section I promised.

Kimi K3 mixes two kinds of layer, and each remembers conversations differently.

Think of it as two ways of saving a video game. The full re-reading layers record every frame, so you can scrub to any moment. The summary layers keep one large save file that gets overwritten as you play, and writing it out is slow and expensive.

Now, when you send a follow-up message to a chatbot, it does not reprocess the whole conversation. It picks up from a saved point. That is why the second message in a chat is much faster than the first, and it is a large part of why running these things is affordable at all.

For Kimi K3, picking up midway needs both records restored at the same moment. The frames and the save file.

But the save file is expensive to write, so it only gets written occasionally. And the system had been built so that the frame-recording schedule was locked to the save-file schedule. Which meant, in effect, that the game only autosaved every 6,000 words.

Follow that through and it is quite bad. A conversation shorter than 6,000 words could never be resumed at all, ever, because it never reached a save point. Long documents streaming in produced nothing reusable until they crossed a boundary. At a million tokens, where a typical coding request carries a 400,000 token history and adds 4,000 new ones, missing the save point means redoing all 400,000. The difference between hitting and missing is not a few percent. It is orders of magnitude of cost.

None of this was chosen. It was implied by a decision about how to mix layer types, made near the beginning of the project, and it stayed invisible until real traffic arrived.

The fix is one sentence. Stop letting the save schedule dictate the frame schedule.

Record frames every 512 words. Keep writing the expensive save file rarely, but only ever at moments that are also frame boundaries, since those are the only places anyone can ask to resume from. Now a resume can happen at any 512 word mark that has both.

What follows in the report is three rules about concurrency, and each reads like it was written the morning after an outage. One says that before you copy anything you must first pin every record you matched, because otherwise the act of making your copy can delete the thing another request was mid-way through using. Another says a record created during the current scheduling round has to be ignored until its copy has actually landed, because otherwise a reader gets the previous owner's data. That is a bug you find once and remember for life.

There is also a throwaway line I enjoyed. They store two structurally different kinds of record in one shared pool, and note that because the two are so different, mixing them up produces obvious garbage rather than plausible-looking data. They call it a free sanity check. That is the voice of someone who has debugged the version where it looked plausible.

Training an agent that runs for days

Kimi K3 is built to work unsupervised for a long time. Not "answer my question", but "spend four hours doing this job". One training task might involve thousands of tool calls across several simulated days.

Training something to do that breaks the normal machinery in a few ways.

You cannot wait for everyone to finish. Some attempts take minutes, some take hours, and waiting for the slowest idles the hardware. So they stop collecting once most are done, freeze the rest exactly where they are, and resume them next round. A single long attempt therefore spreads across several training rounds and its answers arrive somewhat out of date. They handle that by permitting only small corrections from any one answer, so stale information cannot yank the model around.

The agent needs a real computer, not a pretend one. Early on they used containers, the standard lightweight way to sandbox code. Agents kept crashing the host. Not maliciously, just by exploring aggressively, and containers share too much of the underlying machine to survive that.

They could have restricted what agents were allowed to do. They deliberately did not, because hard tasks need an environment where the agent can mount a disk or start a virtual machine if it decides to. So every agent gets its own miniature computer with its own kernel, and if it destroys that one, it destroys only that one.

And then the number that reframes everything.

A sandbox spends up to 98% of its life doing nothing. It is waiting for the model to think of the next command. The workshop is rented, the apprentice is standing still, the meter is running.

So they built the ability to freeze a sandbox mid-task, releasing all its memory and CPU, and thaw it in 49 milliseconds when the next instruction arrives. Read that back with the 98% in mind. This is not an efficiency tweak around the edges. The waiting was the bill.

Scale, for a sense of proportion: across training and evaluation, Moonshot created 51,219,741 sandboxes across just over 1.5 million different machine images.

The part Moonshot did not open-source

Look at what they gave away: the full 2.8 trillion parameter weights, their expert-distribution library, their sandbox system, their attention kernels, a compiler the model itself wrote, and the design files for a chip it produced unsupervised in 48 hours. An unusually generous release.

Now look at the gap. They did not release the self-growing knowledge map their agents built to invent training tasks. Or the fake Gmail, Notion, Slack and Canvas that assistant tasks practise against, complete with weeks of simulated events. Or the graders. Or the cheat detectors.

Those last ones are the most honest thing in the document. Set a model loose on "make this code faster" with a reward for speed and it will discover that faking the measurement is easier than doing the work: caching results, replaying old runs, quietly dropping precision. So Moonshot built a detector for each cheat as it appeared and kept extending it. They also split grading in two. A public grader gives feedback, a hidden one decides the score. Practice test versus real exam. Memorising the practice answers gets you nothing.

Here is the asymmetry. Anyone can read the architecture out of the paper and build it. Nobody can rebuild 51 million environment runs, or the years of learning about how a model cheats that got baked into those detectors.

The weights were the part they could afford to give away.

What Kimi K3 has actually built

The case studies are vendor-chosen best runs, so grade them accordingly. One still lands hard.

Given 24 hours per task, the model rewrote several of its own GPU routines. One went from 283 milliseconds to 114. Another lost three quarters of its runtime. It separately wrote a compiler from scratch, and in an unattended 48 hour run designed a working prototype AI chip, which they published.

Then the line that matters more than any of those numbers: an early version of Kimi K3 was already doing most of Moonshot's routine performance work while K3 was still being built. The model was speeding up the code that runs the model, before it shipped.

What I got wrong about the price

In the launch post I looked at the pricing, three dollars per million words in and fifteen out, and concluded that the era of very cheap Chinese models was ending. The number was right. What I took from it was not.

The report measures cost per completed task instead, and the picture inverts.

On a web research benchmark, Kimi K3 posts the best score in the field at $2.03 per task. That is roughly half what its nearest competitor costs and about a tenth of what the Claude models cost at full effort. On Moonshot's internal coding benchmark it finishes four points behind the leader while costing 38% as much.

The reason is that Kimi K3 was explicitly trained to be brief. Every practice problem came with a word budget, and going over it scored worse than getting the answer wrong. Train that way and you get a model that finishes in fewer words.

Which makes price per word a bad way to compare models, in the way that comparing processors by clock speed stopped working. A consultant with a lower hourly rate is not cheaper if they take four times as long.

What this probably does to the AI world

Guesses. I have marked which ones I would actually bet on.

Long context stops being a spec and becomes a cache problem. Nearly the whole serving half of this report is about reuse: keeping a saved conversation restorable, deciding what stays in fast memory, routing a returning user back to the machine that already has their history. Nobody's bottleneck at a million tokens is raw computing power. It is whether the save point survived. I would bet the next round of competition is fought on hit rates and cost per task rather than the context number printed on the box.

Mixed attention becomes standard. Three cheap layers to one expensive one, no positional ruler, stretching to a million tokens without recalibration, shipping in the largest open model in existence. That is no longer an experiment.

The moat moves from the model to the practice ground. The one I would bet hardest. Weights are becoming the cheap, giveable part. The expensive part is the simulated world you trained them in, and everything you learned about how they cheat in it.

Per-token pricing becomes a bad comparison. The same model has a wide cost range depending on how hard you ask it to think, and that range was engineered on purpose. Expect cost-per-task charts to become normal, and some vendors to resist them.

Open weights and refusal-based safety are on a collision course. Buried in the evaluations: Kimi K3 found sixteen previously unknown security vulnerabilities in widely used software, two in the Linux kernel, one of them remotely exploitable. The report also notes that the leading American models refuse this category of work entirely, which made a direct comparison impossible.

Those two facts sit next to each other and do not resolve. Refusal is the main public safety mechanism the top labs use, and it is precisely the mechanism that cannot survive a weight release, because anyone who downloads the model can remove it over a weekend. It is a speed limiter shipped with the wrench in the glovebox. I have no prediction for what replaces it, which is roughly the problem.

Models improving their own infrastructure is already real. Not a projection. An early checkpoint was doing most of the performance work on the system that produced it. That loop is running inside at least one lab right now.

What you can borrow from all this

You will never train a 2.8 trillion parameter model. These transfer anyway.

Put the parts that change last. Kimi K3's message format puts stable settings before the conversation and per-request settings after it, purely so that changing something small does not invalidate the saved history. If your app puts today's date at the top of a long prompt, you are reprinting the whole document to change one line. Move it to the bottom.

Give expensive and cheap requests separate budgets. Production traffic here mixes 2,000 word requests with million word ones. One shared queue means a burst of the big ones starves all the small ones. Any service with mixed workloads has a small version of this.

Never evaluate an agent under only one setup. Kimi K3 was deliberately trained across five different agent frameworks so it would not learn the quirks of any one. If you only ever test yours under one, you are measuring the framework, not the agent.

FAQ

Can I run Kimi K3 myself?
Not on anything you own. Moonshot suggest 64 or more accelerators. In practice most people will use it through an API, the same way they use a closed model.

Then what does "open" actually get anyone?
Companies can host it on their own hardware and keep their data in-house. Researchers can look inside and modify it. Neither is nothing. But it does mean "open" now covers two very different things, and this kind is not the kind you run on a laptop.

Is it better than Claude or GPT?
Overall, no, and Moonshot say so directly. It trails Claude Fable 5 and GPT-5.6 Sol. It beats everything else they tested, and it leads on specific things: web research, spreadsheets, document understanding, and a human-voted leaderboard for building web pages, where it is the first open model to reach first place.

Do I have to understand any of this to use an AI model?
No. But if you are choosing between them, the price-per-task point in this post is the one worth carrying around.

Where do I read the real thing?
Linked below. It is unusually candid about failures, which is rare in a document published on launch day.

Sources

The weights and technical report on Hugging Face
The tools they released alongside it: MoonEP, AgentENV, FlashKDA
Things the model itself built: MiniTriton, nano-kpu
Release coverage: officechai

My launch-day post, if you want the before picture:

Pranav Raj

Jul 17

Kimi K3 explained in plain terms, and why "open" is getting complicated

#ai #machinelearning #llm #opensource

5 min read

If you have run K3 since the weights dropped, I would like to know whether the save-point behaviour holds up outside Moonshot's own benchmarks. That is the one claim here I have no way to check.

A Sydney server answered in 31 ms. Physics says the floor is 101 ms.

Pranav Raj — Wed, 22 Jul 2026 14:30:00 +0000

TL;DR: Physics puts a hard 101 ms floor on a Mumbai to Sydney round trip. I measured 31 ms. Working out how that was possible taught me more about CDNs than the chapter of High Performance Browser Networking that explains them. Every number here is from ping and traceroute on my own connection.

The measurement that did not make sense

I was reading the chapter on latency and bandwidth, and it makes a claim you can check yourself: light in fiber travels at roughly 200,000 km per second, so the round trip between any two points has a hard floor set by the distance between them.

Mumbai to Sydney is 10,157 km on the great circle. That gives a round-trip floor of about 101 ms in fiber, and about 68 ms even in a vacuum. Nothing you deploy, buy, or optimize gets under that.

So I pinged a university in Sydney.

$ ping -n 4 www.sydney.edu.au

Ping statistics for 2600:9000:237b:7a00:12:109e:7d40:93a1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 31ms, Maximum = 117ms, Average = 62ms

31 ms. That is a third of the physical floor.

Either the chapter was wrong, or the thing answering me was not in Sydney.

Latency is four things, not one

Before getting to the answer, the chapter's actual model is worth having, because "latency" as a single number hides everything interesting. Every hop between me and a server charges four separate delays:

Propagation delay. Distance divided by the speed of the signal in the medium. This is the one governed by physics.
Transmission delay. Time to push the bits onto the wire, set by the link's data rate and the packet size. This has nothing to do with distance.
Processing delay. Reading the header, checking for errors, deciding where the packet goes next.
Queuing delay. Time the packet sits in a buffer waiting its turn.

The split matters because the four scale with completely different things. Propagation scales with distance, and you cannot buy your way out of it. Transmission scales with link speed, and that is genuinely what more Mbps buys you. Processing and queuing scale with how busy the path is right now.

My route to google.com was 12 hops. Every router added processing and queuing, and every link between them added propagation and transmission. All four, twelve times over.

What the numbers actually looked like

I pinged a spread of hosts and checked what each one resolved to. The last column is the theoretical fiber floor for the city I believed the service was in.

Target	Resolved to	Min RTT	Avg RTT	Fiber floor
www.sydney.edu.au	`d1puhpgqrhw451.cloudfront.net`	31 ms	62 ms	101 ms (Sydney)
dev.to	`151.101.130.217` (Fastly)	26 ms	65 ms	n/a
github.com	`20.207.73.82` (Azure)	26 ms	38 ms	n/a
google.com	Google	37 ms	121 ms	n/a
ping.online.net	Scaleway, Paris	135 ms	150 ms	70 ms

The Paris host is the control. It is a real machine in a known city with no CDN in front of it. Mumbai to Paris is 7,009 km, so the floor is 70 ms, and I measured 135 ms. Roughly twice the theoretical minimum, which is exactly what the chapter predicts once you add real routing, real queues, and the fact that cables do not follow great circles.

That line held up everywhere except Sydney.

The Sydney answer

Look at the resolved column. www.sydney.edu.au is a CNAME to d1puhpgqrhw451.cloudfront.net.

My packets never went to Australia. They went to a CloudFront edge somewhere near me, which had the page already. The origin in Sydney is only involved on a cache miss, and I never caused one.

This is the whole argument for a CDN, and I understood it much better after watching physics appear to break than I did from reading the paragraph that explains it. You cannot make light faster. The only remaining move is to make the distance shorter, which means putting the bytes closer to the person asking for them.

It also means a latency number is meaningless without knowing what answered. "31 ms to Sydney" was never true. It was 31 ms to a cache.

The part I did not expect

I assumed a ping to a fixed IP would be roughly stable. It is not. Twenty back-to-back pings to google.com:

Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 37ms, Maximum = 294ms, Average = 121ms

Same destination, same route, same distance, and nearly an 8x spread between best and worst.

None of that variance is propagation. The distance was identical on every one of those twenty packets. It is queuing delay, which is the chapter's point about buffers, and it is the same mechanism behind bufferbloat: routers with oversized buffers that would rather delay your packet for 200 ms than drop it.

The useful thing I took from this:

The minimum tells you about geography. The spread tells you about load.

My 37 ms minimum to google.com says there is a Google edge close to me. The 294 ms maximum says something on the path was congested. Those are two different facts, and reporting a single average of 121 ms throws both of them away. Latency is a distribution. Measuring it once tells you close to nothing.

Traceroute lies about the middle

The chapter suggests traceroute, so I ran it. My first hop, the home router, came back at 3 ms. The second hop, my ISP's first aggregation point, returned 28 ms, 65 ms, and 259 ms on three consecutive probes. That is the last-mile problem the chapter describes, and it was the least stable part of the entire path.

One caveat I would add to the book: intermediate hop times in traceroute are not very trustworthy. Routers treat the ICMP replies that traceroute depends on as low-priority work, so a hop can look slow while forwarding real traffic perfectly fine. A hop showing 607 ms is not proof of a problem there. I trust the end-to-end ping numbers a lot more, and I use traceroute for the shape of the path rather than its timings.

Why more bandwidth would not have helped

This is the part I had backwards. I had assumed a faster connection makes pages faster, roughly linearly.

Bandwidth is a supply problem, and supply is fixable. You can light up more fiber, lay more cable, or improve the multiplexing on cable you already have. The chapter notes that between 2007 and 2011 more than half the added trans-Pacific capacity came from better equipment on the ends of existing cables.

Latency has no such move. We are already running at about two thirds of the speed of light in fiber. Perfect materials and instant routers would buy something like 30 percent, once, forever. After that the only lever left is distance.

So if a page needs six sequential round trips before it renders, and each round trip costs 60 ms, that is 360 ms you cannot buy your way out of. Doubling the bandwidth changes none of it. Cutting the round trips, or moving them closer, changes all of it.

What I am doing with this

Three things I will actually carry forward:

Check what a hostname resolves to before drawing conclusions from a latency number. Half the time you are measuring a CDN edge, not the service.
Look at the minimum and the spread separately. An average hides the two facts you wanted.
Count round trips, not kilobytes, when something feels slow.

I am reading this book chapter by chapter and writing up what survives testing on a real connection. Chapter 2 is TCP, which is where the round-trip cost I keep hand-waving at actually gets explained.

If you want to try this yourself it is three commands and about two minutes. Pick a host you believe is far away, run nslookup on it, then ping -n 20. I would like to know what your minimum RTT looks like against a CDN-fronted host versus a bare server in a city you can name, and where you are pinging from. My guess is that a lot of people are going to find their "distant" server is sitting a few kilometres away.

Book: High Performance Browser Networking by Ilya Grigorik, free to read online.

Kimi K3 explained in plain terms, and why "open" is getting complicated

Pranav Raj — Fri, 17 Jul 2026 08:12:25 +0000

TL;DR: Moonshot AI released Kimi K3 on July 16, a 2.8 trillion parameter open-weight model with a 1 million token context window. This post explains what those numbers actually mean, goes one level into the architecture (MoE, Kimi Delta Attention), and ends with my take: the era of dirt-cheap Chinese models may be ending, and "open" now means something you can inspect but probably never run.

I have not used K3 myself yet. This is me reading the launch material and independent coverage carefully so you get the picture in ten minutes instead of an afternoon.

What just happened

A Google News notification about "the largest open-source model ever" is what pulled me in. The claim sounded big enough to be worth checking properly.

Moonshot AI, the Beijing startup behind the Kimi models, released Kimi K3. The headline claims from their launch post:

2.8 trillion total parameters, which they call the first open model in the 3T class
A 1 million token context window
Native multimodality: it processes images and video in the same model, not through a bolted-on vision encoder
Benchmarks that, by Moonshot's own admission, still trail Claude Fable 5 and GPT 5.6 Sol, but beat everything else they tested, including Claude Opus 4.8 and GLM-5.2

The weights are not out yet. They are promised by July 27. Until then, nobody outside Moonshot can independently verify or run the model. Keep that in mind for everything below: these are launch-day numbers from the vendor, partially confirmed by early independent testing from Artificial Analysis.

The numbers, translated

2.8 trillion parameters does not mean 2.8 trillion at once. K3 is a Mixture of Experts (MoE) model. Think of it as 896 specialist sub-networks ("experts") sitting behind a router. For each token, the router picks 16 of those 896 to actually do the work. So the model has a huge library of knowledge on disk, but only a small slice of it fires per token. That is how you make a 2.8T model affordable to serve at all. This is the same broad trick behind most large open models today; K3 just pushes the sparsity further than usual.

A 1 million token context window means roughly 700,000+ words of input. That is an entire large codebase, or several books, in one prompt. The catch with long context has never been "can you fit it" but "can the model actually use token 900,000 when answering," and that is exactly what K3's attention changes are aimed at (next section).

"Open" here means open weights, later. You will be able to download the model and run it yourself. But at this scale, "yourself" means a serious GPU cluster. Moonshot recommends deploying on nodes with 64 or more accelerators. So it is open in the sense that researchers and companies can inspect, fine-tune, and self-host it. It is not open in the sense that you or I will run it on anything we own.

One level deeper: what is actually new in the architecture

Two named components carry the release, and both are about how information flows through the model.

Kimi Delta Attention (KDA). Standard attention gets expensive as context grows, because every token can look at every other token. KDA is Moonshot's answer for scaling attention efficiently to very long sequences. They also mention it breaks conventional prefix caching, and that they contributed a fix to vLLM, which tells you they expect people to actually serve this thing, not just benchmark it.

Attention Residuals (AttnRes). In a deep network, information from early layers has to survive dozens or hundreds of layers to influence the output. AttnRes, per Moonshot, lets the model selectively retrieve representations from earlier depths instead of accumulating everything uniformly. Intuition: rather than a message being whispered down a long chain of people, later layers can go back and re-read earlier notes directly.

On top of that, a stack of training tricks with names like Quantile Balancing (keeping the 896 experts evenly loaded without a fragile hyperparameter) and quantization-aware training in MXFP4/MXFP8 (so the released weights run efficiently on a wide range of hardware). Moonshot claims all of this together gives roughly 2.5x better scaling efficiency than their previous K2 generation, meaning more capability per unit of compute.

The full technical report is not out yet, so the honest status of every claim above is "vendor-described, plausible, unverified." The July 27 weights release is when the community gets to check.

The parts of the launch post worth raising an eyebrow at

The demos are genuinely impressive on paper: a Triton-like GPU compiler built from scratch, a chip designed in a 48-hour autonomous run, a research pipeline reproducing astrophysics results in two hours. But launch-day case studies are always the best runs, selected by the vendor. The Decoder's coverage notes that independent testing broadly confirmed the benchmark story, but also that K3's hallucination rate went up compared to its predecessor. Moonshot's own limitations section admits the model can be "excessively proactive," making decisions on your behalf when instructions are ambiguous. For an agent meant to run unsupervised for hours, that cuts both ways.

My take: two quiet shifts hiding in this release

First, cheap Chinese models may be over. K3 costs $3 per million input tokens and $15 per million output tokens. That is not DeepSeek-style pricing; that is Western mid-tier pricing. The old story was that Chinese labs compete on being 10x cheaper. K3's story is "we are nearly frontier, pay accordingly." If that holds, the pricing gap that made a lot of hobby projects viable is closing.

Second, "open source AI" is splitting into two different things. There is open-as-in-runnable, models like the smaller Llama and Qwen variants that a student with a decent GPU can actually use. And there is open-as-in-inspectable, 2.8T monsters that are technically downloadable but practically only serve as public research artifacts and enterprise self-hosting options. Both are valuable. But when headlines say "the largest open model ever," it is worth being clear that its openness will be experienced by almost everyone through an API, same as a closed model.

Where does that leave me? Honestly, nowhere near K3. It is far too big to run or experiment with locally, and I do not currently have a project that needs an AI API. Which is kind of the point: a model this "open" changes nothing about my day to day, and I suspect that is true for most individual developers reading this.

What I would watch next

July 27: do the weights actually land, and what does the technical report reveal about KDA and AttnRes?
Whether independent benchmarks hold up once anyone can run it.
DeepSeek's response, which reporting suggests is close.

If you have already tried K3 through the API or Kimi Code, I would genuinely like to hear how it compares to whatever you use now. That is the data point launch posts can't give us.

Sources: Moonshot's K3 launch post, VentureBeat, The Decoder, TechCrunch.

I built vector search by hand out of curiosity, and my toy caught a mistake I did not know I had made

Pranav Raj — Tue, 14 Jul 2026 14:20:24 +0000

TL;DR: I shipped a "second brain" app at a hackathon that searched notes by meaning, and I did not understand the search part. So I rebuilt it by hand in numpy, 20 fake products in 8 dimensions. The mechanism turned out to be about ten lines. The interesting part was that the toy quietly told me two of my four categories overlapped, which I had done to myself and never noticed. That is the thing real embeddings will never let you see.

The question that started it

At a hackathon I built second-brain, which stores notes as memories and searches them by meaning instead of exact words. It runs sentence-transformers (all-MiniLM-L6-v2) to turn text into 384-dimensional vectors, and Qdrant to store and search them. It worked. You could type a rough idea and get back the note you meant, with none of the same words in it.

But I had wired that up without knowing what it did. It was an embed() call and a Qdrant query, and it felt like magic. Magic in my own project bothers me, so I wrote down the actual question:

What happens between embed(query) and the results coming back?

Not "what is an embedding". I could recite that. I wanted the mechanism, the part that takes a pile of vectors and picks the close ones. So I built one. No model, no vector database, no framework. Just numpy, 20 made-up products across four categories (electronics, clothing, cooking, stationary), each clustered around a hand-picked center in 8-dimensional space with noise added so they were not identical.

import numpy as np

# each category is a cluster center in 8-dimensional space
electronics_center = np.array([0.9, 0.1, 0.2, 0.08, 0.1, 0.3, 0.79, 0.2])
clothing_center    = np.array([0.1, 0.8, 0.09, 0.1, 0.09, 0.2, 0.1, 0.89])
cooking_center     = np.array([0.2, 0.3, 0.95, 0.2, 0.1, 0.9, 0.3, 0.1])
stationary_center  = np.array([0.8, 0.2, 0.86, 0.9, 0.08, 0.1, 0.2, 0.1])

# real embeddings vary, so simulate that with noise around each center
noise = 0.1
electronics = electronics_center + noise * np.random.randn(5, 8)
# ...same for the other three categories

Using fake vectors was the whole point, not a shortcut. I did not want to learn how a transformer produces a vector. I wanted to isolate the half I did not understand, which is what you do with the vectors once you have them. Remember the four centers above. They come back later and they are the best part of this post.

Question 1: how do you actually compare two vectors?

The answer is short enough to be annoying. Normalize every vector to unit length, then rank by dot product. For unit vectors, the dot product is cosine similarity, because the denominator in the cosine formula is just the two magnitudes, and you already made both of them 1.

def normalize(vectors: np.ndarray) -> np.ndarray:
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    norms = np.where(norms == 0, 1e-10, norms)  # avoid divide by zero
    return vectors / norms

class VectorIndex:
    def __init__(self):
        self.vectors = None
        self.labels = None

    def add(self, vectors: np.ndarray, labels: list):
        self.vectors = normalize(vectors)   # normalize once, at index time
        self.labels = labels

    def search(self, query_vector: np.ndarray, top_k: int = 3):
        query = normalize(query_vector.reshape(1, -1))
        scores = np.dot(self.vectors, query.T).flatten()  # cosine similarity
        top = np.argsort(scores)[::-1][:top_k]
        return [(self.labels[i], float(scores[i])) for i in top]

That is the entire engine. It ran, and every query came back with the right category on top:

Query: 'Cooking'
  1. [0.9948] Ceramic pan
  2. [0.9816] Iron cast pan
  3. [0.9749] wodden spoon

Query: 'Clothing'
  1. [0.9935] Silk saree
  2. [0.9808] Oversized black t-shirt
  3. [0.9775] Women's crop top

Two things clicked here that no amount of reading had done. Normalizing is not a ritual, it is what buys you the cheap dot product later. And it belongs in add(), not search(), because you index once and you query forever.

The other thing I finally understood is why it is cosine and not plain Euclidean distance. Cosine only looks at direction, so it ignores magnitude. For text that matters, because a long document and a short one about the same topic can differ a lot in magnitude while pointing the same way. Distance would call them far apart. Cosine calls them the same thing.

Qdrant adds a great deal on top of this, and I will get to what, but the core of what my second-brain runs on is those ten lines.

Question 2: can I see the space?

8 dimensions, so no. Not directly. I projected down to 2D with PCA and plotted the products as dots and the queries as stars. The four clusters separated cleanly and each query star landed inside its cluster. Very satisfying.

Then I printed how much of the variance those two components actually captured:

PC1: 45.92%
PC2: 26.45%
Total (PC1 + PC2): 72.36%

So the picture I was happily trusting is showing me about 72 percent of the structure. More than a quarter of what distinguishes these vectors is not on that plot at all, and 8 dimensions is a toy. Real embeddings are 384 or 1536. Whatever a t-SNE or UMAP plot of those looks like, the honest reading is that it is a heavily lossy sketch, and two points sitting on top of each other in the picture may not be near each other in the space.

That was the first thing I got out of the build that I did not expect: a healthy distrust of every embedding visualization I had ever nodded along to, including my own.

Question 3: is a high score proof the result is right?

Top-3 always looked perfect, which is exactly the problem, because top-3 is all a search UI ever shows you. So I stopped looking at the top and scored the query against all twenty products, then sorted.

That is when the run got interesting. Here is the tail of each query, the highest-scoring wrong-category item:

Query	Worst in-category score	Best wrong-category score	Margin
Clothing	0.9572	0.4457 (wodden spoon)	0.51
Electronics	0.9441	0.6542 (Parker pen)	0.29
Cooking	0.9649	0.6689 (Glue stick)	0.30
Stationary	0.9704	0.7110 (Nakiri knives)	0.26

A cutoff around 0.8 separates right from wrong for every query here, so the toy did not fail. But look at the margins. Clothing has twice the breathing room that stationary does. Same code, same noise, same everything. Why would one category be twice as easy as another?

The thing I did not go looking for

The queries were generated near the cluster centers. So if some categories are getting confused with each other, maybe the centers themselves are the problem. I had never thought to check them, because I picked those numbers by hand, in about thirty seconds, to "look different". So I measured the cosine similarity between the four centers:

electronics  vs clothing     0.3318
electronics  vs cooking      0.5287
electronics  vs stationary   0.6202
clothing     vs cooking      0.3872
clothing     vs stationary   0.2911
cooking      vs stationary   0.6475

There it is. Clothing sits between 0.29 and 0.39 from everything else, off in its own corner. Stationary is 0.62 from electronics and 0.65 from cooking, wedged right between them. Go back and look at the centers I hand-wrote: stationary is high on dimension 0 (0.8), and so is electronics (0.9). Stationary is also high on dimension 2 (0.86), and so is cooking (0.95). I built a category that shares a strong axis with two others without noticing, and then I built the two other categories it overlaps with.

And the search told me. The two most-confused pairs in the search results, stationary and cooking bleeding into each other at 0.71 and 0.67, are exactly the two highest-similarity pairs in that table. The retrieval quality was a direct readout of a geometry mistake I had made and could not see.

That is the payoff of using fake vectors, and it is the opposite of what I expected. I thought the toy would teach me the algorithm and the algorithm was the boring part. What the toy actually gave me is the only setting where I get to see the ground truth of the space and the search behavior at the same time, and compare them. With all-MiniLM-L6-v2 I get 384 numbers I cannot interpret, and if two of my note categories overlap in that space, nothing tells me. The search just quietly gets worse and I blame the threshold.

What this changed about my actual app

The retrieval in second-brain is not pure similarity, and building the naive version showed me why it cannot be:

payload = {
    "vector": embed(query),         # semantic similarity
    "limit": limit,
    "score_threshold": 0.5,         # drop "near but irrelevant" hits
    "with_payload": True,
    "filter": {                     # hard constraints, not similarity
        "must": [
            {"key": "memory_type", "match": {"value": "knowledge"}},
            {"key": "importance", "range": {"gte": 0.3}},
        ]
    },
}
requests.post(f"{QDRANT_HOST}/collections/knowledge_memory/points/search", json=payload)

Cosine similarity does exactly one thing: find the nearest vector. Nearest is not "correct", and it is definitely not "allowed". So the similarity finds candidates and everything else enforces the rules. "Notes from last week tagged work" is a database query wearing a search box, not a similarity question, and no amount of embedding quality will turn it into one.

The same logic explains the failure I have hit most in practice: exact matches get worse, not better. Search a file name, an ID, or a tag, and semantic search will confidently hand you something close in meaning and bury the exact hit. Plain string matching wins there and it is not close.

I also now think that score_threshold: 0.5 is a number I made up. It works, in the sense that the app stopped returning nonsense. I have no principled reason for it, and the margin table above is why that bothers me. The gap between right and wrong moved by 2x across four categories in a dataset I built myself. In a real embedding space I have no idea what it is.

The questions I left with

The mechanism took an afternoon. These did not:

How do you pick a score threshold honestly? Mine is a vibe. Doing it properly seems to need labelled query and result pairs and a look at where the score distributions actually cross, which means I need ground truth for my own notes, which I do not have. This is the one I want to solve next.

How does the geometry change at 384 dimensions? Everything above is 8 dimensions, where I can reason about a single axis being shared. I have read that similarity scores bunch up as dimensions increase, so everything starts looking moderately similar to everything. I have not tested it, and my whole intuition here was built in a space small enough to be unrepresentative.

How does a real index avoid comparing everything? My search() is a dot product against every vector, O(n) per query, which is fine for 20 products and absurd for a million. Qdrant uses HNSW to approximate. So it can be wrong, on purpose, for speed. I would like to build that next and find out what "approximate" actually costs, because right now I take that trade on faith.

Filtering and approximate search must fight each other. If HNSW is walking a graph to find near neighbors, and I also demand importance >= 0.3, does it filter before it walks or after? Filter after and you can get back fewer results than you asked for. Filter before and it is not clear how the graph still works. I know Qdrant solves this. I do not know how.

How do you route a query to the right search? The exact-match problem has a known answer, hybrid search: run keyword and vector search, merge the rankings. What I do not have is a clean way to decide which path a query deserves, or whether you just always run both and fuse. This is where I am stuck.

If you have built hybrid retrieval in production, I would like to know how you make that routing decision, and whether you calibrated your score threshold or picked it the way I did. That is the honest state of it.

Second-brain is here:

CoffeeAurCode / second-brain-ai_MVP

second-brain-ai (qdrant-based)

this project is a minimal Second brain AI system using qdrant as the primary vector memory store

How to Run

Start Qdrant locally docker run -p 6333:6333 qdrant/qdrant
Install dependencies: pip install -r requirements.txt
Run the any of the demo file: python Demo_1.py/Demo_2.py/Demo_3.py

The demo will store knowledge, retrieve relevant memories, apply memory evolution, and retrieve again.

Dependencies

This project uses Qdrant as the primary vector search engine. Qdrant is run locally using Docker, which is the official and recommended deployment method.

Docker is required only to start the Qdrant service. All application logic runs in Python.

Memory Architecture

The system uses Qdrant as the primary memory store.

Three typed memory collections are defined:

knowledge_memory: long-term semantic knowledge(hard disk)
context_memory: short-lived working context(RAM)
interaction_memory: conversation traces and user intent(EQ)

All future retrieval and memory evolution operates exclusively on these collections.

Embeddings

Text is converted into semantic vector representations…

View on GitHub

The paper reported jumps like 21% to 97%. My replication got +2%.

Pranav Raj — Tue, 07 Jul 2026 14:43:50 +0000

TL;DR: I ran a small replication of the paper "Prompt Repetition Improves Non-Reasoning LLMs": 100 MMLU questions, one non-reasoning model, prompt sent once versus twice. Baseline 59%, repetition 61%. Probably not statistically significant, and that turned out to be the interesting part. The gap between the paper's headline numbers and my +2% taught me more about transformer attention than the trick itself.

Why I tried this

The claim sounds like a joke: paste your prompt twice and a non-reasoning LLM gets better. No chain-of-thought, no fine-tuning, no extra instructions. The paper, Prompt Repetition Improves Non-Reasoning LLMs from Google Research, tested Gemini, GPT, Claude, and Deepseek models across seven benchmarks and found repetition won 47 of 70 model-benchmark combinations with zero losses, including improvements as large as 21% to 97% on long-context tasks like NameIndex, where the model has to pull one item out of a long list.

A trick that cheap deserves a replication, and I wanted to know two things. Does it show up on ordinary short questions? And if it does, why does duplicating text change anything at all inside a transformer?

The setup

I kept it deliberately small and fixed everything I could:

Dataset: 100 multiple-choice questions sampled from MMLU science and math subjects (high school and college math, physics, chemistry, biology, abstract algebra, astronomy). Four options, one correct answer. The dataset was built once with a fixed seed and frozen to questions.json so both conditions saw identical questions.
Model: llama-3.1-8b-instant via the Groq API. A small, fast, genuinely non-reasoning model, which is the population the paper is about.
Conditions: the baseline sends the question once with "Answer with one letter." The repetition condition sends the exact same block twice (n = 2), nothing else changed.

# baseline                      # repetition (n = 2)
<Question>                      <Question>
A. ...                          A. ...
B. ...                          B. ...
C. ...                          C. ...
D. ...                          D. ...
Answer with one letter          Answer with one letter
                                <Question>
                                A. ...
                                ...
                                Answer with one letter

Results

Method	Correct	Accuracy
Baseline (n = 1)	59 / 100	59%
Repeated prompt (n = 2)	61 / 100	61%

Two more questions correct. Before calling that a win, the honest check: for paired correct/incorrect outcomes on the same questions, the right test is McNemar's, which looks only at the questions where the two conditions disagree. With 100 questions and a swing this small, the improvement is likely not statistically significant. I am not claiming the effect; I am reporting what 100 questions showed.

So the replication "worked" in the least satisfying way possible: a positive direction, too small to trust on its own. The useful part was figuring out why the effect should be small here when the paper's numbers are so large.

Why my effect was small and the paper's was big

The paper's dramatic gains come from long-context retrieval tasks. Give a model a list of 50 names and ask for the 25th, and attention has to hold position across hundreds of tokens. Attention weight spreads thin across a long sequence, the signal for any single token gets diluted, and the model misses. Repeating the prompt puts a second copy of every important token in the context, doubling the chances that the answer-relevant tokens get attended to. On tasks that stress attention, that rescue is worth a lot.

My questions were short. A four-option MCQ barely stresses attention at all; the model can already see everything clearly. Repetition amplifies a signal that was never weak, so there is not much headroom to gain. The effect size is not a property of the trick, it is a property of how starved the task is for attention. That is the actual takeaway, and it is also a testable prediction: rerun this on long-context questions and the gap should open up.

Why repetition changes anything at all

This is the part that got me to actually read about attention instead of nodding along to diagrams.

When the model predicts its answer token, attention is distributed across every prompt token. A key concept might get a weight of, say, 0.07. With the prompt repeated, each copy might get 0.05 and 0.04, but the combined signal is stronger than before. Nothing about the model changed; the input just gave the mechanism two chances to find the same information.

The KV cache view says the same thing: during inference every prompt token's key and value vectors sit in the cache, and repetition puts two copies of the relevant vectors in there. Retrieval by attention becomes more reliable when the target exists twice. Prompt repetition is signal amplification you perform from outside the model, which is also a decent one-line explanation of why prompt engineering works at all.

The paper itself adds a framing I had not considered: causal masking. In a causal LLM, tokens can only attend backwards, so a token early in your question never gets to see the tokens that come after it. In the second copy of the prompt, every token effectively has the full query behind it. Repetition is the cheapest possible workaround for one-directional attention.

Limitations, plainly

100 questions is small, and the headline result is within noise.
Short MCQs only, which the previous section argues is the worst case for this trick.
One model, one repetition factor (n = 2).

What I would run next

The obvious follow-ups, roughly in order of what I am most curious about: long-context tasks like the paper's NameIndex setup where the prediction says the effect should be large, repetition factors n = 3 to 5 to find where returns diminish, and a comparison against chain-of-thought on the same frozen dataset, since both tricks spend extra tokens to buy accuracy.

Repo with the dataset builder, notebook, and raw per-question CSVs:

CoffeeAurCode / Prompt-Repetition-Improves-Non-Reasoning-LLMs

I tried replicating the paper “Prompt Repetition Improves Non-Reasoning LLMs.”

Prompt Repetition Improves Non-Reasoning LLMs

A Small Replication Study

This repository contains a small experiment replicating the core idea from the paper:

Prompt Repetition Improves Non-Reasoning LLMs

The goal of this experiment was to understand whether repeating a prompt improves accuracy on a custom multiple-choice dataset and to explore why such a simple trick can affect transformer behavior.

Rather than focusing on achieving large gains, the purpose of this project was to investigate the mechanics of prompt repetition and understand what it reveals about transformer attention.

Motivation

Recent work has shown that repeating a prompt multiple times can significantly improve the performance of non-reasoning LLMs on certain tasks.

Example reported improvements from the paper: 21% -> 97% accuracy

on tasks requiring long context reasoning such as NameIndex.

This raises interesting questions:

Why does prompt repetition help?
Does it work on simple MCQ tasks?
What does this reveal about…

View on GitHub

If you have run prompt repetition on anything with real context length, I would like to know whether the effect showed up for you, and how large it was. And if you see a flaw in the setup that could explain the +2% besides the trick itself, that is exactly the kind of comment I am hoping for.