DEV Community: Rickesh T N

I told the model to separate fields with . It did exactly that, and I lost 79 percent of my data.

Rickesh T N — Sat, 05 Sep 2026 04:42:30 +0000

My extraction prompt said:

Output lines of the form <key><TAB><value>

The model output:

Elizabeth<TAB>I have not the pleasure of understanding you

Those are five literal characters — angle bracket, T, A, B, angle bracket — not
a tab. My parser looked for \t, found none, and dropped the line.

Across one document, 246 of 310 records were discarded. Only 5 were dropped
for the reason the mechanism actually existed to catch.

The model did what I asked. I asked for the wrong thing.

Why I did not notice for a whole benchmark run

This is the part worth stealing, because the bug is trivial and the reason it
survived is not.

My harness counted discarded records in a single field called rejected. That
field had one documented meaning: a record whose evidence could not be verified
against the source text. In other words, it was the hallucination-catching
counter.

So the run printed rejected=246 and I read it as the verifier working
hard. A parser failure was wearing the costume of a successful integrity
check. The number went up, and up was the direction I wanted.

The fix is not a better parser. It is that these two things must never share a
counter:

unparseable — our format failed, the model is not at fault
ungrounded — the model asserted something that is not in the text

One of those is my bug. The other is the model's. Summing them produces a
number that cannot be acted on, and which flatters whichever explanation you
already believe.

After splitting them, the same run reads:

records: lines=310 kept=64 | unparseable=246 ungrounded=5
separator: canonical=0 tab=64 literal_TAB=246

There is no way to misread that.

Fix one: show the delimiter, never name it

A model can reproduce a character it can see. It cannot reliably reproduce a
character you have described in prose, because your description is text and
copying text is what it does.

So instead of naming the separator, I showed a worked example using a pipe. I
picked | after checking the corpus contained zero of them in 7,987,517
characters.

Result: canonical separator compliance went from 0 to 9 out of 9 on the first
fragment, and unparseable loss went from 79 percent to zero.

Fix two: your example must be obviously not data

The first version of that fix failed differently, and worse.

I used a real line from the source novel as the worked example:

Elizabeth | I have not the pleasure of understanding you

The model copied the example verbatim and extracted nothing else. One line of
output for a 60,000-character fragment.

It looked plausible. The name was a real character, the quote was real dialogue.
The only reason I caught it was that the evidence verifier rejected it — that
sentence is not in the fragment I sent, it is from a later chapter.

The working version uses a name that cannot appear in any novel and says so:

Zolvane | we must leave before the tide turns

That line is ONLY a format example. Zolvane is not in this text; never copy
the example into your answer.

A worked example is an instruction to imitate. If your example is
indistinguishable from valid output, some fraction of the time it will be
returned as output.

Fix three: make the failure loud, at the right granularity

I added a check that aborts when more than 20 percent of output lines have no
recognised separator, printing a sample of the offending line.

My first version aborted the whole document. That was worse than the bug: the
model follows the format on most fragments and abandons it on a few, so
discarding the document threw away every good fragment along with the bad one.
Four documents produced zero records.

Now a violation fails one fragment. Good fragments survive, the failure is
still visible and still blocks a headline answer, and one document went from
zero records to 148 of 169 kept.

The forty-second diagnostic

The single highest-value thing I added was a --probe flag: run one fragment,
print the raw model reply verbatim, print the loss breakdown, exit.

It takes about seven seconds. It caught both the literal-<TAB> bug and the
example-copying bug immediately.

Before that, my shortest path to noticing a contract problem was a full
benchmark run — about forty minutes — and even then only if I read the right
counter and interpreted it correctly, which the first time around I did not.

If your pipeline sends prompts to a model in a loop, you want a way to see
exactly one of them, unedited, in under a minute. I wrote three throwaway
scripts to do that by hand before admitting it should be a flag.

Temperature 0 is not reproducible. I measured 30 percent of my output changing between identical runs.

Rickesh T N — Sat, 05 Sep 2026 04:41:53 +0000

I set temperature to 0, ran the same document through the same model twice with
the same prompt, and got different answers. Not subtly different. Roughly a third
of the extracted records changed.

The cause is not sampling. It is batching.

same document, same model, same prompt, temperature 0	records	byte-identical	Jaccard
`--concurrency 1`, run twice	137 / 137	yes	1.0000
`--concurrency 4`, run twice	160 / 164	no	0.705
concurrency 1 vs concurrency 4	137 / 160	no	0.549

Serving one request at a time is exactly reproducible. Serving four at a time is
not, and it is not reproducible against itself either.

Why this happens

Batched inference does not compute the same arithmetic as unbatched inference.
Different batch shapes select different kernels, matrix multiplications get
tiled differently, and floating-point addition is not associative. The logits
come out fractionally different, and anywhere two tokens were nearly tied,
argmax picks the other one. Temperature 0 removes sampling randomness. It does
nothing about the arithmetic underneath.

This is well known in the abstract. What surprised me is the size: 30 percent of
records, not 3.

How I found it, which is the embarrassing part

I was not looking for this. I was testing whether a new filter reduced false
positives, comparing a run with the filter against a run without it.

One name showed up as a false positive only in the filtered run.

A filter cannot add records. It can only remove them. So either my filter was
broken, or the two runs had not seen the same model output at all. It was the
second. My A/B test had been comparing two different underlying extractions and
attributing the difference to the flag.

That experiment was worthless and I nearly published its result.

The claim this retires

I had a determinism check in the repo already. It ran the same corpus five times
and confirmed the final answer was identical each time. It passed. I cited it.

It was measuring almost nothing. The pipeline extracts hundreds of records per
document and then reduces them in code to one answer. A reduction like "which of
these three names appears in the most sections" is extremely tolerant: you can
churn a third of the evidence and still land on the same name, because the
argmax has a margin.

So the final answer was stable while the thing underneath it was not. The check
confirmed the margin was wide, and I read it as confirming the system was
deterministic.

Record-level agreement is the measurement. Answer-level agreement is a
consequence, and a weak one.

What it did and did not change

I re-ran the headline benchmark at concurrency 1, where output is byte-identical.

metric	batched	deterministic
the headline metric	0.980	0.980
a positional ordering task	0.942	0.904
overall	0.899	0.870

The headline number did not move at all. That is not luck: it counts how many
distinct regions of a document a name appears in, and it does not care which
particular quoted span proves the name was in a region, only that some span
does. Redundant evidence absorbs the churn.

The positional tasks moved by up to 0.04, because those depend on the single
earliest or latest piece of evidence, and the extremes are exactly where losing
30 percent of records bites.

So the honest summary is: the noise is real, it reaches some metrics and not
others, and you cannot know which without measuring.

What to do

Report concurrency with your numbers. It is a experimental condition, not a
performance tuning detail. Two papers can differ on it and disagree for no other
reason.

Measure agreement at the record level, not the answer level. Run the same
input twice, diff the intermediate output, report Jaccard. If your pipeline has
no intermediate output to diff, that is worth fixing on its own.

Run final numbers at concurrency 1. It cost me less than I expected: about
20 minutes to about 23 for a nine-document sweep, because the server was not
parallelising as much as the flag implied. Measure the cost before assuming you
cannot afford it. Iterate batched, publish serial.

Be suspicious of any A/B where the treatment could not have caused the
difference you see. That single impossible data point was the only reason I
found this. If the filtered run had merely looked better, I would have shipped
it.

Your agent truncates the corpus and answers anyway. Two harnesses, and a router that picks between them.

Rickesh T N — Sat, 05 Sep 2026 03:58:52 +0000

I built a harness that streams an oversized corpus past a small model and aggregates in code. Then I found PrimeIntellect's prime-agent (PrimeIntellect-ai/prime-agent), installed it, pointed it at the same Ollama server on the same 6GB laptop GPU, and got PA_OK back.

They have close to 20,000 stars, a funded team, and a better implementation of the recursive-language-model idea than mine. So this is not a "we beat them" post. It is about the one line where the two architectures genuinely diverge, why that line is a question-class boundary rather than a quality one, and how to route across it.

The one line

I grepped their repo for how it handles a corpus bigger than the window.

splitText        0
chunkSize        7      (terminal-image.ts, snapshot-transcript-cache.ts)
truncate       142
summarize       73
contextWindow   73

Plus packages/coding-agent/docs/compaction.md. Their overflow strategy is compaction: truncate and summarize. Mine is exhaustive sweep: segment the whole corpus, extract from every fragment, aggregate in code.

That is nearly every agent, by the way. Compaction is the default because it is correct for the questions agents are usually asked.

Why the boundary is a question class

question	compaction	exhaustive sweep
"what does this codebase do?"	correct, and cheaper	wasteful
"where is retry logic implemented?"	correct	wasteful
"how many entries have label X?"	cannot work	correct
"which user appears least often?"	cannot work	correct
"is there any file that does Y?"	cannot work	correct

The failure in the bottom three is not that compaction is imprecise. It is that you cannot count what you summarized away, and nothing downstream can tell that it happened. Coverage looks fine. No error is raised. The agent answers confidently from the fraction it kept.

That is the actual danger: not a wrong answer, but a confident wrong answer with no signal attached.

What each side is measurably good and bad at

Everything below is measured on one 6GB RTX 3060 laptop, same Ollama server, same 4B model.

	prime-agent (compaction)	ctxstream (sweep)
overflow strategy	truncate + summarize	segment every byte, reduce in code
model's job	plan, act, summarize	extract from one fragment
aggregation	in-model	in code
agentic loop, tools, sessions	yes	none
ecosystem	~20k stars, extensions, RPC, skills	one binary
targets constrained hardware	`num_ctx`, `VRAM`, `gguf`: 0 hits	the entire premise
published benchmark numbers	none found	see below, including the losses

And my own results, which do not uniformly flatter the sweep:

workload	result
957k-char corpus, literal `category=` field	correct; Claude Opus on the same corpus: 2/3, self-inconsistent on byte-identical input
OOLONG-synth 131k, public benchmark	0.340 vs 0.428 direct — and 0.500 vs 0.513 once exact-count questions are separated out
262k tokens on 6GB VRAM	4.23–4.54 GB peak across four runs; direct ceiling on that card is 32,768 tokens
5 repeat runs, temperature 0	3 answered (all correct), 2 produced no output

The OOLONG number needs decomposing. Sweeping first scored 0.155, because OOLONG asks "which user has the most instances with the label True" — a two-dimensional group-by over labels the model must infer, while my extraction contract emitted <key><TAB><count>, one dimension. Of 27 rows, 2 were the shape it could represent. Three contract fixes later it reads 0.340.

Split by question class, the remaining gap is entirely one thing:

	direct	sweep
rank-order / comparison (n=18)	0.513	0.500
exact count (n=9)	0.257	0.02

Parity on ranking, near-total failure on counting. The model abandons enumeration partway through a fragment — per-fragment counts of [64, 69, 174, 64, 76, 162, 188, 59, 54, 172, 186, 172, 28] where each held ~154 lines. Rankings survive that because the error is roughly proportional; exact counts do not, because the answer is the number.

So: a sweep with the wrong extraction contract is worse than not sweeping, and my own corpus flattered me because I had designed the corpus and the contract together without noticing.

The 2-of-5 empty outputs are worth naming too. The harness refuses to print a headline answer when any fragment fails, because a partial sweep undercounts every key. I believe those two runs were refusals rather than failures — but I did not capture stderr per run, so I cannot prove it, and an unverified explanation is not a result.

The synthesis: route, don't choose

These do not compete. One is an agent; the other is an aggregation primitive. The agent should call the primitive when the question needs it.

Route on a single test: does the answer depend on data you might have dropped?

superlative        most / least / top / fewest
cardinality        how many / count / total
universal          every / all / any / none
group-by           per user, per label, per file
                                    -> exhaustive sweep
anything else                       -> compaction

That is a cheap classifier over the question, not the corpus, and it fails safe: routing a locate-question to a sweep is merely wasteful, while routing a count-question to compaction is silently wrong.

Concretely, in prime-agent's own extension system, the sweep is a tool:

pi.registerTool({
  name: "aggregate_corpus",
  description:
    "Answer counting, most/least, or every/any questions over a file or " +
    "directory too large for the context window. Sweeps every byte and " +
    "aggregates in code. Do NOT use for 'what does this do' questions — " +
    "normal reading is cheaper and better.",
  // shells out to a binary; returns key/count pairs
});

The agent keeps planning, editing, and tool use — which it is good at. The sweep handles the one class where dropping data is fatal. Neither pretends to be the other.

What I would tell anyone building either

Verify the context arrived. My serving layer auto-sized context up to a ceiling then silently fell back: 30,021 tokens passed intact; 50,000 and 70,000 both clipped to exactly 16,387. No error, confident answer from the remainder. Compare reported prompt tokens against what you sent and make the mismatch fatal.

A partial sweep must refuse. If any fragment fails, every count is low. Exiting non-zero beats printing a plausible number.

Run on a benchmark you did not build. This is the whole post. I had a working system, a real win against a frontier model, and a tidy story. One public benchmark showed the generality was imaginary. It cost an afternoon and was the cheapest thing I did.

Report where you lose. prime-agent publishes no benchmark numbers. I publish 0.340 against a 0.428 baseline, and a 0.02 on the one question class my approach cannot handle. One of those is more useful to you, and it is not the one that looks better.

A 4B on a 6GB laptop matched frontier-model accuracy on aggregation — except when the answer is a number

Rickesh T N — Sat, 05 Sep 2026 03:58:16 +0000

I spent a week getting a 4B model to answer questions over corpora eight times larger than its context window. It went from 0.155 to 0.340 on a public benchmark. Three of the improvements were me fixing my own bugs. The fourth thing I found is a real limit, and it splits cleanly along one line: the model can rank, but it cannot count.

	direct read	streaming harness
rank-order and comparison questions (n=18)	0.513	0.500
exact-count questions (n=9)	0.257	0.02
overall (n=27)	0.428	0.340

Parity on one class. Near-total failure on the other. The overall number hides both.

The setup

OOLONG-synth, a public long-context benchmark with an official scorer. 131,072-token contexts, 384,019 characters each. One RTX 3060 laptop, 6GB. The direct-read arm was already measured at 0.428.

The harness streams: segment the corpus in code, show the model one fragment at a time, have it emit <key><TAB><count>, aggregate in code. The model never plans a traversal and never does arithmetic across fragments.

Direct reading is not actually possible here, incidentally. The measured window ceiling on that card is 32,768 tokens — 3.3GB resident, and 65,536 jumps to 10.4GB and spills to CPU. The 0.428 baseline comes from the model reading what fits and answering from that.

Three bugs, in order

0.155. One hardcoded extraction contract for every question. OOLONG asks "which user has the most instances with the label True" — a two-dimensional group-by. My contract emitted <label><TAB><count>, one dimension. Of 27 rows, 2 were the shape it could represent. The other 25 were structurally unanswerable and every fragment reported success.

0.182. Contract now derived from the question: user-grouped questions ask fragments for <user id><TAB><count>. Marginal gain, because of the next bug.

0.340. Two fixes. First, I was routing on whether the word "user" appeared anywhere in the question — so "among User 123's entries, which label is most common" went to user-mode and answered User: 60629 for a gold of formal. The reliable signal is the stated answer format, "in the form 'Label: answer'". Second, no key validation: tallies contained 268 to 411 distinct keys where the answer space is a handful of labels. Every reduction ran on noise.

None of that is insight. It is three ways of asking the wrong question and one way of not checking the answer.

The part that is not a bug

NUMERIC_ONE_CLASS — "how many data points should be classified as label True" — scored 0.02 across 9 rows. Direct reading gets 0.257 on the same rows. Fixing contracts moved it 0.10 to 0.02, which is to say it never worked and still does not.

The traces say why. Asked to count rows in a fragment holding ~154 lines, the model returns:

[64, 69, 174, 64, 76, 162, 188, 59, 54, 172, 186, 172, 28]

That is not undercounting. It is bimodal: roughly half the fragments land near correct, the rest report about 40 percent. The model abandons enumeration partway and reports what it has.

Which explains the whole table. A rank-order question survives this, because abandonment is roughly proportional — if every label is undercounted by a similar factor, the ordering holds and argmin still returns the right label. An exact-count question does not survive it at all, because the answer is the number.

I tested the obvious fix. Smaller fragments, 60k chars down to 15k: no accuracy change, 3.6x the calls. It made abandonment more frequent, not less, because there were more fragments to abandon.

What this means if you are building one

Report per question class, not per benchmark. A single 0.340 would have told me nothing. The split — 0.500 against 0.513 on one class, 0.02 against 0.257 on another — tells me exactly which component to fix and which claim I am allowed to make.

Proportional error is survivable; absolute error is not. Design questions to need rankings rather than magnitudes wherever the task allows it. "Which label is rarest" is answerable by a model that miscounts. "How many are there" is not.

Your own corpus will flatter you. Before OOLONG, this harness answered a 957,493-character question correctly while Claude Opus, reading the same corpus in one call, was right 2 times out of 3 and contradicted itself on byte-identical input. That was a real result on a corpus where every line had a literal category= field — one-dimensional counting was sufficient by construction, because I had designed the corpus and the contract together without noticing. The public benchmark had no such courtesy.

Repeated runs are not free evidence. I ran the same corpus five times at temperature 0 expecting to report perfect reproducibility against the frontier model's 2/3. Result: three runs answered, all correct; two produced no output at all. I believe those two were the harness refusing to answer after a fragment failed, since it exits non-zero rather than print an undercount — but I did not capture stderr per run, so I cannot prove it, and an unverified explanation is not a result.

Where it stands

The streaming harness now beats the recursive-LM harness I started with (0.269) and matches direct reading on rank-order questions, on hardware where direct reading physically cannot see more than a quarter of the corpus.

It cannot count. Everything else in the table follows from that one fact.

Our 4B beat Claude Opus on a 440K-token corpus. Then it came last on the public benchmark.

Rickesh T N — Sat, 05 Sep 2026 03:57:39 +0000

Two results from the same system, three weeks apart, and the second one is the useful one.

Result A. A 4B model on a 6GB laptop GPU answered a 440,000-token aggregation question correctly. Claude Opus 4.8, reading the same corpus in one call, got it right 2 times out of 3 and contradicted itself on byte-identical input.

Result B. We ran the same system on OOLONG, a published long-context benchmark, against arms we had already measured. It came last by a wide margin.

arm	n	score
Haiku 4.5, direct	27	0.428
Haiku 4.5 + recursive harness	32	0.269
our streaming harness	27	0.155

Result A is the one you would put on a landing page. Result B is the one that taught us something, and it turned out not to mean what the number suggests.

What the system does

The failure we started from: a 4B asked to answer a question over a corpus 8x larger than its context window would sweep every fragment correctly and then, asked which label was least common, reply

Status: beta, Status: delta, Status: gamma, Status: alpha

It listed the candidates instead of selecting one. An earlier variant read a third of the corpus and narrated what it saw.

Neither is a knowledge problem. We were asking one model to plan a traversal, extract from text, and do arithmetic across 65 partial results, inside a loop. So we split those apart, borrowing the shape from video streaming:

video	here
manifest	segment plan computed in code, before any model call
buffer	N fragments in flight
decoder	model sees ONE fragment, emits `key<TAB>number`, never prose
playback	reduce — aggregation in code, strategy chosen explicitly

The model only extracts. Planning is deterministic, summing is a loop, selecting a minimum is a comparison. On our own 957,493-character corpus that produced the correct answer in 17 fragments, 611 parsed records, 3 unparseable lines, zero failures.

Then we ran it on someone else's benchmark

Self-graded results on a corpus you built yourself are worth very little. OOLONG-synth is public, published, and ships an official scorer we ported verbatim. We had two arms on it already.

We scored 0.155. Worst of the three.

The instinct is to explain that away. Here is what the trace actually said:

MOST_FREQ   keys=406  answer: "Label: Male parent (also used as a term of address to your father)"
                      gold:   72232

Our harness emitted 406 distinct keys on a task whose answer is a user ID. Median across rows: 119 keys. It was extracting sentence fragments as if they were labels.

Then we looked at what the questions actually ask:

In the above data, which user has the most instances with the label True?

That is a two-dimensional aggregation. Group by user, filter by a label the model has to infer, count. Our extraction contract emits <key><TAB><number> — one dimension. It cannot express "user × label → count" at all.

Of the 27 rows, 10 wanted a number, 8 a comparison, 7 a user ID, and 2 were the label-frequency shape our contract was designed for.

So 0.155 does not measure "streaming is worse than direct reading." It measures one hardcoded extraction contract applied to six question types, 25 of which it structurally could not represent. That is our bug, and the benchmark found it in a way our own corpus never would have.

Why this is the more valuable result

The bespoke corpus flattered us because we had, without noticing, designed the corpus and the contract together. Every line had a literal category= field. One-dimensional counting was sufficient by construction.

A public benchmark had no such courtesy. It contains question shapes we did not anticipate, and it exposed that our "general" harness was a specialised one wearing a general interface.

The fix is not a better prompt. It is that the extraction contract has to be derived from the question — a group-by question needs a <groupkey><TAB><subkey><TAB><count> contract — and a harness that ships one fixed contract will silently score near zero on anything shaped differently. Silently, because every fragment succeeded. Coverage was 1.00 and fragment errors were 0 on every row. Nothing failed. It just answered a question nobody asked.

The other thing that did not work

We also implemented the outer loop from Meta-Harness (Lee et al., 2026): an agentic proposer that reads prior candidates' source, scores and traces from disk and proposes new harness code.

Three candidates in:

candidate	accuracy	calls
baseline	0.50	16
smaller fragments	0.50	58
self-reported checksum	0.25	16

Zero improvements. One regression. 3.6x the cost for nothing.

The traces were still worth having. The counts a fragment reported were bimodal — [64, 69, 174, 64, 76, 162, 188, 59, 54, 172, 186, 172, 28] where every fragment held ~154 lines. Not uniform undercounting: about half the fragments get abandoned partway and the model reports what it had. That is why smaller fragments did not help, and it is invisible in the score.

Which is the paper's actual claim, landing in the least flattering way available: rich access to prior experience beats compressed feedback. The score said 0.50 twice. The trace said why.

Three things worth stealing

Verify context reached the model. Our serving layer auto-sized context up to a ceiling, then silently fell back: 30,021 tokens passed intact, 50,000 and 70,000 both clipped to exactly 16,387 — no error, confident answer from the fragment it kept. Compare reported prompt tokens against what you sent, and make the mismatch fatal.

Never score with substring matching. One of our runs reported "correct" because the model dumped raw corpus rows and the dump happened to contain the gold label.

Run on a benchmark you did not build. This is the whole post. The paired OOLONG run data behind these numbers is public: huggingface.co/datasets/Rickesh/rlm-oolong-reproduction. We had a working system, a real win, and a plausible story. One public benchmark showed the generality was imaginary. That cost an afternoon and it was the cheapest thing we did.

The system genuinely does what Result A says on the workload it was built for. What we cannot yet claim is that it generalises — and we would have shipped that claim if we had stopped at our own corpus.

My inference server decided my second GPU no longer exists. Here is how I got it back without upgrading a driver.

Rickesh T N — Mon, 24 Aug 2026 23:29:36 +0000

My desktop runs two mismatched GPUs: a 20GB Ampere card and a 16GB Pascal Tesla. For months, a 21GB vision-language model ran split across both at full GPU speed. Then a routine upgrade of the inference server dropped Pascal support behind a driver-version gate, demoted the old card to a Vulkan device the scheduler refuses to mix with CUDA, and silently rescheduled my model to one GPU plus CPU spill.

Nothing errored. The model still answered. It just answered at 20 to 40 seconds per item instead of 8, which turned a 12-hour benchmark into a 60-hour one. The only evidence was a log line saying the driver was "too old" for a card that had been running CUDA workloads that same morning.

The gate is policy, not physics

The interesting discovery came from reading the server's own startup logs: the bundled CUDA kernels still listed the Pascal compute capability in their build targets. The kernels existed. The Go-side scheduler was refusing to use them based on a driver-version check, while the C++ inference runtime underneath had no such opinion.

That asymmetry is the exploit. The upgraded server ships its actual inference engine as a standalone binary, a vendored build of llama-server, with its CUDA backend as a dynamically loaded library sitting in a subdirectory. The driver gate lives entirely in the scheduler process that launches it.

So: launch the vendored binary directly.

Two mechanics matter. First, the dynamic backend is not found automatically because it lives in a subdirectory the loader does not scan; it needs an explicit environment variable pointing at the .so file itself, not the directory. Get that wrong and the binary silently falls back to CPU while still accepting requests, which cost me one confusing benchmark restart. Second, the vendored build understands the server's own single-file model format, weights and vision projector packed together, which upstream llama.cpp of the same vintage refuses to load. The vendored binary logged "detected combined format, translating" and just worked.

Tensor splits are a policy decision too

With both GPUs visible again, the default split put layers proportional to total VRAM. That is wrong for mismatched cards: the Ampere card is roughly four times faster per layer, so every layer on the Pascal card costs four on the fast one. Skewing the split heavily toward the fast card, leaving the slow card holding only what does not fit, took per-item latency from 17 seconds to 9. Combined with a bigger prefill batch, the full benchmark ran in 12 hours instead of a projected 60-plus.

The general shape of the fix:

The scheduler said no; the runtime said yes. When a managed serving layer refuses your hardware, check whether the engine underneath actually shares the objection.
Point the backend loader at the exact library file and verify placement with nvidia-smi before trusting any run. A server that fell back to CPU serves identical responses, slower.
On mismatched GPUs, split by throughput, not by capacity. Free VRAM on a slow card is not free.

The clean fix is a driver upgrade, and it is scheduled. But the bypass took forty minutes including the two failed attempts, needed no root, touched nothing system-wide, and taught me more about the serving stack than a year of it working silently ever did. The scheduler is there to protect the average user from edge cases. If you can read its logs, you are allowed to disagree with it.

A generic fine-tuning playbook, written after doing it wrong several times

Rickesh T N — Mon, 24 Aug 2026 23:29:00 +0000

Every fine-tuning guide I read before my first serious attempt was a tutorial about knobs: learning rates, LoRA ranks, quantization settings. None of them covered the part that actually decides whether the project succeeds, which happens before and after the training run, not during it. This is the playbook I now follow for any model on any task. It is deliberately generic: the same sequence has carried me through vision-language models on driving data, small text models for domain QA, and RL-style preference tuning, on hardware ranging from a single consumer GPU to rented cloud boxes.

Step 0: try not to fine-tune

Fine-tuning is the most expensive intervention in the stack, so it goes last. The ladder, cheapest first:

Better prompting. A system prompt with three good few-shot examples routinely closes half the gap that people reach for fine-tuning to close. It costs an afternoon.
Retrieval. If the failure is missing knowledge rather than missing behavior, RAG beats weights. Knowledge changes; your fine-tune will not.
A bigger or different base model. Run the comparison honestly: a zero-shot larger model against your imagined fine-tuned smaller one. I have watched an untouched open-weights model beat an in-domain fine-tune on the fine-tune's own benchmark. It happens more often than leaderboards suggest.
Fine-tune. Only when the behavior you need is demonstrably not in the base model and cannot be retrieved or prompted in: output format compliance, domain-specific reasoning patterns, a persona that must survive thousands of turns, latency budgets that force a small model.

Write down, in one sentence, which failure of steps 1 to 3 justifies the fine-tune. If you cannot write that sentence, stop. That sentence also becomes your evaluation target later, which is the real reason to write it.

Step 1: build the evaluation before the dataset

This ordering feels backwards and is the single highest-leverage decision in the playbook.

Before collecting training data, build a held-out evaluation that measures the sentence from step 0, and run the base model through it. That number is your baseline, and it does three jobs. It tells you the true size of the gap. It occasionally kills the project on the spot because the base model was already good enough and nobody had measured it. And it validates the harness itself: an evaluation that has never scored a known model is untested code that emits numbers.

Two rules for the eval that I no longer break:

The test split is designed before training, and nothing from training may touch it. Not for hyperparameter selection, not for checkpoint picking, not once. Use a validation split for those. If your splits share source documents, scenes, or sessions with the training set, you are measuring memorization and calling it generalization.
Run a sanity control. Delete the input and measure again. If a vision model scores far above chance with the images removed, your benchmark leaks answers through priors and question phrasing, and every score it has produced is inflated. The same control exists for text: shuffle the context, drop the retrieved passages, feed the question alone. Cheap to run, devastating when it fires, and better fired at you than at a reviewer.

Step 2: data is the model

The dataset decides what you get. The recipe decides only how efficiently you get it.

A few thousand excellent examples beat a hundred thousand scraped ones for behavior tuning. For format compliance and persona, hundreds can be enough with LoRA.
Deduplicate against your evaluation. Near-duplicates leak. Exact-match dedup is not sufficient; hash at the level the data actually repeats (documents, scenes, sessions).
Audit a random hundred by hand. Not the first hundred, a random hundred. The first hundred were curated by whoever built the file; the random hundred tell you the truth about label noise. Every bad label teaches the model confidently.
Match the training distribution to the inference distribution. If production inputs will be messy, OCR-damaged, or truncated, train on that, not on the clean version.

Expect data work to consume more than half the project's wall-clock. When it does not, that is usually a sign it was skipped, and the bill arrives later, denominated in GPU-hours.

Step 3: choose the lightest recipe that can express the change

LoRA or QLoRA is the default. Behavior shaping, format compliance, domain adaptation, personas: adapters handle all of it at a fraction of the memory, and the artifact is small enough to version, ship, stack, and roll back.
Full fine-tuning is for when the change is deep: new modalities, new tokenizers, or when adapters measurably plateau below target. Prove the plateau before paying for the parameters.
Preference or RL methods (DPO, GRPO and relatives) are for objectives that supervised examples cannot express: relative quality, verifiable rewards, multi-step outcomes. They are also where silent failure lives, so they come with an extra rule: track a metric that measures the actual objective, not a proxy. I once watched token accuracy sit at 99 percent while the policy learned the opposite of the intended behavior. The proxy was fine; the behavior was not.

One decision that outranks the recipe choice: change one variable at a time. A run that changes base model, dataset, rank, and learning rate simultaneously produces a result that cannot be attributed to anything. Reviewers catch this in papers; production catches it in incidents. If you must move fast, move fast serially.

Step 4: engineer the run like it will be interrupted, because it will be

Training runs die. Disks unmount, drivers hiccup, a colleague's job lands on your GPU, spot instances vanish. The runs that survive share the same boring infrastructure:

Checkpoint on a fixed cadence and verify resume actually works before the long run, not during the outage. A checkpoint you have never resumed from is a hope, not a checkpoint.
Log to an experiment tracker, not a terminal. Scalars, config, git commit, environment. The question you will ask in three weeks is "what exactly produced this file", and scrollback does not answer it.
Emit progress with the failure count first. A silent job that is working and a silent job that is failing on every batch look identical from outside. If the error count is climbing, you want to see it in the first minute, not after the run completes.
Pin the environment. Container images beat requirements files; requirements files beat memory. Half of my hardest debugging sessions were two libraries disagreeing about something as small as position ids, and the fix was environmental, not algorithmic.

Step 5: evaluate like an adversary, then decide

When training finishes, resist the demo. Run the same held-out evaluation from step 1, and read three numbers together, never one:

Target metric versus base model. Did the gap from step 0 actually close, with the improvement larger than your seed-to-seed noise? If you have not measured seed noise, run the fine-tune twice before believing any margin smaller than a point or two. Single-seed margins evaporate embarrassingly often.
Regression suite. What did the model lose? General capability regressions are the default outcome of narrow fine-tuning, not the exception. A small fixed battery of out-of-domain checks is enough to catch the worst of it.
Train-versus-held-out gap. Large gap: you memorized; get more data or regularize. No gap and no improvement: capacity or recipe; move up the ladder from step 3.

Then make the deployment decision with the same honesty as step 0: is the fine-tuned model better than the best non-fine-tuned alternative, at the quality, latency, and cost that production actually needs? Sometimes the answer is no, and the fine-tune becomes a lesson rather than a deployment. That outcome is not a failure of the playbook. Discovering it for the price of one training run instead of one production incident is the playbook working.

The one-page version

Prompt, retrieve, or upgrade the base model first; fine-tune last, and write the sentence that justifies it.
Build the eval and score the base model before touching training data. Design the splits before training. Run the deleted-input control.
Spend most of the project on data: dedup against eval, audit random samples, match production distribution.
Default to adapters; escalate only on proven plateaus; one variable per run.
Checkpoint, track, and emit errors-first progress; pin the environment.
Judge with three numbers: target delta versus noise, regressions, generalization gap. Then decide like you have not already sunk the cost, because the model does not care that you did.

A 4B model on a 6GB laptop beat Claude Opus on our 440K-token corpus. The fix was giving the model less to do.

Rickesh T N — Mon, 24 Aug 2026 23:28:23 +0000

Private AI has a hardware story nobody measures honestly. The pitch is that your data never leaves the building. The unstated cost is that the building contains a 6GB laptop GPU, and the corpus is 440,000 tokens that a frontier model would swallow in one call.

We measured the whole thing. One RTX 3060 Laptop, 6144 MiB. One 2,625-record driving-QA dataset rendered to 957,493 characters. One question with a single correct answer: which category is least common. Ground truth computed in Python, so neither model gets to define success.

The 4B won. It took four failed attempts to understand why.

The context problem was never the hard part

Direct context on that card, measured with flash attention and a q8_0 KV cache:

num_ctx	resident	fits 5.5GB usable
8,192	3.2 GB	yes
16,384	3.3 GB	yes
32,768	3.3 GB	yes, the ceiling
65,536	10.4 GB	no, 31 percent on GPU
131,072	10.9 GB	no

KV is nearly free to 32k, then the allocator falls off a cliff. Switching the KV cache to q4_0 changed those numbers not at all, so the cliff is not the KV cache and you cannot quantize your way past it. Direct ceiling: 32,768 tokens.

Now stream a 261,226-token corpus through the same card in chunks, keeping the text in CPU RAM and only ever showing the model a fragment. Peak GPU across four runs: 4.23, 4.24, 4.25, 4.54 GB. Every one fits.

That is an 8x context multiple at constant VRAM. Constant is the load-bearing word: the corpus never enters the KV cache, so the limit stops being memory and becomes wall-clock.

Which is a solved problem. The unsolved one was that the answers were wrong.

Four ways to fail the same question

attempt	sub-calls	answer
1	7	prose describing the data, having read about a third
2	11	`Spatial` — right arithmetic, truncated label
3	65	`Counterfactual` — swept everything, well-formed, wrong
4	72	`Status: beta, Status: delta, Status: gamma, Status: alpha`

Attempt 4 is the one that gives it away. Asked which label was least common, the model listed all four candidates instead of selecting one. Attempt 3 had read all 425,054 input tokens and still combined the partial counts wrongly.

We had been calling this a capability gap. It was not. A 4B can count rows in a fragment. What it cannot reliably do is plan a traversal and then perform arithmetic across 65 partial results — and we had been asking it to do both, inside a loop, while also formatting an answer.

The fix: treat the corpus like a video stream

Nothing about aggregation requires a language model. So we stopped asking one.

video streaming	the port
manifest / playlist	segment plan, computed in code before any model call
buffer	N segments in flight, latency hidden behind compute
decoder	model sees ONE segment, emits `key<TAB>number`, never prose
playback	reduce phase — aggregation in code, strategy chosen explicitly

The model's entire job becomes extraction from a window it comfortably fits. Planning is deterministic. Summation is a loop. Selecting the minimum is one comparison.

Result on the same corpus, same 4B, same 6GB card:

17 segments · failed=0 · records=611 · unparsed_lines=3 · keys=15 · 641s
Category: Spatial Relationship

Correct. 15 keys recovered, matching the 15 real categories. Of 611+ emitted lines, 3 failed the output contract and were counted as failures rather than silently dropped.

The frontier comparison, which is not flattering to the frontier

We used Claude Opus over the full 439,742-token context as the reference. It answered the same question two different ways on byte-identical input — correct once, wrong once — and landed 2 out of 3 across three samples, at $4.79 per call.

	answer	correct	cost
Opus 4.8, one call, 439,742 tok	varies by run	2/3	$4.79
4B + streaming, 6GB laptop	`Spatial Relationship`	yes	$0.00

A reference that disagrees with itself is not an oracle. Ground truth has to come from code, with the frontier model scored as just another candidate.

For private AI this cuts two ways. The bar is lower than the marketing implies, because context rot is real and frontier models are not deterministic at 400K tokens. But "matches the frontier model" is also the wrong success criterion. Match the ground truth, and measure both.

What we would tell anyone building this

Recursion is not free. On a benchmark where the context already fit the model's window, wrapping the same model in a recursive harness scored 0.269 against 0.428 for a plain direct read. Recursion only pays when the context genuinely does not fit. Reach for it as a last resort, not a default. That comparison is from our OOLONG reproduction, and the paired run data is public: huggingface.co/datasets/Rickesh/rlm-oolong-reproduction.

Silent truncation is the dangerous failure. Our serving layer auto-sized context to the prompt up to a ceiling, then quietly fell back: 30,021 tokens passed intact, 50,000 and 70,000 both clipped to exactly 16,387, with no error and a confident answer from the fragment. Compare processed-token counts against what you sent, and make the mismatch fatal.

Substring scoring manufactures success. One run reported correct because the model dumped raw corpus rows and the dump happened to contain the gold label. Require the declared answer form and reject anything that looks like regurgitated input.

Report what you could not parse. unparsed_lines=3 is the number that makes the rest of the output trustworthy. A harness that silently drops what it cannot read will happily report a clean answer over half the data.

A partial sweep must refuse to answer. If any segment fails, every key is undercounted. Ours exits non-zero rather than printing a number that looks fine.

The engine is C++17 with zero third-party dependencies, and the planning and reduction stages are covered by 61 tests that need no GPU, no network and no tokens — because once the model is only doing extraction, everything else is ordinary code you can actually test.

99% token accuracy, zero learning. Field notes from fine-tuning vision models with RL.

Rickesh T N — Sun, 23 Aug 2026 23:33:17 +0000

Over the past year I have been fine-tuning open vision-language models - 9B dense up to a 35B mixture-of-experts - with supervised fine-tuning and GRPO-style reinforcement learning on verifiable rewards. Most of what I learned was not about algorithms. It was about the ways a training run can look healthy while doing nothing, or crash for reasons that have nothing to do with your code.

Three failures, in increasing order of how long they fooled me.

Failure 1: the metric that measured the wrong thing (18 hours)

I ran an 18-hour supervised fine-tune that reported token accuracy climbing steadily to 99%. Looked like a textbook run. The real evaluation metric - accuracy on multiple-choice questions - never moved.

The cause was a mismatch between what I supervised and what I evaluated. The training loss was over free-text reasoning traces; the evaluation scored a single extracted answer letter. The model got extremely good at reproducing the shape of the training text - hence 99% token accuracy - without that transferring to the decision I actually cared about.

Token accuracy is a proxy, and proxies drift from the target exactly when you stop checking. The fix was structural, not a hyperparameter: supervise the thing you evaluate. If the deliverable is a constrained answer, the training signal has to reach that answer, not just the prose around it.

The general rule I took: any training metric that is not your evaluation metric is a hypothesis about correlation, and you should check that correlation before you spend GPU-days on it.

Failure 2: the crash that was two libraries disagreeing about position ids

The GRPO trainer for the 9B vision model crashed in the forward pass, deep inside rotary position embedding code. Nothing in my training code had changed.

The diagnosis took a while because the bug lived at the boundary between components: the text sequence length was derived from token-type ids, while the vision sequence length came from the image grid - and image-pad tokens ended up counted twice. Two parts of the same stack, each internally consistent, disagreeing about how long the input was.

For the 35B MoE variant of the same family, an equivalent rope bug was fixable by monkeypatching the model's position-id computation. I shipped the patch with a GPU-free regression test: a tiny script that constructs the exact failing input shape and runs just the position-id path on CPU. It runs in seconds, needs no cluster, and fails loudly if an upstream update reintroduces the bug.

Two lessons. First, when you fine-tune at the edge of a model family's tooling support, the bugs you hit are integration bugs, and the stack trace points at the victim, not the culprit. Second, every monkeypatch deserves a regression test that costs nothing to run - otherwise the next library upgrade silently un-fixes it.

Failure 3: the RL loop that was learning the opposite (the quiet one)

In a separate project I fine-tune a 9B model with reinforcement learning where the reward comes from realized real-world outcomes rather than a labelled dataset. For a long stretch the training signal was flat - not diverging, not collapsing, just flat, which is the least informative failure there is.

Two compounding problems. One was label noise in the reward pipeline: some outcomes were being attributed to the wrong decisions, which dilutes any gradient. The other was worse: a sign error meant part of the advantage signal was inverted. The model was being gently pushed away from behaviour that had worked.

Nothing crashed. Every batch processed. Every log line looked like a training run. The only symptom was the absence of learning, and the only way I found it was working backwards from "the held-out metric should have moved by now" to auditing every stage of the reward computation by hand.

After both fixes I got the first genuinely monotonic learning curve on that task. I still treat it as training signal only - the decider is held-out evaluation against the base model, and I do not report improvements that exist only in the training curve.

The harness rules I now run everything under

These came out of the failures above plus a benchmarking programme across 70+ vision-language models. They are boring, and they are the difference between numbers and noise.

Smoke test before committing compute. A five-step GRPO run with two numbers watched: the PPO-style clip ratio and the fraction of outputs that parse. If the clip ratio is degenerate or parseability is low, the full run will be garbage in a way five steps already reveals.

Runs are gated fail-closed. Nothing publishes a result unless the evaluation stage actually scored. "The eval crashed but training finished" is not a result; it is an unscored run, and unscored runs must be impossible to mistake for scored ones.

Infrastructure failures and poor performance are different columns. An unparseable output, an OOM, a crashed kernel - these are exceptions whose count must be exactly zero. A weak model produces zero exceptions and simply scores badly. If a quality threshold can absorb an infrastructure failure, a totally broken run can pass your gate.

The held-out metric is the only decider. Training curves, token accuracy, reward trends - all of it is telemetry. If the held-out number did not move, nothing happened.

None of this is novel. All of it is the difference between the runs I trust and the 18 hours I lost.

I write about ML evaluation, world models, and the ways measurement quietly fails. More at dev.to/rickeshtn.

Your multi-agent system isn't hitting prompt cache. Your system prompt is the reason.

Rickesh T N — Tue, 11 Aug 2026 21:49:39 +0000

I run a multi-agent setup where ten agents analyse the same input. Same document, same market data, same everything. The only difference between them is persona: each one is instructed to look at the material through a different lens.

Ten agents, one shared context. That should be the ideal case for prompt caching. Send the expensive context once, pay full price for it once, and let the other nine reads come back at a fraction of the cost.

My cache hit rate was zero percent on three of the five models I was using, and under seven percent on the other two.

I had been reading the bill for a while and optimising the wrong thing. Here is what was actually happening, because the mistake is structural and I doubt I am the only one making it.

What prompt caching actually matches on

Hosted inference providers cache on a prefix. The provider hashes your request from the first token forward and looks for the longest run it has already computed. If your request starts with the same 3,000 tokens as a recent one, those 3,000 tokens are a cache read, typically around five times cheaper than a fresh read. The moment the token stream diverges, caching stops for the rest of the request. There is no re-syncing later.

That word — prefix — is doing all the work, and I had not thought about it carefully.

The setup that broke it

My call looked like every example in every SDK doc:

messages = [
    {"role": "system", "content": agent_persona},   # differs per agent
    {"role": "user",   "content": shared_context},  # identical for all 10
]

The persona is short. A couple of hundred tokens describing how this particular agent should reason. The shared context is large: several thousand tokens of source material.

Read that message array as a flat token stream, which is what the provider does. The first thing in the stream is the persona. The persona is different for every agent. So the prefix diverges at roughly token one, and the several thousand tokens of identical context sitting behind it can never match anything.

Ten agents. Ten identical copies of the same context. Ten full-price reads.

Proving it rather than assuming it

I did not want to guess, so I hashed both halves of every call for a single work item and counted the distinct values.

SELECT
  COUNT(DISTINCT prompt_sha256) AS distinct_user,
  COUNT(DISTINCT system_sha256) AS distinct_system
FROM agent_calls
WHERE item_id = ?

The answer:

distinct_user   = 1
distinct_system = 10

One user prompt. Ten system prompts. The expensive half was byte-identical across all ten calls, and the cheap half in front of it was unique every time.

This lines up exactly with the provider's own usage report, which broke my spend into cached and uncached input tokens:

model	cached share of input
A	0.0%
B	0.0%
C	3.4%
D	6.2%
E	11.0%

Those low non-zero numbers are incidental collisions between unrelated calls, not the structural reuse I should have been getting. If the design were right, nine out of every ten context reads would be cache hits.

The fix

Put the shared, expensive, identical part first. Put the small, varying part last.

messages = [
    {"role": "system", "content": shared_context},          # identical -> caches
    {"role": "user",   "content": f"{agent_persona}\n\n{question}"},  # varies, small, last
]

Now the first several thousand tokens are the same for all ten agents. The first agent pays full price and warms the cache. The other nine read it back at cache rates. The only uncached part is the couple of hundred persona tokens at the tail, which is what you actually want to be paying for.

The general rule, which I now think should be a design constraint rather than an optimisation:

Order your prompt from most shared to most specific. Caching rewards a stable prefix, and every byte that varies early poisons everything after it.

This also composes with how you batch. If you run the same model across many items back to back, you keep hitting a warm prefix. If you round-robin across models for each item, you cold-start the cache on every single call. Grouping by model, not by work item, keeps the cache warm.

The part I am not comfortable with

Moving the persona out of system and into user is not free.

Some models weight system instructions more strongly than user content. That is often the point of a system prompt. If one of my agents is specifically instructed to argue an unpopular position, and I demote that instruction from system to user, it may hedge more. I would be trading spend for behaviour, and I would not necessarily notice, because the output would still be well-formed and plausible.

So this is not a change I would ship straight to production off the back of a cost argument. It needs an A/B on a sample of items, comparing the actual decisions each layout produces, not just checking that the responses parse.

There is a middle path worth trying first: keep a short stable instruction in system that is identical across all agents, and move only the per-agent differentiation into the user message. You get a shared prefix and keep a system-role framing. Whether that is enough depends on how much of your agents' behaviour hangs off the system role, which is an empirical question about your prompts and your models.

What I would take from this

Prefix means prefix. Anything that varies early destroys caching for everything after it, no matter how much identical material follows.
Instrument it. Hash the components of your requests and count distinct values per work item. It took one query to turn a vague suspicion into a definite structural bug.
Read the cached-versus-uncached split in your usage report. A near-zero cache rate on a workload with obvious shared context is not a pricing quirk. It is a design bug, and it is telling you the prefix is broken.
The default SDK message shape is not cache-aware. Persona-in-system, content-in-user is the shape in every tutorial. It is exactly wrong for fan-out workloads where many personas share one context.

I had spent real effort choosing cheaper models before I checked whether I was paying for the same tokens ten times over. The model swap was worth doing. It was also the second-biggest lever, and I found it first because it was the one I was looking for.

Addendum: what the comments corrected

This post got better feedback than it deserved, and three points are important enough to belong in the body rather than below it.

There is a third option I missed, and it is the right one where available. @skillselion pointed out that on APIs with explicit cache breakpoints, the system prompt can be an array of content blocks: shared context as the first block carrying a cache_control marker, persona as a second block after it. The cached prefix ends at the marker, so every agent hits the same cache while the persona keeps its system-role framing. That dissolves the trade-off I spent a section being uncomfortable about. The role-versus-cost tension is an artifact of implicit prefix caching, not something inherent to the problem. Where the provider gives you breakpoints, use them; my middle path is for providers that only do implicit prefix matching.

"Put the shared context first" is necessary but not sufficient. @max_quimby and @kartik-nvjk both made the same point: the shared prefix has to be byte-identical across calls. A per-run request id, a timestamp, or a reordered JSON key injected anywhere before the boundary resets the prefix, and you are back to zero with a layout that looks correct. Enforce a canonical serialization for the cached block.

The prefix starts earlier than your message array. @mads_hansen noted that "identical text" is not the same as an identical provider prefix — tool schemas, response formats, model ids and SDK serialization defaults can all sit ahead of or inside your visible messages. Worth confirming against your provider's documented render order; on Anthropic's API, for instance, it is tools, then system, then messages, which means an unstable tool list invalidates the cache before your system prompt is even reached. Hash the canonical request envelope, not just the parts you wrote by hand.

Mads also proposed a three-layer layout I have adopted: stable policy shared by every agent, stable source context shared by the work item, then the small role and question suffix — each versioned separately, so a persona change invalidates only the cheap tail while a policy change invalidates the shared prefix deliberately. That turns cache behaviour into an observable property of the prompt architecture instead of a billing surprise.

Can a Cheap Model Beat a Frontier Model? Rebuilding Recursive Language Models with Codex

Rickesh T N — Sun, 09 Aug 2026 15:05:19 +0000

Large language models have enormous context windows now. That does not mean they use all of that context reliably.

As prompts grow, models can miss details, lose track of relationships, or produce plausible summaries instead of doing the exhaustive work a question requires. The Recursive Language Models (RLM) paper proposes a different interface: keep the large context outside the model, expose it as a variable in a persistent programming environment, and let the model inspect, partition, and recursively query smaller pieces.

We rebuilt that method with an unusual constraint:

no OPENAI_API_KEY;
Codex CLI as the model backend;
gpt-5.4-mini for both the RLM root and every subcall;
a direct frontier model only as a separate baseline.

The result was encouraging, expensive, and more nuanced than “cheap model equals frontier model.”

What an RLM changes

A normal model call looks roughly like this:

large prompt -> model -> answer

An RLM instead gives the root model metadata about the input and a Python REPL containing the real context:

question
   |
root model
   |
persistent REPL holding the context
   |-- inspect and search with code
   |-- split context into useful chunks
   |-- call smaller LMs over those chunks
   |-- validate and aggregate results
   `-- return the final answer

The important detail is that the root model does not need to carry every document, record, tool result, and partial answer in its own context window. Large intermediate values can remain in REPL variables. Subcalls receive focused, locally understandable tasks.

That makes RLM less like a bigger prompt and more like an out-of-core data-processing system whose semantic operator happens to be a language model.

What we actually tested

We used an OOLONG trec_coarse validation example from the protocol described in the RLM work.

The input was a 308,367-character context containing 3,182 general-knowledge questions. Each question implicitly belonged to one of six answer types:

numeric value
entity
human being
location
abbreviation
description and abstract concept

The labels were not present in the context. The task was to infer the labels and identify the least-common category.

We compared:

A direct gpt-5.6-sol Codex call.
An RLM where the root and all leaf calls were locked to gpt-5.4-mini.

The direct frontier call answered abbreviation and scored zero. The mini-only RLM answered numeric value, matching the gold answer.

Method	Result	Model calls	Elapsed time
Direct frontier call	Incorrect	1	40.1 seconds
RLM with `gpt-5.4-mini` only	Correct	At least 238	6,120.3 seconds

The RLM root first inspected the structure of the context. It then classified chunks, retried malformed responses, reduced the chunk size, reclassified all 3,182 questions using structured JSON outputs, checked that it had coverage, and calculated the minimum.

This is exactly the sort of work that a direct model call often approximates but a recursive program can force itself to perform.

The uncomfortable but useful audit

Getting the final answer right did not mean every intermediate judgment was right.

We compared the mini model's inferred counts against the validated labels:

Label	True count	Mini inferred
Numeric value	398	402
Entity	521	623
Human being	544	488
Location	571	493
Abbreviation	571	560
Description and abstract concept	577	616

The model made substantial row-level classification errors. It still found the correct minimum because numeric value had a 123-item margin over the next-smallest true category.

That distinction matters. This run shows that decomposition changed the outcome and allowed a cheap model to solve one problem that the direct frontier call missed. It does not prove that the cheap model reconstructed the data exactly, and one row does not establish general equality between the two systems.

The honest claim is narrower:

On suitable long-context tasks, a cheap model inside an RLM can match or outperform a direct frontier-model call.

Use cases already demonstrated by RLM research

The paper evaluates four useful task shapes:

1. Semantic aggregation

OOLONG requires labeling and aggregating information spread throughout a large input. Real applications include:

customer-feedback analysis;
support-ticket taxonomies;
survey aggregation;
incident and application-log analysis;
quality-control statistics over text records.

Our experiment belongs to this category.

2. Multi-document research

BrowseComp-Plus requires joining evidence across documents in a very large offline corpus. Analogous applications include:

literature reviews;
technical-documentation research;
contract and policy comparison;
due-diligence document rooms;
evidence-backed competitive research.

3. Repository-scale understanding

The paper includes LongBench-v2 CodeQA, where questions require reasoning across files in a codebase. Probable uses include:

architecture mapping;
migration-impact analysis;
dependency and license audits;
security triage;
locating missing tests;
comparing implementation against documentation.

4. Cross-record and pairwise reasoning

OOLONG-Pairs asks the system to construct relationships between combinations of records. Applications could include:

entity resolution;
policy-conflict detection;
matching candidates against constraints;
finding related incidents;
identifying incompatible configurations;
relationship discovery across an archive.

These workloads can grow quadratically, so they need strict budgets and deterministic post-processing.

A practical new use case: making sense of agent-session archives

While exploring our local Claude Code history, we found a single session transcript that was 242 MB and contained 39,570 JSONL records. All project transcripts together occupied about 3.6 GB.

The large session was not 242 MB of useful conversation:

about 176 MB was attachment records;
about 29 MB was assistant events;
about 20 MB was user and tool-result events;
about 12 MB was file-history snapshots.

This is an excellent RLM-shaped problem.

A deterministic first pass can stream the JSONL, hash duplicate attachments, reconstruct parent-child event relationships, merge subagent logs, and extract messages, commands, file changes, tests, commits, errors, and outcomes. An RLM can then analyze normalized episodes and recursively build:

a cross-session project timeline;
a decision register;
a map of attempted and abandoned approaches;
recurring failure patterns;
unresolved tasks;
evidence-linked summaries of what actually shipped.

The final report should cite session IDs, event IDs, timestamps, commands, and Git commits. Otherwise, it is merely another plausible summary.

Other probable use cases

The same decomposition pattern should transfer to:

long incident timelines assembled from logs, tickets, and chat;
scientific evidence extraction across papers and experiment records;
compliance control-to-evidence mapping;
large archives of meetings, email, or project documents;
ranking records against a nuanced rubric;
graph filtering and multi-hop relationship discovery;
reconciling conflicting claims across many sources;
constructing structured datasets from heterogeneous text.

The recurring requirement is not simply “the input is long.” A good RLM task has four properties:

The context can be partitioned or searched programmatically.
Smaller semantic subtasks remain understandable to the cheap model.
Intermediate results can be stored in a structured form.
The final result can be verified or recomputed.

Where RLM is probably the wrong tool

RLM is a poor default for:

low-latency chat;
simple questions that fit comfortably in one prompt;
sparse retrieval where grep or conventional search is sufficient;
creative writing that depends on a single coherent voice;
exact high-stakes decisions without an independent verifier;
public execution of untrusted model-generated Python;
high-volume synchronous APIs with tight latency budgets.

Our successful row took roughly 102 minutes. That is acceptable for a research run or an overnight audit, not for an interactive endpoint.

What a reusable package should look like

The useful abstraction is not an OOLONG runner and not one universal prompt. It is a context-compute runtime with a small set of reusable recipes:

run(
  context,
  objective,
  recipe,
  answer_schema,
  verifier,
  budget
) -> answer + evidence + validation + trajectory + usage

Initial recipes could include:

aggregate_records
evidence_synthesis
repository_analysis
cross_record_join
timeline
candidate_ranking

For our intended configuration, the Codex backend would keep both root and subcalls locked to gpt-5.4-mini. A frontier model would appear only in evaluation runs, never inside the RLM call tree.

Production use would also require an isolated execution environment, call and token limits, schema validation, redaction, prompt-injection defenses, resumable runs, and source-level evidence for every important claim.

What comes next

The one-row result is a proof of mechanism, not a benchmark victory.

The immediate research questions are:

Does the advantage survive across all 50 paired OOLONG tasks?
Can concurrency reduce the 102-minute runtime without changing quality?
Which decomposition recipes transfer cleanly between domains?
How much verification is required for exact row-level work?
Can a mini-only RLM turn multi-gigabyte agent histories into a reliable, source-linked development narrative?

RLMs do not magically turn a cheap model into a frontier model. They change the computation available to that model. Sometimes that difference is enough to turn a wrong one-shot answer into a correct, auditable process.

That is a more interesting result than the slogan.

Reproduction data: the paired OOLONG runs behind these numbers are published at huggingface.co/datasets/Rickesh/rlm-oolong-reproduction.

We made our world model smaller and it got better. Then "efficient" attention made nothing faster.

Rickesh T N — Fri, 07 Aug 2026 05:56:04 +0000

Two experiments on the same world model, both of which came out the opposite way to how I expected. One is now an IJCNN 2026 paper, the other an ICPR 2026 paper. Both are really about the same thing: the scaling intuitions most of us carry around are older than the hardware we run on.

What a world model has to do here

A dynamic scene reconstruction model takes sparse multi-view camera observations across time and predicts a full 3D scene, including how things are moving. Point a few cameras at a street for two seconds, get back a 3D representation of that street with the cars in it moving correctly.

This is the perception half of a world model for driving and robotics. If you want to simulate "what happens if I turn left here," you first need a model that can build the scene at all.

The architecture we worked from is STORM (Yang et al.), a transformer that predicts 3D Gaussian primitives and their motion in a single forward pass. Feed-forward, so no per-scene optimization: a big deal if you ever want this on a vehicle rather than in an offline pipeline. That's prior work, not ours. What follows is what we found building on top of it.

Setup throughout: a subset of the Waymo Open Dataset, 2-second clips at 10 fps, up to three synchronized camera views, images downscaled to 160x240, 150K-300K training iterations.

Experiment 1: we allocated the parameters differently and the smaller model won

The default move when a transformer underperforms is to make it deeper. More layers, more capacity, better results. That instinct comes from language modelling and image classification, and it is mostly right there.

Dense spatio-temporal prediction is not those tasks.

We built a width-dominant, shallow variant. Concretely: fewer transformer layers, larger embedding dimension, and fewer attention heads each with a bigger head dimension. Then we ran it in a smaller configuration against the standard baseline in a larger one, under an identical training and data pipeline.

Model	PSNR	SSIM	Depth RMSE
STORM baseline (B8)	26.82	0.770	12.59
Ours, width-dominant (V6, S8 config)	27.57	0.806	15.59

+0.75 PSNR and +0.036 SSIM, from the smaller model.

The training dynamics were the more interesting part. The width-dominant model converged faster, to a higher plateau, along a visibly smoother trajectory, and its gradient norms were substantially more stable. It was not squeezing out a marginal win at the end of a hard optimization. It was simply an easier model to optimize.

Why this is plausible: self-attention already provides global mixing within every single layer. Stacking more layers buys you more sequential refinement, but sequential depth is exactly what makes optimization harder. Widening instead gives each attention head a richer representation to work with while leaving global information exchange intact. For a task where every token needs to talk to every other token anyway, spending your parameter budget on width rather than depth is not obviously the wrong call, and here it was the right one.

Now the part I want to be honest about: look at the depth RMSE column. The baseline is clearly better on geometry, 12.59 against our 15.59. We got prettier images and worse depth.

That trade-off is real and we reported it as a finding rather than burying it. Photometric quality and geometric accuracy are not the same axis, and if you are building a world model whose output feeds a planner rather than a display, the column we lost on may be the column you actually care about. Anyone reading "+0.75 PSNR" as an unqualified win has read the paper badly.

The practical upshot is deployability. A smaller model that trains on consumer GPUs and beats a larger one on photometric quality changes who can work on this at all.

Experiment 2: attention reuse, and a result I did not want

Self-attention costs O(N² · d_h · H). For spatio-temporal input the token count N is brutal, because you are multiplying pixels by views by timesteps. So the obvious efficiency target is attention itself.

There is a neat line of work on attention reuse, built on the observation that attention maps stabilize as you go deeper into a transformer. If layer 9's attention pattern looks a lot like layer 8's, why recompute the query-key interaction at all? Reuse and transform the previous map instead. You keep the global token interaction and drop a chunk of the compute.

We integrated Less-Attention layers into the STORM-B/8 backbone and evaluated across a range of reuse ratios, measuring runtime, memory, parameter growth and reconstruction quality, under both naive and optimized implementations.

Attention reuse did not improve wall-clock time over optimized full attention. At any reuse ratio. Even with careful implementation.

The reason is not that the theory is wrong. The FLOPs really do go away. The reason is that fused attention kernels on modern GPUs have already moved the bottleneck somewhere else. When full attention runs as a single fused kernel that never materializes the attention matrix in HBM, the thing you were paying for was mostly memory traffic, not arithmetic — and "skip some of the arithmetic" optimizes the wrong resource. You also add parameters and complexity for the reuse machinery, and you now have two kernels where the fused path had one.

This is the whole lesson: an algorithmic optimization derived from a FLOP count is a hypothesis about the hardware, not a fact about it. The hardware moved. FlashAttention-style kernels changed which operations are expensive, and a good chunk of the "efficient attention" literature was implicitly costed against a machine that no longer exists.

I would rather this had worked. Publishing a null result is less fun than publishing a speedup. But a null result that saves other people from re-implementing the same thing is worth more than another 3% on a benchmark.

What I take from both

Capacity allocation matters more than total capacity. Where you put the parameters is a real design decision with real consequences, and "make it bigger" is the answer you give when you have not measured which axis is actually binding.

Measure wall-clock on your target hardware. Always. Not FLOPs, not parameter counts, not asymptotic complexity. Those are proxies, and proxies drift as hardware evolves. Every efficiency claim has a silent "on the machine I tested" attached to it.

Report the column you lost on. Our depth RMSE got worse. Someone building on this needs to know that far more than they need another decimal place of PSNR.

The intuitions are load-bearing and mostly untested. "Deeper is better" and "fewer FLOPs is faster" are both reasonable priors that happened to be wrong in our setting. I only found that out by running the ablation instead of assuming.

If you work on neural rendering, world models, or transformer efficiency, I would genuinely like to hear whether the width-over-depth result holds in your setting. My prior is that it generalizes to dense prediction tasks and not much further, but that is a prior, and this post is largely about how those go.

Papers: "Rethinking Transformer Design for Dynamic Scene Reconstruction: An Efficient, Width-Dominant Approach" (IJCNN 2026) and "Evaluating Attention Reuse in Dynamic 3D Gaussian Reconstruction" (ICPR 2026).

I write about world models, robotics data pipelines, and the ways measurement quietly fails. linkedin.com/in/rickeshnatarajan