DEV Community: Rickesh T N

My inference server decided my second GPU no longer exists. Here is how I got it back without upgrading a driver.

Rickesh T N — Mon, 24 Aug 2026 23:29:36 +0000

My desktop runs two mismatched GPUs: a 20GB Ampere card and a 16GB Pascal Tesla. For months, a 21GB vision-language model ran split across both at full GPU speed. Then a routine upgrade of the inference server dropped Pascal support behind a driver-version gate, demoted the old card to a Vulkan device the scheduler refuses to mix with CUDA, and silently rescheduled my model to one GPU plus CPU spill.

Nothing errored. The model still answered. It just answered at 20 to 40 seconds per item instead of 8, which turned a 12-hour benchmark into a 60-hour one. The only evidence was a log line saying the driver was "too old" for a card that had been running CUDA workloads that same morning.

The gate is policy, not physics

The interesting discovery came from reading the server's own startup logs: the bundled CUDA kernels still listed the Pascal compute capability in their build targets. The kernels existed. The Go-side scheduler was refusing to use them based on a driver-version check, while the C++ inference runtime underneath had no such opinion.

That asymmetry is the exploit. The upgraded server ships its actual inference engine as a standalone binary, a vendored build of llama-server, with its CUDA backend as a dynamically loaded library sitting in a subdirectory. The driver gate lives entirely in the scheduler process that launches it.

So: launch the vendored binary directly.

Two mechanics matter. First, the dynamic backend is not found automatically because it lives in a subdirectory the loader does not scan; it needs an explicit environment variable pointing at the .so file itself, not the directory. Get that wrong and the binary silently falls back to CPU while still accepting requests, which cost me one confusing benchmark restart. Second, the vendored build understands the server's own single-file model format, weights and vision projector packed together, which upstream llama.cpp of the same vintage refuses to load. The vendored binary logged "detected combined format, translating" and just worked.

Tensor splits are a policy decision too

With both GPUs visible again, the default split put layers proportional to total VRAM. That is wrong for mismatched cards: the Ampere card is roughly four times faster per layer, so every layer on the Pascal card costs four on the fast one. Skewing the split heavily toward the fast card, leaving the slow card holding only what does not fit, took per-item latency from 17 seconds to 9. Combined with a bigger prefill batch, the full benchmark ran in 12 hours instead of a projected 60-plus.

The general shape of the fix:

The scheduler said no; the runtime said yes. When a managed serving layer refuses your hardware, check whether the engine underneath actually shares the objection.
Point the backend loader at the exact library file and verify placement with nvidia-smi before trusting any run. A server that fell back to CPU serves identical responses, slower.
On mismatched GPUs, split by throughput, not by capacity. Free VRAM on a slow card is not free.

The clean fix is a driver upgrade, and it is scheduled. But the bypass took forty minutes including the two failed attempts, needed no root, touched nothing system-wide, and taught me more about the serving stack than a year of it working silently ever did. The scheduler is there to protect the average user from edge cases. If you can read its logs, you are allowed to disagree with it.

A generic fine-tuning playbook, written after doing it wrong several times

Rickesh T N — Mon, 24 Aug 2026 23:29:00 +0000

Every fine-tuning guide I read before my first serious attempt was a tutorial about knobs: learning rates, LoRA ranks, quantization settings. None of them covered the part that actually decides whether the project succeeds, which happens before and after the training run, not during it. This is the playbook I now follow for any model on any task. It is deliberately generic: the same sequence has carried me through vision-language models on driving data, small text models for domain QA, and RL-style preference tuning, on hardware ranging from a single consumer GPU to rented cloud boxes.

Step 0: try not to fine-tune

Fine-tuning is the most expensive intervention in the stack, so it goes last. The ladder, cheapest first:

Better prompting. A system prompt with three good few-shot examples routinely closes half the gap that people reach for fine-tuning to close. It costs an afternoon.
Retrieval. If the failure is missing knowledge rather than missing behavior, RAG beats weights. Knowledge changes; your fine-tune will not.
A bigger or different base model. Run the comparison honestly: a zero-shot larger model against your imagined fine-tuned smaller one. I have watched an untouched open-weights model beat an in-domain fine-tune on the fine-tune's own benchmark. It happens more often than leaderboards suggest.
Fine-tune. Only when the behavior you need is demonstrably not in the base model and cannot be retrieved or prompted in: output format compliance, domain-specific reasoning patterns, a persona that must survive thousands of turns, latency budgets that force a small model.

Write down, in one sentence, which failure of steps 1 to 3 justifies the fine-tune. If you cannot write that sentence, stop. That sentence also becomes your evaluation target later, which is the real reason to write it.

Step 1: build the evaluation before the dataset

This ordering feels backwards and is the single highest-leverage decision in the playbook.

Before collecting training data, build a held-out evaluation that measures the sentence from step 0, and run the base model through it. That number is your baseline, and it does three jobs. It tells you the true size of the gap. It occasionally kills the project on the spot because the base model was already good enough and nobody had measured it. And it validates the harness itself: an evaluation that has never scored a known model is untested code that emits numbers.

Two rules for the eval that I no longer break:

The test split is designed before training, and nothing from training may touch it. Not for hyperparameter selection, not for checkpoint picking, not once. Use a validation split for those. If your splits share source documents, scenes, or sessions with the training set, you are measuring memorization and calling it generalization.
Run a sanity control. Delete the input and measure again. If a vision model scores far above chance with the images removed, your benchmark leaks answers through priors and question phrasing, and every score it has produced is inflated. The same control exists for text: shuffle the context, drop the retrieved passages, feed the question alone. Cheap to run, devastating when it fires, and better fired at you than at a reviewer.

Step 2: data is the model

The dataset decides what you get. The recipe decides only how efficiently you get it.

A few thousand excellent examples beat a hundred thousand scraped ones for behavior tuning. For format compliance and persona, hundreds can be enough with LoRA.
Deduplicate against your evaluation. Near-duplicates leak. Exact-match dedup is not sufficient; hash at the level the data actually repeats (documents, scenes, sessions).
Audit a random hundred by hand. Not the first hundred, a random hundred. The first hundred were curated by whoever built the file; the random hundred tell you the truth about label noise. Every bad label teaches the model confidently.
Match the training distribution to the inference distribution. If production inputs will be messy, OCR-damaged, or truncated, train on that, not on the clean version.

Expect data work to consume more than half the project's wall-clock. When it does not, that is usually a sign it was skipped, and the bill arrives later, denominated in GPU-hours.

Step 3: choose the lightest recipe that can express the change

LoRA or QLoRA is the default. Behavior shaping, format compliance, domain adaptation, personas: adapters handle all of it at a fraction of the memory, and the artifact is small enough to version, ship, stack, and roll back.
Full fine-tuning is for when the change is deep: new modalities, new tokenizers, or when adapters measurably plateau below target. Prove the plateau before paying for the parameters.
Preference or RL methods (DPO, GRPO and relatives) are for objectives that supervised examples cannot express: relative quality, verifiable rewards, multi-step outcomes. They are also where silent failure lives, so they come with an extra rule: track a metric that measures the actual objective, not a proxy. I once watched token accuracy sit at 99 percent while the policy learned the opposite of the intended behavior. The proxy was fine; the behavior was not.

One decision that outranks the recipe choice: change one variable at a time. A run that changes base model, dataset, rank, and learning rate simultaneously produces a result that cannot be attributed to anything. Reviewers catch this in papers; production catches it in incidents. If you must move fast, move fast serially.

Step 4: engineer the run like it will be interrupted, because it will be

Training runs die. Disks unmount, drivers hiccup, a colleague's job lands on your GPU, spot instances vanish. The runs that survive share the same boring infrastructure:

Checkpoint on a fixed cadence and verify resume actually works before the long run, not during the outage. A checkpoint you have never resumed from is a hope, not a checkpoint.
Log to an experiment tracker, not a terminal. Scalars, config, git commit, environment. The question you will ask in three weeks is "what exactly produced this file", and scrollback does not answer it.
Emit progress with the failure count first. A silent job that is working and a silent job that is failing on every batch look identical from outside. If the error count is climbing, you want to see it in the first minute, not after the run completes.
Pin the environment. Container images beat requirements files; requirements files beat memory. Half of my hardest debugging sessions were two libraries disagreeing about something as small as position ids, and the fix was environmental, not algorithmic.

Step 5: evaluate like an adversary, then decide

When training finishes, resist the demo. Run the same held-out evaluation from step 1, and read three numbers together, never one:

Target metric versus base model. Did the gap from step 0 actually close, with the improvement larger than your seed-to-seed noise? If you have not measured seed noise, run the fine-tune twice before believing any margin smaller than a point or two. Single-seed margins evaporate embarrassingly often.
Regression suite. What did the model lose? General capability regressions are the default outcome of narrow fine-tuning, not the exception. A small fixed battery of out-of-domain checks is enough to catch the worst of it.
Train-versus-held-out gap. Large gap: you memorized; get more data or regularize. No gap and no improvement: capacity or recipe; move up the ladder from step 3.

Then make the deployment decision with the same honesty as step 0: is the fine-tuned model better than the best non-fine-tuned alternative, at the quality, latency, and cost that production actually needs? Sometimes the answer is no, and the fine-tune becomes a lesson rather than a deployment. That outcome is not a failure of the playbook. Discovering it for the price of one training run instead of one production incident is the playbook working.

The one-page version

Prompt, retrieve, or upgrade the base model first; fine-tune last, and write the sentence that justifies it.
Build the eval and score the base model before touching training data. Design the splits before training. Run the deleted-input control.
Spend most of the project on data: dedup against eval, audit random samples, match production distribution.
Default to adapters; escalate only on proven plateaus; one variable per run.
Checkpoint, track, and emit errors-first progress; pin the environment.
Judge with three numbers: target delta versus noise, regressions, generalization gap. Then decide like you have not already sunk the cost, because the model does not care that you did.

A 4B model on a 6GB laptop beat Claude Opus on our 440K-token corpus. The fix was giving the model less to do.

Rickesh T N — Mon, 24 Aug 2026 23:28:23 +0000

Private AI has a hardware story nobody measures honestly. The pitch is that your data never leaves the building. The unstated cost is that the building contains a 6GB laptop GPU, and the corpus is 440,000 tokens that a frontier model would swallow in one call.

We measured the whole thing. One RTX 3060 Laptop, 6144 MiB. One 2,625-record driving-QA dataset rendered to 957,493 characters. One question with a single correct answer: which category is least common. Ground truth computed in Python, so neither model gets to define success.

The 4B won. It took four failed attempts to understand why.

The context problem was never the hard part

Direct context on that card, measured with flash attention and a q8_0 KV cache:

num_ctx	resident	fits 5.5GB usable
8,192	3.2 GB	yes
16,384	3.3 GB	yes
32,768	3.3 GB	yes, the ceiling
65,536	10.4 GB	no, 31 percent on GPU
131,072	10.9 GB	no

KV is nearly free to 32k, then the allocator falls off a cliff. Switching the KV cache to q4_0 changed those numbers not at all, so the cliff is not the KV cache and you cannot quantize your way past it. Direct ceiling: 32,768 tokens.

Now stream a 261,226-token corpus through the same card in chunks, keeping the text in CPU RAM and only ever showing the model a fragment. Peak GPU across four runs: 4.23, 4.24, 4.25, 4.54 GB. Every one fits.

That is an 8x context multiple at constant VRAM. Constant is the load-bearing word: the corpus never enters the KV cache, so the limit stops being memory and becomes wall-clock.

Which is a solved problem. The unsolved one was that the answers were wrong.

Four ways to fail the same question

attempt	sub-calls	answer
1	7	prose describing the data, having read about a third
2	11	`Spatial` — right arithmetic, truncated label
3	65	`Counterfactual` — swept everything, well-formed, wrong
4	72	`Status: beta, Status: delta, Status: gamma, Status: alpha`

Attempt 4 is the one that gives it away. Asked which label was least common, the model listed all four candidates instead of selecting one. Attempt 3 had read all 425,054 input tokens and still combined the partial counts wrongly.

We had been calling this a capability gap. It was not. A 4B can count rows in a fragment. What it cannot reliably do is plan a traversal and then perform arithmetic across 65 partial results — and we had been asking it to do both, inside a loop, while also formatting an answer.

The fix: treat the corpus like a video stream

Nothing about aggregation requires a language model. So we stopped asking one.

video streaming	the port
manifest / playlist	segment plan, computed in code before any model call
buffer	N segments in flight, latency hidden behind compute
decoder	model sees ONE segment, emits `key<TAB>number`, never prose
playback	reduce phase — aggregation in code, strategy chosen explicitly

The model's entire job becomes extraction from a window it comfortably fits. Planning is deterministic. Summation is a loop. Selecting the minimum is one comparison.

Result on the same corpus, same 4B, same 6GB card:

17 segments · failed=0 · records=611 · unparsed_lines=3 · keys=15 · 641s
Category: Spatial Relationship

Correct. 15 keys recovered, matching the 15 real categories. Of 611+ emitted lines, 3 failed the output contract and were counted as failures rather than silently dropped.

The frontier comparison, which is not flattering to the frontier

We used Claude Opus over the full 439,742-token context as the reference. It answered the same question two different ways on byte-identical input — correct once, wrong once — and landed 2 out of 3 across three samples, at $4.79 per call.

	answer	correct	cost
Opus 4.8, one call, 439,742 tok	varies by run	2/3	$4.79
4B + streaming, 6GB laptop	`Spatial Relationship`	yes	$0.00

A reference that disagrees with itself is not an oracle. Ground truth has to come from code, with the frontier model scored as just another candidate.

For private AI this cuts two ways. The bar is lower than the marketing implies, because context rot is real and frontier models are not deterministic at 400K tokens. But "matches the frontier model" is also the wrong success criterion. Match the ground truth, and measure both.

What we would tell anyone building this

Recursion is not free. On a benchmark where the context already fit the model's window, wrapping the same model in a recursive harness scored 0.269 against 0.428 for a plain direct read. Recursion only pays when the context genuinely does not fit. Reach for it as a last resort, not a default. That comparison is from our OOLONG reproduction, and the paired run data is public: huggingface.co/datasets/Rickesh/rlm-oolong-reproduction.

Silent truncation is the dangerous failure. Our serving layer auto-sized context to the prompt up to a ceiling, then quietly fell back: 30,021 tokens passed intact, 50,000 and 70,000 both clipped to exactly 16,387, with no error and a confident answer from the fragment. Compare processed-token counts against what you sent, and make the mismatch fatal.

Substring scoring manufactures success. One run reported correct because the model dumped raw corpus rows and the dump happened to contain the gold label. Require the declared answer form and reject anything that looks like regurgitated input.

Report what you could not parse. unparsed_lines=3 is the number that makes the rest of the output trustworthy. A harness that silently drops what it cannot read will happily report a clean answer over half the data.

A partial sweep must refuse to answer. If any segment fails, every key is undercounted. Ours exits non-zero rather than printing a number that looks fine.

The engine is C++17 with zero third-party dependencies, and the planning and reduction stages are covered by 61 tests that need no GPU, no network and no tokens — because once the model is only doing extraction, everything else is ordinary code you can actually test.

99% token accuracy, zero learning. Field notes from fine-tuning vision models with RL.

Rickesh T N — Sun, 23 Aug 2026 23:33:17 +0000

Over the past year I have been fine-tuning open vision-language models - 9B dense up to a 35B mixture-of-experts - with supervised fine-tuning and GRPO-style reinforcement learning on verifiable rewards. Most of what I learned was not about algorithms. It was about the ways a training run can look healthy while doing nothing, or crash for reasons that have nothing to do with your code.

Three failures, in increasing order of how long they fooled me.

Failure 1: the metric that measured the wrong thing (18 hours)

I ran an 18-hour supervised fine-tune that reported token accuracy climbing steadily to 99%. Looked like a textbook run. The real evaluation metric - accuracy on multiple-choice questions - never moved.

The cause was a mismatch between what I supervised and what I evaluated. The training loss was over free-text reasoning traces; the evaluation scored a single extracted answer letter. The model got extremely good at reproducing the shape of the training text - hence 99% token accuracy - without that transferring to the decision I actually cared about.

Token accuracy is a proxy, and proxies drift from the target exactly when you stop checking. The fix was structural, not a hyperparameter: supervise the thing you evaluate. If the deliverable is a constrained answer, the training signal has to reach that answer, not just the prose around it.

The general rule I took: any training metric that is not your evaluation metric is a hypothesis about correlation, and you should check that correlation before you spend GPU-days on it.

Failure 2: the crash that was two libraries disagreeing about position ids

The GRPO trainer for the 9B vision model crashed in the forward pass, deep inside rotary position embedding code. Nothing in my training code had changed.

The diagnosis took a while because the bug lived at the boundary between components: the text sequence length was derived from token-type ids, while the vision sequence length came from the image grid - and image-pad tokens ended up counted twice. Two parts of the same stack, each internally consistent, disagreeing about how long the input was.

For the 35B MoE variant of the same family, an equivalent rope bug was fixable by monkeypatching the model's position-id computation. I shipped the patch with a GPU-free regression test: a tiny script that constructs the exact failing input shape and runs just the position-id path on CPU. It runs in seconds, needs no cluster, and fails loudly if an upstream update reintroduces the bug.

Two lessons. First, when you fine-tune at the edge of a model family's tooling support, the bugs you hit are integration bugs, and the stack trace points at the victim, not the culprit. Second, every monkeypatch deserves a regression test that costs nothing to run - otherwise the next library upgrade silently un-fixes it.

Failure 3: the RL loop that was learning the opposite (the quiet one)

In a separate project I fine-tune a 9B model with reinforcement learning where the reward comes from realized real-world outcomes rather than a labelled dataset. For a long stretch the training signal was flat - not diverging, not collapsing, just flat, which is the least informative failure there is.

Two compounding problems. One was label noise in the reward pipeline: some outcomes were being attributed to the wrong decisions, which dilutes any gradient. The other was worse: a sign error meant part of the advantage signal was inverted. The model was being gently pushed away from behaviour that had worked.

Nothing crashed. Every batch processed. Every log line looked like a training run. The only symptom was the absence of learning, and the only way I found it was working backwards from "the held-out metric should have moved by now" to auditing every stage of the reward computation by hand.

After both fixes I got the first genuinely monotonic learning curve on that task. I still treat it as training signal only - the decider is held-out evaluation against the base model, and I do not report improvements that exist only in the training curve.

The harness rules I now run everything under

These came out of the failures above plus a benchmarking programme across 70+ vision-language models. They are boring, and they are the difference between numbers and noise.

Smoke test before committing compute. A five-step GRPO run with two numbers watched: the PPO-style clip ratio and the fraction of outputs that parse. If the clip ratio is degenerate or parseability is low, the full run will be garbage in a way five steps already reveals.

Runs are gated fail-closed. Nothing publishes a result unless the evaluation stage actually scored. "The eval crashed but training finished" is not a result; it is an unscored run, and unscored runs must be impossible to mistake for scored ones.

Infrastructure failures and poor performance are different columns. An unparseable output, an OOM, a crashed kernel - these are exceptions whose count must be exactly zero. A weak model produces zero exceptions and simply scores badly. If a quality threshold can absorb an infrastructure failure, a totally broken run can pass your gate.

The held-out metric is the only decider. Training curves, token accuracy, reward trends - all of it is telemetry. If the held-out number did not move, nothing happened.

None of this is novel. All of it is the difference between the runs I trust and the 18 hours I lost.

I write about ML evaluation, world models, and the ways measurement quietly fails. More at dev.to/rickeshtn.

Your multi-agent system isn't hitting prompt cache. Your system prompt is the reason.

Rickesh T N — Tue, 11 Aug 2026 21:49:39 +0000

I run a multi-agent setup where ten agents analyse the same input. Same document, same market data, same everything. The only difference between them is persona: each one is instructed to look at the material through a different lens.

Ten agents, one shared context. That should be the ideal case for prompt caching. Send the expensive context once, pay full price for it once, and let the other nine reads come back at a fraction of the cost.

My cache hit rate was zero percent on three of the five models I was using, and under seven percent on the other two.

I had been reading the bill for a while and optimising the wrong thing. Here is what was actually happening, because the mistake is structural and I doubt I am the only one making it.

What prompt caching actually matches on

Hosted inference providers cache on a prefix. The provider hashes your request from the first token forward and looks for the longest run it has already computed. If your request starts with the same 3,000 tokens as a recent one, those 3,000 tokens are a cache read, typically around five times cheaper than a fresh read. The moment the token stream diverges, caching stops for the rest of the request. There is no re-syncing later.

That word — prefix — is doing all the work, and I had not thought about it carefully.

The setup that broke it

My call looked like every example in every SDK doc:

messages = [
    {"role": "system", "content": agent_persona},   # differs per agent
    {"role": "user",   "content": shared_context},  # identical for all 10
]

The persona is short. A couple of hundred tokens describing how this particular agent should reason. The shared context is large: several thousand tokens of source material.

Read that message array as a flat token stream, which is what the provider does. The first thing in the stream is the persona. The persona is different for every agent. So the prefix diverges at roughly token one, and the several thousand tokens of identical context sitting behind it can never match anything.

Ten agents. Ten identical copies of the same context. Ten full-price reads.

Proving it rather than assuming it

I did not want to guess, so I hashed both halves of every call for a single work item and counted the distinct values.

SELECT
  COUNT(DISTINCT prompt_sha256) AS distinct_user,
  COUNT(DISTINCT system_sha256) AS distinct_system
FROM agent_calls
WHERE item_id = ?

The answer:

distinct_user   = 1
distinct_system = 10

One user prompt. Ten system prompts. The expensive half was byte-identical across all ten calls, and the cheap half in front of it was unique every time.

This lines up exactly with the provider's own usage report, which broke my spend into cached and uncached input tokens:

model	cached share of input
A	0.0%
B	0.0%
C	3.4%
D	6.2%
E	11.0%

Those low non-zero numbers are incidental collisions between unrelated calls, not the structural reuse I should have been getting. If the design were right, nine out of every ten context reads would be cache hits.

The fix

Put the shared, expensive, identical part first. Put the small, varying part last.

messages = [
    {"role": "system", "content": shared_context},          # identical -> caches
    {"role": "user",   "content": f"{agent_persona}\n\n{question}"},  # varies, small, last
]

Now the first several thousand tokens are the same for all ten agents. The first agent pays full price and warms the cache. The other nine read it back at cache rates. The only uncached part is the couple of hundred persona tokens at the tail, which is what you actually want to be paying for.

The general rule, which I now think should be a design constraint rather than an optimisation:

Order your prompt from most shared to most specific. Caching rewards a stable prefix, and every byte that varies early poisons everything after it.

This also composes with how you batch. If you run the same model across many items back to back, you keep hitting a warm prefix. If you round-robin across models for each item, you cold-start the cache on every single call. Grouping by model, not by work item, keeps the cache warm.

The part I am not comfortable with

Moving the persona out of system and into user is not free.

Some models weight system instructions more strongly than user content. That is often the point of a system prompt. If one of my agents is specifically instructed to argue an unpopular position, and I demote that instruction from system to user, it may hedge more. I would be trading spend for behaviour, and I would not necessarily notice, because the output would still be well-formed and plausible.

So this is not a change I would ship straight to production off the back of a cost argument. It needs an A/B on a sample of items, comparing the actual decisions each layout produces, not just checking that the responses parse.

There is a middle path worth trying first: keep a short stable instruction in system that is identical across all agents, and move only the per-agent differentiation into the user message. You get a shared prefix and keep a system-role framing. Whether that is enough depends on how much of your agents' behaviour hangs off the system role, which is an empirical question about your prompts and your models.

What I would take from this

Prefix means prefix. Anything that varies early destroys caching for everything after it, no matter how much identical material follows.
Instrument it. Hash the components of your requests and count distinct values per work item. It took one query to turn a vague suspicion into a definite structural bug.
Read the cached-versus-uncached split in your usage report. A near-zero cache rate on a workload with obvious shared context is not a pricing quirk. It is a design bug, and it is telling you the prefix is broken.
The default SDK message shape is not cache-aware. Persona-in-system, content-in-user is the shape in every tutorial. It is exactly wrong for fan-out workloads where many personas share one context.

I had spent real effort choosing cheaper models before I checked whether I was paying for the same tokens ten times over. The model swap was worth doing. It was also the second-biggest lever, and I found it first because it was the one I was looking for.

Addendum: what the comments corrected

This post got better feedback than it deserved, and three points are important enough to belong in the body rather than below it.

There is a third option I missed, and it is the right one where available. @skillselion pointed out that on APIs with explicit cache breakpoints, the system prompt can be an array of content blocks: shared context as the first block carrying a cache_control marker, persona as a second block after it. The cached prefix ends at the marker, so every agent hits the same cache while the persona keeps its system-role framing. That dissolves the trade-off I spent a section being uncomfortable about. The role-versus-cost tension is an artifact of implicit prefix caching, not something inherent to the problem. Where the provider gives you breakpoints, use them; my middle path is for providers that only do implicit prefix matching.

"Put the shared context first" is necessary but not sufficient. @max_quimby and @kartik-nvjk both made the same point: the shared prefix has to be byte-identical across calls. A per-run request id, a timestamp, or a reordered JSON key injected anywhere before the boundary resets the prefix, and you are back to zero with a layout that looks correct. Enforce a canonical serialization for the cached block.

The prefix starts earlier than your message array. @mads_hansen noted that "identical text" is not the same as an identical provider prefix — tool schemas, response formats, model ids and SDK serialization defaults can all sit ahead of or inside your visible messages. Worth confirming against your provider's documented render order; on Anthropic's API, for instance, it is tools, then system, then messages, which means an unstable tool list invalidates the cache before your system prompt is even reached. Hash the canonical request envelope, not just the parts you wrote by hand.

Mads also proposed a three-layer layout I have adopted: stable policy shared by every agent, stable source context shared by the work item, then the small role and question suffix — each versioned separately, so a persona change invalidates only the cheap tail while a policy change invalidates the shared prefix deliberately. That turns cache behaviour into an observable property of the prompt architecture instead of a billing surprise.

Can a Cheap Model Beat a Frontier Model? Rebuilding Recursive Language Models with Codex

Rickesh T N — Sun, 09 Aug 2026 15:05:19 +0000

Large language models have enormous context windows now. That does not mean they use all of that context reliably.

As prompts grow, models can miss details, lose track of relationships, or produce plausible summaries instead of doing the exhaustive work a question requires. The Recursive Language Models (RLM) paper proposes a different interface: keep the large context outside the model, expose it as a variable in a persistent programming environment, and let the model inspect, partition, and recursively query smaller pieces.

We rebuilt that method with an unusual constraint:

no OPENAI_API_KEY;
Codex CLI as the model backend;
gpt-5.4-mini for both the RLM root and every subcall;
a direct frontier model only as a separate baseline.

The result was encouraging, expensive, and more nuanced than “cheap model equals frontier model.”

What an RLM changes

A normal model call looks roughly like this:

large prompt -> model -> answer

An RLM instead gives the root model metadata about the input and a Python REPL containing the real context:

question
   |
root model
   |
persistent REPL holding the context
   |-- inspect and search with code
   |-- split context into useful chunks
   |-- call smaller LMs over those chunks
   |-- validate and aggregate results
   `-- return the final answer

The important detail is that the root model does not need to carry every document, record, tool result, and partial answer in its own context window. Large intermediate values can remain in REPL variables. Subcalls receive focused, locally understandable tasks.

That makes RLM less like a bigger prompt and more like an out-of-core data-processing system whose semantic operator happens to be a language model.

What we actually tested

We used an OOLONG trec_coarse validation example from the protocol described in the RLM work.

The input was a 308,367-character context containing 3,182 general-knowledge questions. Each question implicitly belonged to one of six answer types:

numeric value
entity
human being
location
abbreviation
description and abstract concept

The labels were not present in the context. The task was to infer the labels and identify the least-common category.

We compared:

A direct gpt-5.6-sol Codex call.
An RLM where the root and all leaf calls were locked to gpt-5.4-mini.

The direct frontier call answered abbreviation and scored zero. The mini-only RLM answered numeric value, matching the gold answer.

Method	Result	Model calls	Elapsed time
Direct frontier call	Incorrect	1	40.1 seconds
RLM with `gpt-5.4-mini` only	Correct	At least 238	6,120.3 seconds

The RLM root first inspected the structure of the context. It then classified chunks, retried malformed responses, reduced the chunk size, reclassified all 3,182 questions using structured JSON outputs, checked that it had coverage, and calculated the minimum.

This is exactly the sort of work that a direct model call often approximates but a recursive program can force itself to perform.

The uncomfortable but useful audit

Getting the final answer right did not mean every intermediate judgment was right.

We compared the mini model's inferred counts against the validated labels:

Label	True count	Mini inferred
Numeric value	398	402
Entity	521	623
Human being	544	488
Location	571	493
Abbreviation	571	560
Description and abstract concept	577	616

The model made substantial row-level classification errors. It still found the correct minimum because numeric value had a 123-item margin over the next-smallest true category.

That distinction matters. This run shows that decomposition changed the outcome and allowed a cheap model to solve one problem that the direct frontier call missed. It does not prove that the cheap model reconstructed the data exactly, and one row does not establish general equality between the two systems.

The honest claim is narrower:

On suitable long-context tasks, a cheap model inside an RLM can match or outperform a direct frontier-model call.

Use cases already demonstrated by RLM research

The paper evaluates four useful task shapes:

1. Semantic aggregation

OOLONG requires labeling and aggregating information spread throughout a large input. Real applications include:

customer-feedback analysis;
support-ticket taxonomies;
survey aggregation;
incident and application-log analysis;
quality-control statistics over text records.

Our experiment belongs to this category.

2. Multi-document research

BrowseComp-Plus requires joining evidence across documents in a very large offline corpus. Analogous applications include:

literature reviews;
technical-documentation research;
contract and policy comparison;
due-diligence document rooms;
evidence-backed competitive research.

3. Repository-scale understanding

The paper includes LongBench-v2 CodeQA, where questions require reasoning across files in a codebase. Probable uses include:

architecture mapping;
migration-impact analysis;
dependency and license audits;
security triage;
locating missing tests;
comparing implementation against documentation.

4. Cross-record and pairwise reasoning

OOLONG-Pairs asks the system to construct relationships between combinations of records. Applications could include:

entity resolution;
policy-conflict detection;
matching candidates against constraints;
finding related incidents;
identifying incompatible configurations;
relationship discovery across an archive.

These workloads can grow quadratically, so they need strict budgets and deterministic post-processing.

A practical new use case: making sense of agent-session archives

While exploring our local Claude Code history, we found a single session transcript that was 242 MB and contained 39,570 JSONL records. All project transcripts together occupied about 3.6 GB.

The large session was not 242 MB of useful conversation:

about 176 MB was attachment records;
about 29 MB was assistant events;
about 20 MB was user and tool-result events;
about 12 MB was file-history snapshots.

This is an excellent RLM-shaped problem.

A deterministic first pass can stream the JSONL, hash duplicate attachments, reconstruct parent-child event relationships, merge subagent logs, and extract messages, commands, file changes, tests, commits, errors, and outcomes. An RLM can then analyze normalized episodes and recursively build:

a cross-session project timeline;
a decision register;
a map of attempted and abandoned approaches;
recurring failure patterns;
unresolved tasks;
evidence-linked summaries of what actually shipped.

The final report should cite session IDs, event IDs, timestamps, commands, and Git commits. Otherwise, it is merely another plausible summary.

Other probable use cases

The same decomposition pattern should transfer to:

long incident timelines assembled from logs, tickets, and chat;
scientific evidence extraction across papers and experiment records;
compliance control-to-evidence mapping;
large archives of meetings, email, or project documents;
ranking records against a nuanced rubric;
graph filtering and multi-hop relationship discovery;
reconciling conflicting claims across many sources;
constructing structured datasets from heterogeneous text.

The recurring requirement is not simply “the input is long.” A good RLM task has four properties:

The context can be partitioned or searched programmatically.
Smaller semantic subtasks remain understandable to the cheap model.
Intermediate results can be stored in a structured form.
The final result can be verified or recomputed.

Where RLM is probably the wrong tool

RLM is a poor default for:

low-latency chat;
simple questions that fit comfortably in one prompt;
sparse retrieval where grep or conventional search is sufficient;
creative writing that depends on a single coherent voice;
exact high-stakes decisions without an independent verifier;
public execution of untrusted model-generated Python;
high-volume synchronous APIs with tight latency budgets.

Our successful row took roughly 102 minutes. That is acceptable for a research run or an overnight audit, not for an interactive endpoint.

What a reusable package should look like

The useful abstraction is not an OOLONG runner and not one universal prompt. It is a context-compute runtime with a small set of reusable recipes:

run(
  context,
  objective,
  recipe,
  answer_schema,
  verifier,
  budget
) -> answer + evidence + validation + trajectory + usage

Initial recipes could include:

aggregate_records
evidence_synthesis
repository_analysis
cross_record_join
timeline
candidate_ranking

For our intended configuration, the Codex backend would keep both root and subcalls locked to gpt-5.4-mini. A frontier model would appear only in evaluation runs, never inside the RLM call tree.

Production use would also require an isolated execution environment, call and token limits, schema validation, redaction, prompt-injection defenses, resumable runs, and source-level evidence for every important claim.

What comes next

The one-row result is a proof of mechanism, not a benchmark victory.

The immediate research questions are:

Does the advantage survive across all 50 paired OOLONG tasks?
Can concurrency reduce the 102-minute runtime without changing quality?
Which decomposition recipes transfer cleanly between domains?
How much verification is required for exact row-level work?
Can a mini-only RLM turn multi-gigabyte agent histories into a reliable, source-linked development narrative?

RLMs do not magically turn a cheap model into a frontier model. They change the computation available to that model. Sometimes that difference is enough to turn a wrong one-shot answer into a correct, auditable process.

That is a more interesting result than the slogan.

Reproduction data: the paired OOLONG runs behind these numbers are published at huggingface.co/datasets/Rickesh/rlm-oolong-reproduction.

We made our world model smaller and it got better. Then "efficient" attention made nothing faster.

Rickesh T N — Fri, 07 Aug 2026 05:56:04 +0000

Two experiments on the same world model, both of which came out the opposite way to how I expected. One is now an IJCNN 2026 paper, the other an ICPR 2026 paper. Both are really about the same thing: the scaling intuitions most of us carry around are older than the hardware we run on.

What a world model has to do here

A dynamic scene reconstruction model takes sparse multi-view camera observations across time and predicts a full 3D scene, including how things are moving. Point a few cameras at a street for two seconds, get back a 3D representation of that street with the cars in it moving correctly.

This is the perception half of a world model for driving and robotics. If you want to simulate "what happens if I turn left here," you first need a model that can build the scene at all.

The architecture we worked from is STORM (Yang et al.), a transformer that predicts 3D Gaussian primitives and their motion in a single forward pass. Feed-forward, so no per-scene optimization: a big deal if you ever want this on a vehicle rather than in an offline pipeline. That's prior work, not ours. What follows is what we found building on top of it.

Setup throughout: a subset of the Waymo Open Dataset, 2-second clips at 10 fps, up to three synchronized camera views, images downscaled to 160x240, 150K-300K training iterations.

Experiment 1: we allocated the parameters differently and the smaller model won

The default move when a transformer underperforms is to make it deeper. More layers, more capacity, better results. That instinct comes from language modelling and image classification, and it is mostly right there.

Dense spatio-temporal prediction is not those tasks.

We built a width-dominant, shallow variant. Concretely: fewer transformer layers, larger embedding dimension, and fewer attention heads each with a bigger head dimension. Then we ran it in a smaller configuration against the standard baseline in a larger one, under an identical training and data pipeline.

Model	PSNR	SSIM	Depth RMSE
STORM baseline (B8)	26.82	0.770	12.59
Ours, width-dominant (V6, S8 config)	27.57	0.806	15.59

+0.75 PSNR and +0.036 SSIM, from the smaller model.

The training dynamics were the more interesting part. The width-dominant model converged faster, to a higher plateau, along a visibly smoother trajectory, and its gradient norms were substantially more stable. It was not squeezing out a marginal win at the end of a hard optimization. It was simply an easier model to optimize.

Why this is plausible: self-attention already provides global mixing within every single layer. Stacking more layers buys you more sequential refinement, but sequential depth is exactly what makes optimization harder. Widening instead gives each attention head a richer representation to work with while leaving global information exchange intact. For a task where every token needs to talk to every other token anyway, spending your parameter budget on width rather than depth is not obviously the wrong call, and here it was the right one.

Now the part I want to be honest about: look at the depth RMSE column. The baseline is clearly better on geometry, 12.59 against our 15.59. We got prettier images and worse depth.

That trade-off is real and we reported it as a finding rather than burying it. Photometric quality and geometric accuracy are not the same axis, and if you are building a world model whose output feeds a planner rather than a display, the column we lost on may be the column you actually care about. Anyone reading "+0.75 PSNR" as an unqualified win has read the paper badly.

The practical upshot is deployability. A smaller model that trains on consumer GPUs and beats a larger one on photometric quality changes who can work on this at all.

Experiment 2: attention reuse, and a result I did not want

Self-attention costs O(N² · d_h · H). For spatio-temporal input the token count N is brutal, because you are multiplying pixels by views by timesteps. So the obvious efficiency target is attention itself.

There is a neat line of work on attention reuse, built on the observation that attention maps stabilize as you go deeper into a transformer. If layer 9's attention pattern looks a lot like layer 8's, why recompute the query-key interaction at all? Reuse and transform the previous map instead. You keep the global token interaction and drop a chunk of the compute.

We integrated Less-Attention layers into the STORM-B/8 backbone and evaluated across a range of reuse ratios, measuring runtime, memory, parameter growth and reconstruction quality, under both naive and optimized implementations.

Attention reuse did not improve wall-clock time over optimized full attention. At any reuse ratio. Even with careful implementation.

The reason is not that the theory is wrong. The FLOPs really do go away. The reason is that fused attention kernels on modern GPUs have already moved the bottleneck somewhere else. When full attention runs as a single fused kernel that never materializes the attention matrix in HBM, the thing you were paying for was mostly memory traffic, not arithmetic — and "skip some of the arithmetic" optimizes the wrong resource. You also add parameters and complexity for the reuse machinery, and you now have two kernels where the fused path had one.

This is the whole lesson: an algorithmic optimization derived from a FLOP count is a hypothesis about the hardware, not a fact about it. The hardware moved. FlashAttention-style kernels changed which operations are expensive, and a good chunk of the "efficient attention" literature was implicitly costed against a machine that no longer exists.

I would rather this had worked. Publishing a null result is less fun than publishing a speedup. But a null result that saves other people from re-implementing the same thing is worth more than another 3% on a benchmark.

What I take from both

Capacity allocation matters more than total capacity. Where you put the parameters is a real design decision with real consequences, and "make it bigger" is the answer you give when you have not measured which axis is actually binding.

Measure wall-clock on your target hardware. Always. Not FLOPs, not parameter counts, not asymptotic complexity. Those are proxies, and proxies drift as hardware evolves. Every efficiency claim has a silent "on the machine I tested" attached to it.

Report the column you lost on. Our depth RMSE got worse. Someone building on this needs to know that far more than they need another decimal place of PSNR.

The intuitions are load-bearing and mostly untested. "Deeper is better" and "fewer FLOPs is faster" are both reasonable priors that happened to be wrong in our setting. I only found that out by running the ablation instead of assuming.

If you work on neural rendering, world models, or transformer efficiency, I would genuinely like to hear whether the width-over-depth result holds in your setting. My prior is that it generalizes to dense prediction tasks and not much further, but that is a prior, and this post is largely about how those go.

Papers: "Rethinking Transformer Design for Dynamic Scene Reconstruction: An Efficient, Width-Dominant Approach" (IJCNN 2026) and "Evaluating Attention Reuse in Dynamic 3D Gaussian Reconstruction" (ICPR 2026).

I write about world models, robotics data pipelines, and the ways measurement quietly fails. linkedin.com/in/rickeshnatarajan

Three Ways Your Training Data Lies to You (And None of Them Throw an Error)

Rickesh T N — Fri, 07 Aug 2026 00:21:27 +0000

Every failure I am about to describe produced a clean run. No exception, no stack trace, no red build. Each one produced a plausible number that I believed for longer than I should have.

That is the category of bug I have come to fear most. A crash tells you it crashed. A silently broken dataset tells you nothing at all, and your metrics will politely agree with it.

Here are three from the last year, all from my own work, all found late.

1. The dataset that was 92% one category

I had a training set of 688 records for a multi-category vision-language task. Thirteen categories. Reasonable size for a fine-tune, already used in a completed training run whose results I had written up.

While preparing a stratified split, I joined the records back against the source annotations and actually counted the categories.

630 of 688 were a single category: scene captions. Zero examples of traffic signals. Zero of planning. Zero of uncertainty. Several categories the evaluation explicitly measured had no representation in training at all.

The previous fine-tune had shown gains on some of those very categories. I had interpreted this as the model learning the task. The real explanation was duller and more useful: the model had learned the answer format from caption supervision, and format alignment alone was enough to move a multiple-choice score. Nothing category-specific had been learned, because nothing category-specific had been shown.

The root cause was upstream and boring. The conversion script I inherited only rewrote file paths and dropped records with missing frames. It faithfully preserved a caption-only selection made further up the chain. It had no opinion about balance because nobody had asked it to have one.

What I changed: the composition of a training set is now an artifact I generate and inspect before any run, not a property I assume. A category histogram takes seconds. I had not looked, for months.

2. The 18-hour run that converged perfectly to nothing

Large model, QLoRA, multi-GPU, 8 camera views per sample. Roughly a full day of compute.

The loss curve was beautiful. It fell from 19.4 to 15.7 over the first 26 steps, then to 0.078 by step 51, then flattened near 0.02 and stayed there. Token accuracy reached 0.99. Gradient norms decayed smoothly. Nothing in the training telemetry looked wrong.

The task metric was 0.10.

The cause: I was supervising on free-text answers, median 11 words, phrased like "One should keep to the right side of the road and drive slowly." The evaluation was four-option multiple choice, scored on the index of the chosen option.

The model learned, quickly and correctly, to reproduce the style of the training answers. That is genuinely what the loss was asking for, and it maxed it out in about 50 steps. It was never asked to select an option, so it never learned to.

Training and evaluation were measuring different tasks. Both were internally consistent. Neither could detect the other's disagreement.

What I changed: before a long run starts, I now write down what the eval measures and what the loss optimizes, in one line each, and check they describe the same thing. When loss collapses to near zero within a few dozen steps, I treat that as an alarm rather than a success. Genuine learning on a hard task does not look like that.

3. The dataset that was 40% smaller than its file size claimed

Assessing a large public robotics corpus for adoption. The headline: 3.91 TB compressed, 4.65 TB extracted, hundreds of shards, dozens of buildings, thousands of hours of teleoperated manipulation.

The number that mattered was buried in the paper's method section. Only about 60% of the raw data converted into reliable 3D flow annotations. The rest failed depth estimation, camera pose optimization, or point tracking. Then a further filter kept only trajectories with actual robot-object contact and real object motion.

Nominal size: thousands of hours. Training-ready size after both filters: roughly 500 hours.

Every plan built on the first number was wrong by a factor of several. Storage estimates, download time, compute budget, and most importantly the question of whether the corpus was even large enough for what we wanted.

What I changed: I now treat yield as the primary dataset metric, not size. What fraction of raw capture survives every stage into training-ready data? For collected data this is a hard operational KPI, and it is the one that determines cost per usable hour. Bytes on disk is a storage figure. It tells you almost nothing about what you can train on.

The pattern

All three share a shape.

A pipeline stage did exactly what it was written to do. A metric moved in the direction that metric moves when things go well. And the thing I actually cared about was not being measured by anything at all.

The defenses that work for me:

Check that work happened, not just that output exists. A gate that verifies "400 rows were produced" while never checking "any row succeeded" will pass a completely dead run. I once had an evaluation return zero correct on every row for 45 minutes because a kernel was unavailable on that GPU. Zero is a valid ratio. Row counts agreed. The gate passed and the sweep moved on.

Count infrastructure failures separately, and require zero. A weak model produces well-formed output and scores badly. A broken pipeline produces nothing and scores identically. Collapsing those two into one number destroys the only signal that distinguishes them.

Emit progress with errors first. Anything running longer than a minute should print failure count before position before metric. Silence is not neutral. A job working perfectly and a job failing on every single item look exactly the same from outside when neither prints anything.

Validate splits as their own gated job. I run split validation as a separate step that must pass before training is allowed to start. It checks class balance, domain distribution across sites, and leakage between splits. It has caught things that would otherwise have cost days of GPU time and, worse, produced a believable number.

Write down what the metric would look like if the thing were broken. If you cannot answer that, you cannot tell success from failure, and you will default to reading any completed run as a successful one.

The uncomfortable part

In every one of these cases the data was the limiting factor, and in every case I spent the first stretch of debugging looking at the model.

That instinct is hard to unlearn, because the model is where the interesting work feels like it lives. But architecture, optimizer, and learning rate are all things I can inspect in a config file in about a minute. The composition of the training set is something I have to deliberately go and measure, and that extra step is exactly why it goes unchecked.

The failure mode is not carelessness. It is that broken data produces output that looks entirely normal.

I work on robotics data pipelines, model evaluation, and the operational side of making training data trustworthy. linkedin.com/in/rickeshnatarajan

Your reasoning model isn't dumb. Your parser is throwing away its best answers.

Rickesh T N — Fri, 07 Aug 2026 00:19:57 +0000

I benchmarked a vision-language model and scored it at 0.31.

The real number was 0.70. Same model, same weights, same hardware, same 100 questions. The only thing that changed was how I read its output.

I had already written up the 0.31 as a capability finding and concluded the model was unsuitable. That conclusion was wrong, and the failure was entirely in my harness. Here is the mistake, because I doubt I am the only one making it.

The setup

I was evaluating a batch of open-weight and frontier models on a multiple-choice benchmark: multi-view driving scenes, four options per question, one correct answer. Standard stuff. The prompt asked for reasoning followed by a final line, Answer: X.

My scoring code did the obvious thing:

m = re.search(r"Answer:\s*([A-D])", output)
pred = m.group(1) if m else None   # None scores as wrong

That last comment is the bug.

What actually happened

The model I was testing is a "thinking" model. It emits a long internal reasoning trace before it commits to an answer. I had a generation budget of 1024 tokens.

On easy questions it reasoned briefly, emitted Answer: B, and scored fine. On hard questions it reasoned at length, hit the token cap mid-thought, and never emitted the answer line at all.

So the harness scored every one of those as wrong.

64 of 100 questions returned no parseable answer. Zero of those were image-loading errors or crashes. They were all truncation. And the truncation was not random:

Uncertainty                 0/8   answered
Counterfactual              0/3   answered
Safety-critical Planning    1/11  answered
Safety-critical Prediction  3/12  answered

Look at that distribution. The questions the model failed to answer were precisely the questions that required the most reasoning. My harness was systematically discarding the model's performance on exactly the hard subset I was trying to measure, and reporting the result as a capability ceiling.

Of the 36 it did answer, it got 86% right. The model was fine. My measurement was garbage.

The fix

Stop parsing free text. Constrain the decoding to a schema.

With Ollama:

resp = ollama.chat(
    model="my-vlm:9b",
    messages=[{"role": "user", "content": prompt, "images": imgs}],
    format={                       # enforced at decode time
        "type": "object",
        "properties": {"answer": {"type": "string", "enum": ["A","B","C","D"]}},
        "required": ["answer"],
    },
    think=False,
)

The equivalent exists nearly everywhere now: response_format with a JSON schema on OpenAI-compatible endpoints, structured outputs in vLLM via guided_json, Outlines, or plain grammar-constrained sampling in llama.cpp.

The point is that the model can no longer produce an unparseable output. The constraint is applied during sampling, not checked afterwards.

Re-ran the same 100 questions: 0.70, with a 100% answer rate. The categories that had been at zero came back at 0.82 and 0.75.

The second-order trap

I hit a related version of this on a larger model in the same family, and the fix was less obvious.

That model also truncated, on 36 of 100 questions. I had a fallback that recovered an answer letter from the tail of a truncated trace, so those rows produced something. The score came out 0.67.

But the recovered answers were junk in a specific, dangerous way. Predicted answers skewed heavily toward option A (48 predictions of A against 23 in the ground truth). When the model gets cut off mid-reasoning, the letter you scrape from the tail is not a decision, it is whatever token happened to be nearby. That is position bias, and it looks exactly like a real answer to your scorer.

Rows where the reasoning actually completed scored 0.83. I re-ran just the truncated rows with a budget of 8192 instead of 3072, and the merged score came to 0.73.

So: tail-recovery is worse than a non-answer. A non-answer is visibly missing. A biased recovered answer silently contaminates your accuracy in a direction you did not choose.

Note also that 16 of those 36 questions still truncated at 8192 tokens. 0.73 remains a floor, not a measurement.

What I do now

Match the evaluator's generation budget to the training or intended-use budget. Mismatched budgets scored 31% of my rows incorrectly in one run. If you fine-tuned at 512 tokens, do not evaluate at 128.

Track truncation as a first-class metric. Log clipped_ratio and parseable_rate next to accuracy. If either moves, your accuracy number is not comparable to the previous run. I now fail a run outright if the parse-failure count is anything other than what I expect.

Distinguish infrastructure failure from poor performance. An unparseable output is a harness event, not a model event. They must be counted separately, always. A weak model still produces well-formed answers and simply scores badly.

Never let a non-answer score as wrong by default. Make it raise, or count it in its own bucket. Silently mapping "I could not read this" to "the model was incorrect" is how you get a confident, published, wrong conclusion.

Verify the harness against a known-good signal before trusting any number it emits. An evaluation harness that has never been checked against a baseline you can sanity-check by hand is untested code that produces numbers, not a measurement instrument.

Why this matters more than it used to

Reasoning models broke an assumption that free-text answer parsing quietly relied on for years: that a model's answer appears in its output. With extended chain-of-thought, the answer arrives last, and last is exactly what a token budget truncates.

Every leaderboard comparing a thinking model against a non-thinking one under a shared token budget is, at minimum, measuring something other than what it claims. The thinking model pays for its reasoning out of the same budget that has to carry its answer.

I published 0.31 as a capability limit and had to retract it. The model was never the problem.

I write about ML evaluation, robotics data pipelines, and the ways measurement quietly fails. linkedin.com/in/rickeshnatarajan