DEV Community: Codecora Dev

A Small Transformer Trained in 1.5 Hours Beat Many LLMs on ARC

Anaz S. Aji — Wed, 02 Sep 2026 06:56:56 +0000

Mithil Vakde trained a small transformer from scratch in 1.5 hours on a rented RTX 5090. The compute cost came to about 67 cents. The model scored 44% on the ARC-AGI-1 public eval, one point behind TRM's 45% and in the same band that left DeepSeek R1, o3-mini, and Gemini 2.5 Pro behind when the TRM paper ran the comparison (arXiv 2510.04871).

This is the third post in his ARC series. The previous result went viral and drew public scrutiny from researchers including Lucas Beyer, Jeremy Howard, and Rohan Anil. The new one raises the score, cuts the cost, and answers every objection in a long appendix.

What the model does

Each puzzle becomes a sequence of tokenized grids. The transformer trains on those sequences from scratch at test time, using both the train puzzles and the eval puzzle inputs, with the eval labels hidden. Test-time training is not a loophole here. ARC is a meta-learning benchmark built around skill acquisition, and Chollet's original paper frames the whole thing as a measure of how efficiently a system gains new skills (arXiv 1911.01547).

The architecture got a 2026 refresh: 8 layers, SwiGLU, RMSNorm, the NorMuon optimizer, 3D RoPE, and per-task embeddings. Inference augments each test input with color and dihedral permutations, inverts the augmentations, and submits the two most common outputs.

The biggest change is the loss function. Previous versions trained on input and output tokens. This one trains on output tokens only, which makes the approach supervised. Score went from 40% to 44%. Vakde admits he does not fully understand why, and the training loss got worse even as scores improved. His guess is finite model capacity. The ablations back him up on what carries the result: drop the 3D RoPE and the score falls to about 24%, drop the per-task embeddings and it falls to about 24%. Representations, not compute, are doing the work.

The lineage it joins

None of these systems pretrain. They learn from the benchmark's roughly thousand examples:

System	Parameters	ARC-AGI-1	Source
CompressARC	76K	20%	arXiv 2512.06104
HRM	27M	40%	arXiv 2506.21734; score from 2510.04871
TRM	7M (see caveats)	45%	arXiv 2510.04871
Vakde's model	small (1.5 hrs training)	44%	author's blog

In about a year, from-scratch systems went from solving a fifth of the eval to nearly half of it, and the cost per result collapsed from research-cluster budgets to spare change on a rented GPU.

Vakde also measured what happens when you strip his advantages. Restricting training data to ARC-1 plus ConceptARC still scores about 40%. A CompressARC-style unsupervised variant drops to about 18%. He thinks 65% is reachable within the current framework, and a union of solved tasks across his own runs already reached 55%. One request to contributors: do not add more training data.

Where the skeptics have a point

The 67-cent figure deserves a careful read. It counts lifetime compute: training from initialization plus inference on all tasks, which works because every task is trained at once. Dividing that total by task count would amortize it unfairly against models charged per task, and Vakde now compares only against TRM, HRM, and CompressARC for exactly this reason. LLM cost figures, on the other side, leave out pretraining entirely, so no single cost axis treats every approach the same.

Parameter claims need the same scrutiny. He argues TRM's famous "7M parameters" leaves out O(100M+) trained embedding weights, and that "7M active weights" would be the honest label. Same caution for HRM.

The score itself stays humble. 44% means 56% of puzzles still fail, and 7% on ARC-2 is early days. OpenAI's o1 reached 75% on ARC-1 through massive post-training (ARC Prize 2024 report), and frontier systems still lead the private leaderboard. This result is about efficiency per training dollar, not about beating GPT-class models across the board. Public eval scores for LLMs are also polluted, since the answers circulate on the internet and end up in training data.

And this is one researcher's unreviewed work. The previous round got picked apart in public, and while the appendix answers the objections convincingly, independent replication is still thin.

Efficiency without the training bill

The pattern matters more to us than the puzzle. Small-model efficiency used to arrive with an invoice: distillation runs, quantization-aware training pipelines, weeks of compute before the first benchmark.

Vakde's result says the invoice is optional on the reasoning side: train from scratch at test time, in 1.5 hours, for 67 cents. Our own work says the same on the storage side. vecq, our training-free vector quantization library, gets its 4-bit and 5-bit codes from Lloyd-Max centroids that are fixed constants. There is no training pass to pay for at all.

In our benchmarks, 4-bit codes hit recall@10 of 0.958 at 5.98x compression, scanning at 0.89 ms per query on ARM, and the current release defaults to 5-bit at 4.79x compression with 0.979 recall@10. The same quantization math scales all the way up to an 87GB model on a CPU. Inside our memory engine, the pipeline reached 98.2% recall against ChromaDB and Mem0 without any learned quantizer. The brute-force scan does give back some speed to HNSW, and why we pick that trade is an architectural choice for edge hardware, not an accident.

What to watch

The interesting shift is the price of admission for credible efficiency results: no pretraining run, one GPU, under two hours, spare change. Chollet's skill-acquisition framing finally has a small-model track with real momentum behind it, and Vakde argues the organizers should go further and ban offline pretraining to make ARC a true sample-efficiency test. His own result is the existence proof that such a rule is survivable.

The author believes 65% is reachable and that hand-written GPU kernels could cut his costs another 10x. Either would have sounded like trolling a year ago.

References

Mithil Vakde, "44% on ARC-AGI-1 in 67 cents"
"Less is More: Recursive Reasoning with Tiny Networks"
"Hierarchical Reasoning Model"
"ARC-AGI Without Pretraining"
Francois Chollet, "On the Measure of Intelligence"
"ARC Prize 2024: Technical Report"

This article is crossposted with blog.codecora.dev as the canonical source.

Why bigger context windows won't kill memory systems

Anaz S. Aji — Wed, 02 Sep 2026 03:57:24 +0000

A claim keeps making the rounds: context windows are at a million tokens and climbing, so external memory is a stopgap that better models will make obsolete. If that's true, we picked a strange moment to publish a memory benchmark. We weren't worried, but we did want to know what the evidence says. So we read the papers.

What a million tokens does to information

The seminal result is Liu et al's "Lost in the Middle" (arXiv 2307.03172). Tested on multi-document QA and key-value retrieval, models performed best when relevant information sat at the beginning or end of the context, and degraded hard when it sat in the middle.

For a while you could hope that was an engineering wart. Then a 2026 paper derived the same U-shaped retrieval profile from a theoretical model of attention dynamics (arXiv 2605.09213). The position bias falls out of the math. Bigger windows inherit it. They don't remove it.

Practice matches theory. In a white-box study of coding agents, pass rate fell from 8 of 10 runs in a clean 10,991-character context to 3 of 10 runs at 299,140 characters, and padding the context with irrelevant material hurt as much as relevant code (arXiv 2607.17937). Half the drop had nothing to do with relevance.

Long-horizon search shows the same shape. Across four flagship models and three benchmarks, researchers documented "premature termination": models give up or answer with falling confidence long before they approach the window limit (arXiv 2606.29718).

The economics nobody mentions

Everything inside the window is re-paid on every call, in every session. A fact from last March costs the same as something the user said ten seconds ago, every single time you send it.

Retrieval inverts that curve. Store once, then pay for a small set of relevant records when they're needed.

There's a real trade-off inside memory design itself. Keep raw records and you preserve detail but burn budget. Consolidate and you cover more ground per token but risk dropping a query-critical fact (arXiv 2607.17545). Both strategies argue for the same thing: a small, deliberately chosen working set, not a firehose.

Memory is not a small context window

LongMemEval (arXiv 2410.10813) breaks long-term memory into five measurable abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Look at that list. None of them mean "hold more tokens at once."

Knowledge updates are the cleanest example. When a user changes deploy targets, the old value has to be overwritten. In a million-token window, the stale fact and the current one both sit there, and the model has to guess which wins. A memory system that tracks updates resolves it by design.

The field is still moving. LongMemEval-V2 landed in May 2026 and pushes evaluation toward whether an agent becomes an experienced colleague in its environment. Nobody builds benchmarks for problems they consider solved.

When big context wins

Credit where it's due. Reasoning over one large document you already have, one-shot analysis that needs the whole thing in view at once, needle-style lookups inside a single snapshot: if the world fits in the window and everything in it is current, fill it. Memory systems earn their keep when history is long and facts change across many sessions, which describes most real assistants after their first week.

Our numbers, reproducible

We build a memory engine (Uteke) and we test it the boring way. On LongMemEval-S with 500 questions, it scores 98.2% recall_any@5. Then we re-ran the entire suite on a different CPU architecture: 107 of 108 questions produced identical rankings, and the one that moved was an adjacent-rank near-tie we traced to float summation order.

The harness is in the repo. Run your own, and treat numbers you cannot reproduce, including ours, as marketing.

Verdict

Bigger windows raise the ceiling on working memory. Persistence, salience, and overwrite stay separate jobs, and the field is now measuring them directly. Context and memory do different jobs. The agents people keep coming back to are the ones that stopped starting from zero.

References

Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172)
Kinetic theory for Transformers and the lost-in-the-middle phenomenon (arXiv 2605.09213)
When and How Context Rot Appears in Coding Agents (arXiv 2607.17937)
Diagnosing and Mitigating Context Rot in Long-horizon Search (arXiv 2606.29718)
Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory (arXiv 2607.17545)
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (arXiv 2410.10813)
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues (arXiv 2605.12493)

Full write-up lives on our blog, canonical source for this post.

We re-ran our benchmark on a different CPU architecture. 107 of 108 matched.

Anaz S. Aji — Tue, 01 Sep 2026 14:06:41 +0000

We publish benchmarks for Uteke, our open-source memory engine. The headline this week: 98.2% recall on LongMemEval-S. 500 questions, zero config, no LLM in the retrieval path.

Then we asked the question every benchmark should answer: does the number survive on different hardware?

The published run came from cloud x86 infrastructure. The re-run happened on a 4-core ARM desktop. Same v0.16.0 binary, same public harness, 108-question subset.

We compared per question, ranking against ranking.

107 of 108 identical.

The one difference is the interesting part. Both runs retrieved the same top-10 sessions. One session sat at rank 5 in the published run and rank 6 in the re-run. Our retrieval fuses two rankings with Reciprocal Rank Fusion, which is floating-point math all the way down. Cross a CPU architecture boundary and near-ties can round differently. That is the whole story: one adjacent-rank flip, and the subset aggregate moved from 100.0% to 99.4%.

If you compare memory systems for a living, the field guide we wrote alongside this matters more than our numbers. Check what the figure is:

recall_any@K means at least one gold session lands in the top K. recall_all@K means all of them. Vendors mostly publish the first family. Our own run scores 98.2% on the first and 88.0% on the second, from the same data. Ten honest points apart.
The cutoff matters. Recall@15 is easier than Recall@5.
Sample size matters. 100 questions is a smoke test, 500 is a validation run.
Version and date matter. Benchmarks are snapshots of moving software.
The harness being public matters most. A table is marketing. A harness you can clone is a benchmark.

The full post has the comparison chart with per-metric disclosures, the full field table (Mnemosyne, MemPalace, agentmemory, Supermemory, each with its metric labeled), and what we deliberately do not claim:

Read the full breakdown: https://blog.codecora.dev/we-published-our-benchmark-then-we-tried-to-break-it/

Don't trust our benchmark. Run your own. The harness and reproduction results are in the repo.

Cross-posted from the CodeCora blog, where we write about local-first AI infrastructure.

An 87GB model ran on a CPU at 8 tok/s. Quantization did that.

Anaz S. Aji — Tue, 01 Sep 2026 06:44:17 +0000

Someone on r/LocalLLaMA benchmarked Qwen3.8-Flash-Next in llama.cpp from CPU-only all the way to 96GB of VRAM. The model file is 87.2 GiB. On a plain CPU it decodes at 8.34 tok/s. With the full 96GB, 109 tok/s.

The number that looks like a party trick is the CPU one. It is a compression story. The weights are 4-bit. At 16-bit the file would be around 4x bigger, north of 300GB, and no consumer setup loads that, GPU or not.

Two more findings from the benchmark worth stealing:

At 245K context, the 96GB advantage over 24GB shrinks from 2.80x to 1.45x. Long context flattens everything.
Dropping the model's 27.2 GiB embedding table onto the GPU cut decode from 108.5 to 1.95 tok/s. RAM placement was 55.6x faster. Where a tensor lives matters as much as how much VRAM you have.

Why this matters here: embeddings hit the same wall at smaller scale. Every f32 embedding at 768 dims costs 3,072 bytes. A 100k-vector agent memory is about 300MB. Fine on a server, real money on a phone.

We built vecq for that. Training-free vector quantization in Rust (Apache-2.0, crate: vecq-core). No calibration data, deterministic results across platforms. On real EmbeddingGemma vectors: 4.78x smaller at default width (642 bytes/vector) with recall@10 of 0.979, and the index builds 14x faster than HNSW.

Honest trade: search is brute force, about 14x slower than HNSW at default width. Made for on-device indexes in the thousands of vectors, not million-vector servers. The file format is versioned and readers accept older versions, so indexes keep loading across releases.

Full breakdown with the benchmark tables is on the blog:

👉 https://blog.codecora.dev/quantization-cpu-to-96gb-and-embeddings/

Repo: https://github.com/codecoradev/vecq
Crate: https://crates.io/crates/vecq-core