A 7M-Param Model Just Beat GPT on ARC. Now What?
Here is the headline that broke Hacker News today: a developer trained a small transformer from scratch in 90 minutes on a single RTX 5090, scored 44% on the ARC-AGI-1 benchmark, and posted the results to a blog that promptly hit 595 points and 155 comments. Meanwhile, Samsung's Tiny Recursive Model (TRM) — just 7 million parameters, two layers deep — has been quietly outscoring DeepSeek R1, o3-mini, and Gemini 2.5 Pro on ARC-AGI-2 since late 2025. And on GitHub, minimind — a project that lets you train a 64M-parameter LLM from scratch in two hours for three dollars — just crossed 57,000 stars with over 1,000 new stars today alone.
📖 Read the full version with charts and embedded sources on ComputeLeap →
Something is happening in the space between "tiny" and "frontier," and it is not what the scaling maximalists predicted.
View discussion on Hacker News →
The Benchmark That Started a War
The ARC-AGI benchmark — the Abstraction and Reasoning Corpus created by Keras creator Francois Chollet — was designed to test the kind of fluid intelligence that LLMs are supposed to lack. Each puzzle is a visual grid transformation: given a few input-output examples, predict the output for a new input. No language, no memorization, no retrieval. Pure pattern recognition and rule inference.
For years, frontier LLMs threw billions of parameters at ARC and barely moved the needle. Then two things happened in quick succession.
First, Samsung SAIL Montreal published TRM. At 7 million parameters — roughly 0.01% the size of GPT-4 — it scored 45% on ARC-AGI-1 and 8% on ARC-AGI-2. For context, Gemini 2.5 Pro manages 4.9% on ARC-AGI-2. DeepSeek R1 hits 1.3%. Claude 3.7 scores 0.7%. A model that fits on a USB drive was outperforming systems that cost hundreds of millions to train.
Nature covered it with the kind of restrained excitement the journal reserves for results that might actually matter.
Second, Mithil Vakde — a solo developer — trained his own small transformer and hit 44% on the public eval set. Single GPU. Ninety minutes. No billion-dollar lab required. His technique used test-time training, 3D RoPE embeddings, and aggressive data augmentation through color and dihedral permutations. In the HN thread, he was blunt about what this means:
💡 "This is NOT an LLM. It's a small autoregressive transformer trained from scratch. Sample efficiency is one of the most important unsolved problems today in AI." — Mithil Vakde, author of the 44% ARC-AGI-1 result
The HN community erupted. The thread drew 595 points and 155 comments — making it the top-debated AI story of the day, running neck-and-neck with the GPU World essay (394 points, 272 comments), which argued the opposite case: that GPU-scale compute is the defining infrastructure of the coming decade.
How a 7M-Parameter Model Thinks
TRM does not work like an LLM. It does not predict the next token. Instead, it maintains two pieces of short-term memory: a current solution ("y") and a scratchpad for intermediate reasoning ("z"). At each step, the model reviews the task, its current answer, and its prior notes, then produces an improved output. This loop repeats up to 16 times.
Think of it as a student who writes a draft answer, reviews it, fixes mistakes, reviews again, and keeps revising. The architecture is a two-layer network — not a deep transformer stack — and the magic is entirely in the recursive refinement loop.
Alexia Jolicoeur-Martineau, the lead researcher at Samsung SAIL Montreal, titled the paper "Less is More" — and Francois Chollet himself called it "impressive work."
But here is the part the breathless headlines miss: TRM is not a language model. It does not generate text. It does not chat. It is a supervised reasoning model for tasks with structured input/output grids that produces deterministic answers. Its dominance on ARC does not mean you can shrink ChatGPT to 7M parameters and get the same results.
The "Build Your Own" Wave
While researchers debated benchmarks, a parallel movement was building on GitHub. The tools to train your own small models have gotten absurdly accessible.
minimind is the poster child. Created by Jingyao Gong, it lets you train a 64M-parameter LLM from scratch in about two hours on a single consumer GPU for roughly $3. The latest release (minimind-3, April 2026) aligns its architecture with Qwen3, includes MoE variants, and even supports agentic RL for multi-turn tool-use training. It crossed 56,900 stars today — gaining over 1,000 in a single day.
Soup takes a different angle: fine-tuning existing models with one YAML file and one command. Its killer feature is layer streaming — training an 8B model on a GPU with just 4GB of VRAM by streaming frozen base layers from host RAM one decoder layer at a time. No SSH. No infrastructure expertise. 4,400+ stars and climbing.
And freeCodeCamp published a full tutorial on training your own LLM from scratch, while AI Engineer hosted a workshop with ElevenLabs on doing it locally.
The message is consistent: training a small model is no longer a research-lab activity. It is a weekend project.
What the Community Is Really Saying
The HN thread on Vakde's ARC result is one of the more technically substantive discussions you will find on that site this year. The debate crystallized around three fault lines.
View discussion on Hacker News →
Fault line 1: Is this real generalization or benchmark gaming? User bbor questioned whether the transductive approach — training on the test inputs (without labels) — exploits dataset limitations rather than achieving genuine intelligence. Vakde's response was pointed: "Weights are irrelevant. The training algorithm is what is AGI."
Fault line 2: Small vs. large is a false binary. Several commenters noted that the small models beating LLMs on ARC are purpose-built for structured grid puzzles. They do not write code, hold conversations, or synthesize information across domains. Commenter ACCount37 put it plainly: "ARC-AGI benchmarks specifically target spatial reasoning weaknesses in LLMs rather than claiming AGI-completeness."
Fault line 3: The real revolution is in training economics. Multiple commenters pointed to minimind and similar projects as the bigger story — not because small models will replace frontier systems, but because the ability to train task-specific models cheaply changes the calculus for every engineering team.
The GPU World thread, running simultaneously, offered the counterweight. Commenter maxnevermind argued: "LLMs gave us nice productivity tools for highly motivated expert knowledge workers, that is all." While david-gpu countered by invoking the early web: "Did you experience the popularization of the WWW? There were massive disagreements even among smart tech-savvy people."
The two threads together — small-model triumphalism vs. GPU-scale maximalism — capture the defining technical argument of this cycle.
⚠️ Contrarian Corner: Why "Small Beats Big" Is Misleading
The narrative is seductive but imprecise. Here is what the results actually show:
Task-specific architectures beat general-purpose ones on task-specific benchmarks. This has always been true. A hand-written regex beats GPT-4 at email validation too.
Recursive refinement is powerful for structured problems. TRM's iterative draft-revise loop works brilliantly on grid transformations. It does not work for open-ended generation, multi-step planning, or tasks where the output space is unbounded.
Scaling skeptics are fighting yesterday's war. Frontier labs have already shifted from raw parameter count to inference-time compute, distillation, and architectural innovation. The next generation of small models — Phi-4, Gemma 4, SmolLM3 — are themselves products of scaling. You need a massive model to distill a good small one.
The "GPU-Poor Manifesto" has a point — but so does the GPU World. Smaller models, better routing, and tighter generation control can handle most production workloads. But someone has to train the frontier models that make distillation possible. That someone needs a lot of GPUs.
The honest conclusion: scale and cleverness are not opponents. They are collaborators on different timescales.
The Tooling Stack: What Works Today
For practitioners who want to experiment with small model training, the tooling has reached a genuine inflection point.
| Tool | What It Does | Hardware Required | Time Investment |
|---|---|---|---|
| minimind | Train 64M LLM from scratch | Single GPU (3090+) | ~2 hours |
| Soup | Fine-tune 8B model | 4GB VRAM laptop GPU | Hours |
| TinyRecursiveModels | Recursive reasoning on ARC | Research GPU | Variable |
| SmolLM3 | Pre-trained 3B reasoning model | Consumer GPU | Minutes (inference) |
The pattern is clear: the "train your own" movement is not about replacing frontier models. It is about expanding the space of problems you can solve without paying per-token API costs. If your problem has structured inputs and outputs — classification, entity extraction, format conversion, domain-specific reasoning — a small trained model is often better, faster, and cheaper than an API call to GPT-4.
Where This Breaks Down
Small models excel at narrow, well-defined tasks. They fail at precisely the things that make frontier LLMs valuable:
- Open-ended generation — writing marketing copy, brainstorming product names, explaining concepts to different audiences
- Multi-step reasoning across domains — "read this contract, cross-reference with the tax code, and summarize the implications"
- Few-shot generalization — performing well on tasks the model has never seen, with only a handful of examples
- Tool use and agentic behavior — orchestrating API calls, managing state, recovering from errors
The GPU-Poor Manifesto makes a compelling case that small models can handle most production workloads. But "most" is doing heavy lifting in that sentence. The workloads that frontier models uniquely enable — complex agentic workflows, multi-modal reasoning, creative generation — are also the ones generating the most commercial value.
Sara Hooker's observation from her 2026 essay "On the Slow Death of Scaling" is the most balanced framing: Falcon 180B (2023) was outperformed by Llama 3 8B (2024) just one year later. The efficiency frontier moves fast. But it moves fast because someone pushed the scale frontier first.
What This Means for You
💡 Practitioner Playbook:
Audit your LLM API calls. If more than 30% of your calls are structured I/O tasks (classification, extraction, format conversion), you are probably overpaying. A fine-tuned small model can handle these at a fraction of the cost.
Try minimind as a learning exercise. Training a 64M model from scratch teaches you more about transformers than a hundred blog posts. The learning roadmap is genuinely good.
Use Soup for production fine-tuning. If you need a task-specific model and have limited GPU budget, layer streaming on consumer hardware is a real option now.
Keep frontier models for frontier tasks. Complex reasoning, creative generation, agentic workflows — these still need the big models. Do not try to solve open-ended problems with a 64M-parameter model.
Watch the distillation pipeline. The most practical "small model" strategy for most teams is distilling from a frontier model, not training from scratch. As Phi-4 and SmolLM3 demonstrate, distilled small models are closing the gap fast.
The Defining Question
Today's Hacker News front page captured something important. Two stories, running simultaneously, making opposite arguments:
- A solo developer proved you can beat billion-parameter LLMs on hard reasoning tasks with a small transformer and 90 minutes of GPU time
- An essay argued that GPU-scale compute is the most important infrastructure buildout since railroads
Both are right. The question is not "scale vs. cleverness" — it is "scale for what and cleverness for what." The 7M-parameter TRM that beats Gemini on grid puzzles cannot write you a business plan. The GPT-4 that writes your business plan cannot match TRM on grid puzzles.
The real winner in this cycle is not team "small" or team "big." It is the practitioner who understands which problems deserve which tool — and who has the skills to train a task-specific model when the situation calls for it.
That is the genuine democratization story. Not that scaling is dead, but that you no longer need a billion-dollar lab to build something useful.
The tools are ready. The question is whether you are.
For more on the economics of AI scaling, see our analysis of why the scaling math is breaking and what inference cost dynamics mean for builders.
Originally published at ComputeLeap
Top comments (0)