Reasoning scores keep climbing. Per-token compute keeps dropping. And factual knowledge — the kind you'd actually need to answer a straightforward question — is quietly being cut from frontier models.
The latest generation of AI models is getting measurably worse at factual recall, and the labs building them know it. It's not a bug. It's a design decision, and understanding why tells you something important about where AI is heading.
The Benchmark Paradox
Here's the strange state of AI in 2026: GLM-5.2 scores 99.2% on AIME 2026 with about 40 billion active parameters per token. Qwen3.5 scores 91.3% with 17 billion active. DeepSeek V4-Flash runs 13 billion active parameters. For comparison, GPT-4 was rumored to run around 280 billion active parameters in 2023, and it could barely solve an AIME problem.
At the small end, Qwen3.5 9B fits in 6GB of VRAM quantized and roughly doubles the score of the next best model under 10B parameters on Artificial Analysis's intelligence index. By every reasoning and math benchmark, models are getting smarter per parameter at an absurd rate.
Ask the same models a plain factual question and the picture flips. On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%. The best factual recall money can buy still misses half the questions. The small models barely register — Artificial Analysis measures Qwen3.5 4B and 9B at hallucination rates of 80 to 82% on its knowledge benchmark. When they don't know a fact, which is most of the time, they make one up.
Ask a 9B model for the birth year of a minor 19th-century mathematician and you get a confident, plausible, wrong answer.
The Trade Nobody Talked About
The parameter count didn't drop for free. Labs are trading world knowledge for reasoning skill, and the trade is deliberate.
Here's why it works: facts take space. Research on knowledge capacity — the "Physics of Language Models" series has the cleanest measurements — puts it on the order of two bits of factual knowledge per parameter. If you want a model that knows the birth year of every minor Wikipedia figure, the population of every Dutch municipality, and the argument order of every function in every npm package, you pay for that in weights. And it's a big part of why frontier models grew to trillions of parameters.
Reasoning compresses much better than facts do. It's a relatively small set of procedures applied over and over: break the problem into parts, track intermediate state, check your own work, backtrack when a step fails. Distillation and reinforcement learning on verifiable tasks turn out to transfer those procedures into small models remarkably well.
Phi-4 is 14 billion parameters, trained heavily on synthetic textbook-style data, and it's good at math and bad at trivia. That mix used to look like a limitation of the synthetic-data approach. It now looks like the design goal.
The Shape of Knowledge That Survives
The knowledge that survives the trade has a shape. These models are generalists: they know a little about nearly everything and almost nothing in depth. Ask one about PostgreSQL and it knows what it is, what it's good at, and roughly how MVCC works. Ask which version added a specific planner feature and you're back to invented facts.
That's the right layer to keep in weights, because breadth is what lets a model understand what a question is about, know what to look up, and judge whether a source is plausible. The depth is cheap to retrieve and expensive to store, so it's the part that goes.
Facts Rot. Procedures Don't.
A frontier training run takes months and costs hundreds of millions of dollars. The moment it finishes, the facts inside it start going stale. Library APIs change, prices change, people change jobs, and half of what a 2024 model believed about the JavaScript ecosystem was outdated before the model shipped.
Every fact you bake into weights has a shelf life, and the only way to refresh it is another training run.
The procedures don't rot. Algebra worked the same way in 1970 as it does now. So does breaking a problem down, spotting a contradiction between two sources, or checking whether an answer is in the right ballpark. A model that's mostly procedure and only lightly loaded with facts doesn't age the way a knowledge-heavy model does. Its training cutoff matters much less, because the current state of world was never stored in the weights in the first place.
This is why the small models can be so good at math. Math doesn't change. The procedures for solving it are timeless. But the same model that aces AIME will confidently tell you the wrong capital of a country it "knows" about.
What This Means for Developers
If you're building on LLMs, you need to understand the trade:
Don't rely on model weights for facts. The trend is clear — models are being optimized for reasoning, not recall. If your application depends on factual accuracy, you need retrieval (RAG, search, databases), not a bigger model. A bigger model will reason better about the facts you give it, but it won't know more facts.
Reasoning is the moat. The models that win are the ones that can take whatever facts you give them and do something intelligent with them. Focus your prompts and tooling on structuring the reasoning, not on teaching the model facts.
Small models are production-ready — for the right tasks. A 9B model that fits in 6GB of VRAM can do math, write code, and structure data. It cannot answer trivia questions. Choose your model size based on what it needs to do, not on a vague notion of "smarter is better."
Hallucination isn't a bug — it's the design. When you strip facts from weights, the model doesn't say "I don't know." It generates plausible-sounding text. This is a feature for reasoning tasks (you want the model to generate, not refuse) and a liability for factual tasks. Guard against it with retrieval and verification, not with bigger models.
The training cutoff matters less than you think. A procedure-heavy model is less dependent on knowing current facts. What it needs is access to current facts through tools — search, APIs, databases. Build for that.
The Uncomfortable Truth
We've been measuring AI progress with the wrong yardstick. Benchmarks that test reasoning show explosive progress. Benchmarks that test knowledge show stagnation or decline. The industry has chosen a direction: models will get better at thinking and worse at knowing.
This isn't necessarily bad. A world where models are excellent reasoners that look up facts on demand is arguably better than one where models are encyclopedias that can't think. But it changes what we need to build. The application layer — retrieval, verification, grounding — becomes more important, not less.
The models aren't getting dumber. They're getting more specialized. And the specialization is away from the thing most people still assume they're buying: a know-it-all. What you're actually getting is a think-it-all, and that's a different — and in many ways better — tool. But only if you build for what it is, not for what it used to be.
Top comments (0)