DEV Community

Cover image for The Billion-Dollar Model That Can't Count: Why Scaling Laws Hit a Wall
VelocityAI
VelocityAI

Posted on

The Billion-Dollar Model That Can't Count: Why Scaling Laws Hit a Wall

GPT-4 can pass the bar exam, score in the 90th percentile on the SAT, and write a coherent essay on the themes of Moby-Dick. It also, until recently, struggled to reliably count the number of r’s in “strawberry.” A model trained on trillions of tokens, at a cost estimated in the hundreds of millions of dollars, failed at a task that a seven-year-old can do. The r’s in strawberry became a meme, and the meme obscured something important: it was not an isolated glitch. It was a symptom of a structural limit that the scaling paradigm has been papering over for years.

Here’s the claim I want to test: the scaling laws are real, but they are laws about a specific kind of capability, and they do not extrapolate to the capabilities that matter most. Bigger models get better at the things that benchmarks measure. They do not reliably get better at the things that benchmarks miss, and the things that benchmarks miss are increasingly the things that determine whether a model is useful.

The field is now at the point where the returns to scale are real but diminishing, and the diminishing returns are concentrated exactly where the hype is loudest. That’s the wall. It’s not a hard limit. It’s a soft limit that the industry is pretending doesn’t exist.

What the Scaling Laws Actually Say

Let me be precise about the claims, because the discourse is sloppy.

The original scaling laws (Kaplan et al. 2020) showed that loss on a next-token prediction objective decreases as a power law with model size, dataset size, and compute. The laws were remarkably clean, and they held across several orders of magnitude. They were the empirical foundation for the “just add more compute” paradigm.

The Chinchilla paper (Hoffmann et al. 2022) refined the laws, showing that for a fixed compute budget, the optimal allocation is roughly 20 tokens per parameter. A 70B model should be trained on about 1.4T tokens. The refinement was important, but it didn’t change the fundamental picture: bigger models, more data, better loss.

Here’s the part that gets glossed over. Loss on next-token prediction is not the same as capability. The scaling laws describe a smooth, predictable improvement in a proxy metric. They do not describe a smooth, predictable improvement in the things we actually care about: reasoning, factual accuracy, instruction following, safety, and robustness.

Those capabilities do improve with scale, on average. But the improvement is not smooth, not predictable, and not uniform. It comes in jumps, plateaus, and sometimes regressions. The scaling laws are a description of the average, and the average is not the thing that matters.

The Sacred Cow: “Emergent Abilities Prove Scaling Works”

Here’s the belief I want to dismantle, and it’s the most cited evidence for the scaling paradigm.

The claim goes like this: emergent abilities. At certain scales, models suddenly develop capabilities they didn’t have before. In-context learning, chain-of-thought reasoning, multi-step arithmetic. These abilities appear abruptly, and their appearance is evidence that scaling unlocks qualitatively new capabilities.

This is the strongest argument for the scaling paradigm, and it’s been quietly falling apart for two years.

The problem is that “emergence” is often an artifact of the metric, not a property of the model. A 2023 paper by Schaeffer et al. showed that many emergent abilities disappear when you use a continuous metric instead of a discontinuous one. The model’s performance on a task improves smoothly with scale; the metric (e.g., exact match on a multi-step problem) is discontinuous, so the improvement looks like a jump. The jump is in the metric, not in the model.

This doesn’t mean emergence is fake. Some abilities do appear to emerge, and the mechanism is not fully understood. But the “emergence” framing has been used to justify a lot of claims that don’t hold up under scrutiny. The claims are that scaling unlocks qualitatively new capabilities, and the evidence is often a metric artifact.

The contrarian take: the emergent abilities narrative is the most successful marketing campaign in the history of AI research. It takes a real phenomenon (some abilities do improve with scale) and inflates it into a story (scaling unlocks everything). The story is compelling, it’s fundable, and it’s mostly wrong.

The correct framing is that scaling improves capabilities on a smooth curve, and the curve is steep in some regions and flat in others. The flat regions are where the wall is. The wall is real, and the emergence narrative has been hiding it.

Where the Wall Actually Is

Let me get concrete about the wall, because “scaling hits a wall” is too vague to be useful.

The wall is not uniform. It’s different for different capabilities, and the differences are informative.

  1. The wall is low for knowledge retrieval.

Bigger models know more facts. A 70B model knows more than a 7B model, and a 700B model knows more than a 70B model. The curve is steep and smooth, and it shows no sign of flattening. If knowledge retrieval is the capability you care about, scaling works.

  1. The wall is medium for instruction following.

Bigger models follow instructions better, on average. But the improvement is not smooth. It comes in jumps, and the jumps are often driven by fine-tuning rather than scale. A well-tuned 7B model can outperform a poorly-tuned 70B model on instruction following. The curve is steep at small scales and flattens at large scales.

  1. The wall is high for reasoning.

Bigger models reason better on some tasks and not at all on others. Multi-step arithmetic, logical deduction, and planning are the hard cases. The improvement with scale is real but slow, and it’s not obvious that more scale is the right lever. A 700B model can still fail at a reasoning task that a 7B model with a tool can solve. The curve is shallow, and it shows signs of flattening.

  1. The wall is very high for counting and precise manipulation.

Counting, precise arithmetic, and manipulation of exact quantities are the hardest cases. The r’s in strawberry is the canonical example. The failure is not about knowledge; it’s about the architecture. Transformers don’t have a natural mechanism for counting, and scale doesn’t give them one. The curve is nearly flat, and it doesn’t show signs of improving.

The pattern: the wall is low where the capability is about knowledge, medium where it’s about pattern matching, and high where it’s about precise manipulation of symbolic structure. The industry has been scaling as if all capabilities are on the same curve. They’re not. The curves are different, and the differences are the thing to pay attention to.

Why Counting Is Hard (and What It Reveals)

Let me be specific about the counting problem, because it’s the clearest example of the wall.

A transformer processes a sequence of tokens through a series of attention layers and feed-forward networks. The attention mechanism allows each token to attend to every other token, which is powerful for pattern matching. But the attention mechanism is not a counter. It doesn’t have a natural way to increment a register, or to maintain a running total, or to compare quantities exactly.

Counting requires a specific kind of computation: a sequential scan with a state that persists across tokens. Transformers don’t have this natively. They can learn to approximate it for short sequences, but the approximation breaks down as the sequence gets longer. The breakdown is not a matter of scale. It’s a matter of architecture.

The counting problem generalizes. Any task that requires precise manipulation of symbolic structure, without pattern matching, is hard for transformers. Arithmetic, logic, planning, and exact comparison are all in this category. Scale improves the approximation, but the approximation is still an approximation, and it has a ceiling.

The fix is not more scale. The fix is architecture. Tool use, external memory, and hybrid models are the ways to get the precise manipulation that transformers can’t do natively. The models that use these techniques are the ones that can count reliably, and they can count reliably at any scale.

The counting problem is a signal that the scaling paradigm has a ceiling. The ceiling is not the end of AI. It’s the end of the assumption that scale is the only lever.

What the Wall Means for Builders

The wall is not a reason to stop building. It’s a reason to build differently.

  1. Stop assuming scale is the answer.

If you’re building a product that depends on a specific capability, test whether scale actually improves that capability. Not the benchmark, the capability. You will often find that a smaller model with the right architecture or the right tool outperforms a larger model without them.

  1. Use tools for the tasks that transformers can’t do.

Counting, arithmetic, logic, and planning are the tasks where tools matter most. A model that calls a calculator is more reliable than a model that does arithmetic in its weights. A model that calls a planner is more reliable than a model that plans in its weights. The tool use is not a workaround. It’s the correct architecture.

A rough sketch:def reliable_arithmetic(model, problem):

Let the model parse the problem

parsed = model.parse(problem)

Let a tool do the arithmetic

result = calculator.evaluate(parsed.expression)

Let the model interpret the result

return model.interpret(result, parsed.context)

The point is not the specific implementation. The point is the division of labor: the model does the parts it’s good at, the tool does the parts it’s not.

  1. Measure the capabilities that matter, not just the benchmarks.

If you’re evaluating models, measure the capabilities that matter for your product. Not just the benchmark scores, but the specific tasks that your users care about. The benchmark scores are a proxy for the average capability. Your product cares about the specific capability, and the specific capability may be on a very different curve.

What To Actually Do

Three moves, concrete enough to start on Monday.

  1. Test the wall in your own domain.

Take the capability you care about most and test it across model sizes. Not the benchmark, the capability. Measure the scaling curve in your domain. You will find that some capabilities scale smoothly and some hit a wall, and the ones that hit a wall are the ones where you need a different architecture.

  1. Build tool use into your pipeline.

If your product depends on counting, arithmetic, or logic, build tool use into the pipeline. Don’t rely on the model to do these tasks natively. The tools are the reliable path, and the reliable path is the one that doesn’t depend on scale.

  1. Stop waiting for scale to solve the problem.

If you’re waiting for the next generation of models to solve your problem, you may be waiting a long time. The next generation will be bigger and better on the benchmarks, and it may not be better on the capability you care about. The capability you care about is the thing to solve with architecture, not scale.

The Mental Model

Here’s the lens I want you to keep.

Scaling laws are real, but they are laws about a proxy metric, not about capability. The proxy metric improves smoothly with scale. The capability improves unevenly, and the unevenness is where the wall is. The wall is low for knowledge, medium for pattern matching, and high for precise manipulation. The high wall is not a hard limit. It’s a signal that architecture matters, and architecture is the lever that scale can’t replace.

The interesting question is not whether scaling is hitting a wall. It is hitting a wall, and the wall is visible in the counting problem, the arithmetic problem, and the reasoning problem. The interesting question is whether the industry will build the architectures that get past the wall, or whether it will keep scaling and hope the wall moves. The wall will move a little. It won’t move as much as the industry needs.

The billion-dollar model that can’t count is not a failure of scale. It’s a success of scale, on a capability that doesn’t require counting. The failure is in the assumption that scale is the only lever, and the failure is being exposed by the tasks that scale can’t reach.

What’s your read? If you’ve tested scaling in your own domain, where did the wall show up? And if you’re building products, what’s the capability that scale can’t solve for you? Drop it in the comments. I’m genuinely curious where the wall is in practice, because the benchmarks are hiding it.

Top comments (0)