Everyone thinks AI progress is just a bigger GPU every year.
More VRAM. More cores. More watts. More "just scale it."
NVIDIA drops a new card and the entire ecosystem nods in reverence like it's a firmware update from God.
And listen — NVIDIA isn't going anywhere.
It's the cockroach of compute. Gaming winter? It survives. Crypto collapse? It pivots. AI boom? It dominates. Data center wars? It adapts.
You don't bet against NVIDIA.
But here's the thing:
The ecosystem's heavy reliance on giant GPUs might not survive unchanged.
And that's not anti-GPU. It's anti-laziness.
Abundance Makes Systems Lazy
We've seen this before, and the pattern is always the same.
Android spent a decade solving software problems by adding RAM. Apps bloated. Frameworks layered abstraction on abstraction. Memory usage exploded. Performance didn't collapse — because hardware kept increasing, so nobody cared. By 2018, a simple messaging app was consuming north of 500MB of RAM. Nobody optimized. Nobody had to.
AI is doing the same thing right now, just at a grander scale and with better PR.
Instead of asking "how do we make this smarter?", the industry asks "how many GPUs can we throw at it?" GPT-4 reportedly ran on tens of thousands of A100s. The answer to every benchmark that underperformed was another rack of compute. That works — until the economics catch up with the ambition.
The Scaling Mirage
Right now, scaling laws still hold. More parameters, more tokens, more compute, more GPUs — the curve bends upward and everyone cheers.
But here's what the benchmarks don't show: a single training run for a frontier model now costs north of $100 million. Microsoft signed a $10 billion deal with OpenAI and the models still hallucinate basic facts. The marginal return on each additional GPU is shrinking, even if the absolute numbers keep climbing. Power grids are straining — Virginia data center demand alone has triggered capacity warnings from Dominion Energy. Cooling is becoming a geopolitical problem. The energy cost per token is starting to show up in earnings calls.
Abundance isn't infinite. It's subsidized by venture capital and electricity infrastructure that was never designed for this. And subsidies expire.
GPUs Won by Convenience, Not Destiny
This is the part nobody wants to say out loud.
GPUs were the right answer for the last decade not because they were architecturally superior for AI — but because they already existed. CUDA already existed. Matrix multiplications happened to map reasonably well onto graphics pipelines. The ecosystem matured around them through sheer momentum, not design.
TPUs told a different story early on. Compiler-first thinking. Dataflow architecture. Hardware-software co-design from the ground up. Google's internal benchmarks showed TPUs delivering 15-30x better performance per watt on inference workloads compared to GPUs as far back as 2017. That paper sat quietly in the industry for years while everyone kept buying A100s.
Specialization matters. It always has. The GPU won the last era by accident of timing. That doesn't make it the permanent answer.
CPUs Aren't Dead. They're Mutating.
Here's the thing nobody wants to admit — CPUs didn't lose the AI race. They never entered it. The industry declared them obsolete before the competition started, because CUDA existed and inertia is powerful and nobody wanted to rewrite the stack.
But quietly, without a press release, CPUs mutated.
AVX-512. AMX matrix units. Sapphire Rapids shipping with dedicated AI acceleration baked directly onto the die. AWS Graviton running serious inference workloads at a fraction of the energy bill. ARM servers going from punchline to legitimate infrastructure in five years.
That's not a dead architecture. That's an architecture that got told to sit in the corner and used the time to do pushups.
The more interesting shift is structural. The GPU model assumes centralization — one massive chip doing everything, all memory local, all compute colocated. CPUs evolved toward something different. Disaggregated memory. Shared compute fabrics. Many modest nodes cooperating across a network instead of one behemoth doing it alone.
That changes what good algorithms look like. An algorithm written to saturate a single H100 is a liability the moment compute gets distributed. An algorithm written for coordination across modest nodes — that ages differently.
CPUs aren't coming back to dominate training. That ship sailed.
But inference is a different question. And inference is where AI actually lives, at scale, in production, in the real economy.
That's where the CPU's mutation starts mattering. And most people aren't watching it.
The Android Moment Nobody Is Talking About
Here's the concrete example that should make you uncomfortable if you're deep in the GPU monoculture.
In 2012, Android was bloated, slow, and RAM-hungry. Then the Moto G launched at $179 with modest specs, sold 10 million units in a year, and forced the entire Android ecosystem to actually optimize its software stack. Constraints created by a cheaper, lower-power device did more for Android performance than three generations of premium hardware had.
AI hasn't had its Moto G moment yet. But it's coming. It might look like edge inference on devices with 4GB of RAM. It might look like on-device LLMs that have to run without a data center behind them. It might look like a jurisdiction that bans cloud AI processing and forces local computation. Whatever the trigger, the optimization pressure will arrive — and the systems built entirely around abundance will not be ready for it.
Performance Smart vs. Performance Big
There is a real difference between performance because you have more, and performance because you waste less.
When compute was constrained in the early days of deep learning, researchers were forced to be clever. Dropout was born from resource scarcity. Batch normalization emerged from the need to train faster with less. Attention mechanisms were partly an efficiency innovation before they became the architecture that ate the world. Scarcity made people think. The researchers working under those constraints were doing the harder intellectual work.
Right now, the dominant research strategy is: scale it and see what happens. That's not a criticism of the researchers — it's a rational response to the incentives abundance creates. But it means whole categories of algorithmic innovation are being skipped because brute force is cheaper than cleverness, for now.
When compute is constrained, allocation matters. Early stopping matters. Uncertainty modeling matters. Every forward pass has to justify itself. That's when systems sweat — and sweating is when you find out what the architecture is actually made of.
The Real Future Question
The future of AI might not be decided by who owns the biggest GPU cluster. It might be decided by who extracts the most intelligence per joule. Per watt. Per millisecond. Per dollar of inference cost.
If hardware scaling slows — even slightly — algorithms that depend purely on brute force will plateau hard. Algorithms that exploit structure, that are hardware-aware by design, that treat energy as a first-class constraint — those will compound. The efficiency gap will start to look like an intelligence gap, because in deployment terms, it is one.
This Isn't Anti-GPU
NVIDIA will survive. GPUs will evolve. AI will still run on them for a long time.
This is not "GPUs die." This is: monoculture reliance creates fragility. If everything depends on scaling compute, then everything depends on infinite abundance — infinite cheap electricity, infinite cooling capacity, infinite capital willing to fund $100M training runs with negative unit economics.
Infinite abundance has historically been a bad bet. The companies that thrived through the PC transition weren't the ones with the most hardware. They were the ones that understood what happened when the hardware became ordinary.
The Real Arms Race
It's not a GPU arms race. It's a laziness arms race — and right now we're winning by being louder and bigger.
The next decade will likely reward efficiency, architectural structure, allocation intelligence, and hardware-aware design. Not raw scaling. The researchers and companies building systems that work under constraint — that sweat — are doing the work that will matter when the subsidy ends.
And honestly, I'm more interested in building systems that sweat than systems that just scale. Because sweating means you understand the problem. Scaling just means you can afford to avoid understanding it, for now.
Here are some papers worth reading :
[1] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., ... & Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
[2] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., ... & Sifre, L. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. https://arxiv.org/abs/2203.15556
[3] Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., ... & Yoon, D. H. (2017). In-Datacenter Performance Analysis of a Tensor Processing Unit. arXiv:1704.04760. https://arxiv.org/abs/1704.04760
[4] Hernandez, D., & Brown, T. B. (2020). Measuring the Algorithmic Efficiency of Neural Networks. OpenAI. https://cdn.openai.com/papers/ai_and_efficiency.pdf
[5] Shen, H., Chang, H., Dong, B., Luo, Y., & Meng, H. (2023). Efficient LLM Inference on CPUs. NeurIPS 2023. arXiv:2311.00502. https://arxiv.org/abs/2311.00502
Top comments (4)
A very honest take on the GPU monoculture I've read... Moto G analogy alone is worth the whole piece
well said i have been following this guy from quite long , and all this guy does is shit talking