Why GPUs Beat CPUs for AI Training (and Why You Can't Just Build a Bigger CPU)

A question came up in a recent interview at Intel that I keep turning over: why is a GPU actually better than a CPU for AI training, and why can't you just solve the problem by making the CPU bigger? It's a deceptively simple question. The honest answer isn't "GPUs are faster," it's that CPUs and GPUs are solving different problems by design, and AI training happens to be exactly the kind of problem a CPU's design actively works against.
The shape of the workload matters more than the chip
Training a neural network is, at its core, repeated matrix multiplication. Millions or billions of independent multiply-accumulate operations, applied to tensors, over and over, for every layer, every batch, every step. The defining property of that workload is that almost none of those operations depend on each other. Multiplying element 47 of a weight matrix doesn't need to know what happened to element 48. That's what people mean when they call it "embarrassingly parallel."
A CPU core is built for the opposite kind of problem. Most of a CPU core's silicon goes toward figuring out what to do next when the next instruction is unpredictable: branch prediction, out-of-order execution, speculative execution, deep multi-level caches, all of it exists to keep one core fed and busy when the code in front of it is full of conditionals and data-dependent branches. That's exactly what you want for parsing a web request or querying a database. It's almost entirely wasted on a workload where the instruction is "multiply these two numbers" repeated a billion times with no branching at all.
A GPU throws most of that machinery away. Instead of a handful of cores that are individually very smart, it packs thousands of simple cores that share control logic across groups and execute the same instruction in lockstep across many threads at once (NVIDIA calls this SIMT, single instruction, multiple threads). Each individual GPU core is far less capable than a CPU core. But the workload doesn't need capable, it needs many.
The numbers, as of right now
Intel's newest server chip, Xeon 6+ ("Clearwater Forest," launched mid-2026), tops out at 288 cores per socket, 576 in a dual-socket box. That's an enormous number for a CPU and represents real engineering work to pack that many general-purpose cores into one die.
NVIDIA's current flagship training and inference GPU, the Blackwell Ultra B300, has 160 streaming multiprocessors, each containing 128 CUDA cores, for 20,480 CUDA cores on a single chip. That's about seventy times the core count of the biggest CPU Intel currently ships, on a single accelerator.
Power tells the same story from a different angle. The top Xeon 6+ part draws up to 450W across 288 cores, around 1.5W per core. The B300 draws up to 1,400W across 20,480 cores, around 0.07W per core. Each individual CPU core costs roughly twenty times more power than each GPU core, because the CPU core is spending that power budget on branch prediction and speculative execution that a matrix multiply will never use.
Memory bandwidth is the other half of the answer
Compute alone doesn't tell the full story. Every one of those cores needs data to chew on, and that's where the gap widens even further. Xeon 6+ supports twelve channels of DDR5-8000 memory, which works out to roughly 770 GB/s of aggregate memory bandwidth per socket. The B300 uses HBM3e, high bandwidth memory stacked directly on the GPU package, and delivers 8 TB/s, more than ten times the CPU's bandwidth.
That gap matters because thousands of parallel cores are useless if they're sitting idle waiting on data. GPUs pair massive parallelism in compute with massive parallelism in memory access specifically so the two scale together. A CPU optimized for low-latency access to a small working set, which is the right design for general-purpose software, is the wrong design when you need to stream enormous tensors through thousands of execution units continuously.
Tensor cores: purpose-built silicon for the exact operation that matters
Beyond the general-purpose CUDA cores, the B300 has dedicated Tensor Cores, four per streaming multiprocessor, fifth generation, purpose-built to execute matrix multiply-accumulate operations directly in hardware rather than through general-purpose instructions. A CPU has to express the same matrix math through vector instruction extensions like AVX-512 or Intel's AMX, which help, but are still general-purpose vector units doing their best impression of a matrix engine. A Tensor Core is not doing an impression. It is silicon built for exactly one job, and it does that job at a fraction of the energy and time cost.
So why not just make the CPU bigger?
This is the part of the question that actually matters. If parallelism is what wins, why not put 20,000 cores on a CPU instead of a GPU?
The honest answer is that you'd be removing the parts that make it a CPU. Every CPU core's value comes from being good at handling unpredictable, branchy, latency-sensitive work, and that capability has a fixed silicon and power cost per core that doesn't shrink no matter how many cores you add. Stack 20,000 of those cores on one die and you hit a power wall and a heat wall long before you hit a useful core count. Clearwater Forest already needs 450W and serious cooling to reach 288 cores. Scaling that same design to GPU-level core counts isn't an engineering inconvenience, it's a different chip with different goals, which is precisely what a GPU already is.
There's also a parallel efficiency mismatch. Even if you somehow fit 20,000 full CPU cores on a die, AI training doesn't need 20,000 cores that are each capable of branch prediction and out-of-order execution. It needs 20,000 cores that can each multiply and add, fast, with shared control logic across groups of them so you're not paying the "smart core" tax 20,000 times over for a job that never uses that intelligence. That's the actual design insight behind a GPU: not "more cores," but "cheaper cores, because the workload doesn't need expensive ones."
A useful way to picture it: a CPU core is a Formula 1 car, exceptional at navigating unpredictable terrain, expensive, and there's a hard limit on how many you can build and fuel. A GPU core is a forklift. Less sophisticated individually, but if the job is moving an identical pallet ten thousand times, a fleet of forklifts beats one very fast car, and you can afford the fleet precisely because each forklift is cheap.
Even Intel agrees, and that's the real answer
Here's what makes this question interesting to ask at Intel specifically: Intel is the company that has spent fifty years perfecting the general-purpose CPU, and even Intel's own roadmap concedes this point. Alongside Xeon 6+, Intel is shipping its own data center GPU line, codenamed Crescent Island, built specifically for AI workloads, with its own memory subsystem optimized for AI inference and training rather than general compute. Intel isn't trying to make Xeon win the AI training race by adding more cores. Intel's own public position is that the CPU's role in AI infrastructure is to be the control plane and handle data preprocessing and orchestration, while purpose-built accelerator silicon, GPUs, in Intel's case increasingly its own, handles the actual tensor math.
That's the answer that actually lands in an interview, I think. It's not "GPUs are better, full stop." It's that modern AI infrastructure is heterogeneous by design: a CPU that's excellent at the unpredictable, control-heavy parts of the job, paired with an accelerator that's excellent at the predictable, massively parallel parts of the job. The interesting systems engineering question isn't which one wins, it's how you split the work between them, and that's a question that shows up constantly in the kind of infrastructure I want to be building.