How AI uses graphics processing comes down to one brutal bottleneck: math. Specifically, massive amounts of matrix multiplication happening millions of times per second. Training a large AI model like GPT-4 requires performing quintillions of floating-point operations — and doing them fast enough to be useful. According to Stanford University's Human-Centered AI group, GPU performance has increased roughly 7,000 times since 2003, with price-per-performance improving 5,600-fold over the same period. That explosion didn't happen because someone wanted better video games. It happened because the architecture that makes GPUs brilliant at rendering pixels turns out to be almost perfectly suited to the core mathematical operations that power modern AI.
What does a GPU actually do differently?
Most people think of a GPU as the thing that makes games look good. That's true — but the reason it makes games look good is the reason AI needs it so desperately.
A Central Processing Unit (CPU) is built for sequential processing. It has a small number of extremely powerful cores — typically between 8 and 64 in a high-end consumer chip — each capable of handling complex, varied tasks one after another. CPUs are generalists. They're brilliant at branching logic, running operating systems, and handling unpredictable workloads. What they're not built for is doing the same simple operation simultaneously across thousands of data points.
A GPU does exactly that. Where a top-of-the-line CPU might have 64 cores, a modern GPU like NVIDIA's H100 — the chip powering most serious AI infrastructure today — contains over 16,000 CUDA cores. These cores are simpler and slower individually, but they work in parallel. Ask a GPU to multiply a million pairs of numbers, and it doesn't queue them up — it handles enormous batches at once.
Think of it this way. A CPU is a team of 10 brilliant surgeons. A GPU is a factory floor of 10,000 workers doing the same precise assembly step simultaneously. For the right job, the factory wins every time.
The right job, it turns out, is almost all of AI. Neural networks are built on layers of matrix operations — multiplying grids of numbers together over and over, adjusting weights, passing results forward. This is exactly the kind of repetitive, parallel numerical work GPUs were designed to handle. Not metaphorically. Almost literally.
Why does training an AI model need this much power?
Training an AI model isn't like running a program. It's closer to running a program billions of times while constantly adjusting every variable inside it.
When a model like ChatGPT is trained, it starts with essentially random internal settings — called weights and parameters. The training process feeds it examples, measures how wrong its output is, then nudges every parameter slightly in the direction that reduces that error. This process, called backpropagation, requires calculating gradients across the entire network for every single training example. GPT-3 has 175 billion parameters. GPT-4's parameter count hasn't been officially confirmed, but estimates put it considerably higher.
Each pass through the network — called a forward pass — involves multiplying enormous matrices together across hundreds of layers. The subsequent backward pass, which computes the error gradients, requires doing it all again in reverse. Then the whole cycle repeats for the next training sample. Then the next. For weeks or months.
The numbers are staggering. Training a large language model can require performing on the order of 10²³ floating-point operations — a number with 23 zeros. No CPU cluster could complete this in any practical timeframe. With thousands of GPUs running in parallel, it becomes merely expensive.
This is why the biggest AI labs — Google DeepMind, OpenAI, Meta AI — operate data centres packed with tens of thousands of GPU chips running continuously. The GPUs aren't doing anything exotic. They're doing arithmetic. Just an almost incomprehensible amount of it, faster than any other hardware can.
What happens after training — does AI still need GPUs?
Training gets most of the attention because it's so resource-intensive. But there's a second phase that matters just as much: inference — the moment the AI is actually responding to you.
Every time you send a message to an AI chatbot, ask an image generator for a picture, or get a recommendation from a streaming platform, the model is performing inference. It's running your input through its learned weights to produce an output. This is computationally lighter than training, but it's happening millions of times per second across millions of simultaneous users.
For inference, the calculation profile is slightly different. The model isn't learning — it's just running forward through its network once per request. But the scale of demand means GPU efficiency becomes critical. A single inference might take microseconds on a dedicated chip, but serving millions of users simultaneously requires parallelising those requests across GPU clusters.
Companies like NVIDIA have responded by designing chips specifically optimised for inference workloads — lower precision arithmetic, faster memory bandwidth, smarter caching. The H100 chip, for instance, uses a new generation of Tensor Cores specifically engineered for the matrix operations that dominate both training and inference in transformer-based models.
There's also a growing area of research around model quantisation — compressing AI models so they can run on less powerful hardware, or even consumer-grade GPUs. This is what allows AI features to appear on laptops and smartphones. The core GPU dependency doesn't disappear; it just shifts down the hardware ladder.
Can't regular computers just run AI?
This is the question most people arrive at eventually — and the answer is nuanced. It depends entirely on the scale of the task.
Your laptop's processor can run small AI models. Consumer applications using compressed versions of language models — like some local AI tools — can function on a standard CPU, though slowly. What cannot run on a standard CPU is the training of a large model, or the real-time inference of a frontier-scale model serving millions of users.
Consider a rough comparison. A high-end consumer CPU might deliver around 1–2 teraflops of floating-point performance relevant to AI workloads. NVIDIA's H100 delivers around 2,000 teraflops for the specific low-precision operations AI relies on most. That's a difference of roughly three orders of magnitude — not a modest gap, but a chasm.
Some companies are building alternative chips to challenge GPU dominance:
- Google's TPUs (Tensor Processing Units) — custom silicon designed specifically for neural network matrix operations, used internally across Google's AI products
- Cerebras Systems' Wafer-Scale Engine — a chip the size of a dinner plate containing more transistors than any GPU
- Apple's Neural Engine — integrated into M-series chips, enabling on-device AI inference for consumer features like Siri and image recognition
But GPUs remain dominant for one practical reason: the software ecosystem. NVIDIA's CUDA programming platform — launched in 2006 — gave researchers a way to write GPU code for non-graphics tasks. Decades of AI libraries, frameworks, and tooling were built on top of it. TensorFlow, PyTorch, and virtually every major AI framework are GPU-first by default. Switching to a different chip means rewriting enormous amounts of infrastructure. That's not impossible, but it's expensive and slow.
Why does any of this actually matter to you?
The GPU dependency at the heart of modern AI has consequences that ripple out far beyond data centres and research labs.
First, there's the economics of AI access. Because GPUs are scarce, expensive, and power-hungry, serious AI capability is concentrated in the hands of a small number of companies with the capital to build or rent GPU infrastructure. Training a frontier model can cost tens of millions of dollars in compute alone. That's not a regulatory barrier or a knowledge barrier — it's a hardware barrier. It shapes which organisations can compete in AI development.
Second, there's the supply chain reality. The vast majority of the world's most advanced AI-grade GPUs are manufactured by TSMC in Taiwan, using designs from NVIDIA and AMD. This concentration in a single geographic location creates geopolitical risks that governments are now spending hundreds of billions of dollars to address — including the US CHIPS Act and similar legislation in Europe and Japan.
Third, there's the energy consequence. GPUs running 24 hours a day consume enormous amounts of electricity. A single NVIDIA H100 GPU draws around 700 watts of power. A data centre with 10,000 of them draws 7 megawatts — enough to power several thousand homes. As AI workloads multiply, this demand is growing faster than the grid can easily accommodate.
Understanding that AI runs on GPUs isn't a technical footnote. It explains why AI development is expensive, why certain companies dominate it, and why the environmental and geopolitical stakes of this technology are so high. The intelligence is in the model. But the model runs on hardware. And right now, that hardware is a GPU.
The story of AI is usually told in terms of data, algorithms, and intelligence. But underneath all of that is silicon. Specifically, thousands of parallel processors doing matrix multiplication at inhuman speed. GPUs weren't designed for AI — they were designed for pixels. The fact that the two tasks share the same mathematical DNA is one of the luckier accidents in the history of computing. Next time you see an AI generate an image or finish your sentence, what you're actually witnessing is a graphics chip doing what it was always built to do — just pointed at a very different problem.
Originally published on SnackIQ
Top comments (0)