DEV Community

Cover image for Why AI Training Actually Eats Your Grid
SnackIQ
SnackIQ

Posted on Originally published at snackiq.app

Why AI Training Actually Eats Your Grid

How much electricity does AI training use? Enough to make power grid engineers nervous. Training GPT-3, OpenAI's 2020 language model, consumed an estimated 1,287 megawatt-hours of electricity, roughly what 120 average American homes use in a year, according to researchers at the University of Massachusetts Amherst. And that was five years ago, before the models got dramatically larger. Today's frontier models require orders of magnitude more. The International Energy Agency projected in 2024 that global data centre electricity demand could double by 2026, with AI workloads driving a significant portion of that growth. This isn't a side effect of AI. It's physics. Understanding why reveals something fundamental about how machine learning actually works.

What actually happens during AI training?

Most people picture AI training as something like a student reading textbooks. It's not. It's closer to a student rewriting the same exam paper billions of times, adjusting one tiny mark each pass, until the answers start improving.

Every AI model, whether it's an image recogniser or a language model, is built from a neural network: a vast web of numerical parameters called weights. At the start of training, these weights are essentially random. The model makes terrible predictions. Then the training process kicks in.

Here's the cycle that repeats billions of times. The model sees a piece of data, a sentence, an image, a chunk of text. It makes a prediction. The prediction is compared to the correct answer. The difference (the error) is calculated. Then an algorithm called backpropagation works backwards through the network, figuring out which weights contributed most to the error. Each weight gets nudged, slightly, in a direction that would reduce that error next time. Then the whole thing starts again.

For a model like GPT-4, which reportedly has hundreds of billions of parameters, that adjustment process runs across an enormous number of weights, for every single example in the training dataset, across potentially trillions of tokens of text. Each individual mathematical operation is simple. Multiply two numbers. Add them. Apply a function. But do that quadrillions of times, and you need serious hardware running flat-out for months.

This is why training is categorically more power-hungry than simply using an AI. Inference, asking the model a question, is expensive too, but training is building the entire brain from scratch, not just consulting it.

Why do the chips themselves burn so much power?

The hardware doing AI training isn't your laptop processor. It's specialised chips called GPUs (Graphics Processing Units) and, increasingly, purpose-built AI accelerators like NVIDIA's H100 or Google's Tensor Processing Units (TPUs).

GPUs were originally designed for rendering video game graphics, a task that requires doing thousands of simple calculations simultaneously rather than a few complex ones sequentially. That same architecture turns out to be ideal for the matrix multiplications at the heart of neural network training. A single NVIDIA H100 GPU has a peak power draw of around 700 watts, roughly the same as a powerful microwave oven. But you don't train frontier models on one chip.

Meta's Llama 3 training run reportedly used over 16,000 H100 GPUs. Running 16,000 chips at roughly 700 watts each means a peak demand approaching 11 megawatts for the chips alone, before you account for the cooling systems, networking hardware, storage, and facility infrastructure that keep everything running. Data centres typically consume as much energy on cooling as they do on computation, which doubles the effective electricity bill.

The measure engineers use for this overhead is called PUE: Power Usage Effectiveness. A PUE of 1.0 would mean every watt goes to computing. Real data centres typically run between 1.2 and 1.5, meaning for every watt of compute, another 0.2 to 0.5 watts goes to keeping the machines from melting.

Scale this up across dozens of simultaneous training runs at major AI labs, and the numbers become genuinely difficult to comprehend.

How does training compare to just using AI?

There's a critical distinction the headlines often blur: training and inference are two completely different electricity problems.

Training is the one-time (or periodic) process of building the model from scratch. Inference is what happens when you type a question into ChatGPT and get an answer back. Training is like constructing a factory. Inference is running the factory every day.

A single training run for a frontier model might consume thousands of megawatt-hours over several months. A single inference, one ChatGPT response, consumes a fraction of a watt-hour. But ChatGPT reportedly handles over 100 million users. Do the maths across billions of daily queries and inference starts adding up too, potentially rivalling training in total annual consumption for widely-deployed models.

Researchers have pointed out an important nuance: the electricity costs aren't evenly distributed across time. Training hits the grid in concentrated bursts, massive sustained loads during a model's training run. Inference is more spread out, more predictable, but relentless. AI labs managing this at scale are essentially running small power utilities.

  • Training: weeks to months, thousands of GPUs, concentrated grid impact
  • Fine-tuning: shorter runs adapting existing models, moderate power demand
  • Inference: continuous, billions of queries, cumulative impact rivals training over time

The IEA estimated that a ChatGPT query uses roughly ten times as much electricity as a standard Google search, a figure that illustrates just how energy-intensive generative AI inference is compared to conventional search.

Are bigger models always more power-hungry?

Here's where it gets counterintuitive. Bigger models don't always mean proportionally more power, and smaller models aren't always cheaper to run.

The relationship between model size and energy cost is governed by scaling laws, mathematical relationships that describe how model performance improves with more parameters, more data, and more compute. Research from DeepMind (published in what's known as the Chinchilla paper) showed that many of the largest models released before 2022 were actually undertrained, they'd used too many parameters relative to the amount of data they were trained on. A smaller, better-trained model could match or beat a larger, undertrained one at lower energy cost.

This insight reshaped how labs design training runs. It doesn't eliminate the energy problem, but it does mean that raw model size is a poor proxy for energy consumption. A 70-billion-parameter model trained carefully on 2 trillion tokens might be more capable, and more energy-efficient per unit of performance, than a 175-billion-parameter model trained on less data.

There's also the question of what happens after training. Models deployed at scale often get compressed through techniques like quantisation (reducing the numerical precision of weights) and distillation (training smaller models to mimic larger ones). These can dramatically reduce inference costs without significantly degrading performance.

The uncomfortable reality, though, is that efficiency gains have historically been outpaced by demand growth. As models get cheaper to run, labs build bigger ones and deploy them more widely. Researchers call this Jevons' paradox applied to AI: efficiency improvements tend to increase total consumption rather than reduce it.

What does AI's electricity appetite mean for the grid?

Power grid operators in the United States, Ireland, and across Europe are already flagging AI data centres as a significant planning challenge. Ireland, which hosts major hyperscale data centres for Google, Meta, and Microsoft, saw its data centres consume roughly 21% of the country's total electricity in 2023, a figure that prompted EirGrid, the national grid operator, to warn about capacity constraints.

In the United States, utilities in Virginia, home to the highest concentration of data centres on Earth, have begun revising their load forecasts dramatically upward. Dominion Energy, which serves northern Virginia, projected in 2024 that it would need to add significant new generation capacity over the coming decade, driven largely by AI and data centre growth.

The response from AI companies has been varied. Microsoft announced a partnership with Constellation Energy to restart Three Mile Island's nuclear reactor in Pennsylvania specifically to power its data centres. Google has signed long-term power purchase agreements for geothermal energy. Amazon has invested in nuclear power startups. These moves signal that the industry itself recognises the electricity problem isn't abstract, it's a near-term operational constraint.

The grid implication cuts both ways. More electricity demand can accelerate investment in new generation, including renewables. But in regions where the grid still relies heavily on fossil fuels, AI training runs are, right now, burning coal and gas. The carbon footprint of a model depends enormously on where in the world it's trained, and at what time of year, a training run in Norway, powered by hydroelectricity, has a fraction of the carbon impact of the same run in a coal-heavy region.

Can AI training ever become truly efficient?

The honest answer is: more efficient, yes. Truly efficient in any absolute sense, almost certainly not at the rate the industry is moving.

There are genuine technical pathways to lower energy consumption. Neuromorphic chips, hardware designed to mimic the brain's sparse, event-driven signalling, could theoretically perform AI computations at a fraction of current energy costs. Intel's Loihi 2 chip and IBM's research into neuromorphic architectures point in this direction, though they remain far from deployment at training scale.

Alternative training methods are also gaining traction. Mixture-of-Experts (MoE) architectures, used in models like Google's Gemini, activate only a subset of the model's parameters for any given input. This means the full model is large on paper, but the active compute per query is much smaller, a meaningful efficiency gain.

Better data curation is perhaps the most underappreciated lever. Research consistently shows that training on higher-quality, more carefully filtered data produces better models with less compute. If you can train a capable model on 500 billion tokens of excellent data rather than 2 trillion tokens of mixed-quality data, you've cut the energy bill dramatically.

But none of these advances change the fundamental physics. Moving information through silicon at scale takes energy. Until we have room-temperature superconductors or some other physics-defying breakthrough, every forward pass and every backpropagation step will require electrons flowing through resistive circuits, generating heat, demanding cooling, and drawing from the grid. The question for the next decade isn't whether AI will use electricity. It's whether that electricity will be clean.

The electricity demand of AI training isn't a bug or a PR problem. It's a direct consequence of how machine learning works: learning requires repetition, repetition requires computation, and computation requires power. The scale at which modern AI operates makes that power demand extraordinary. What changes when you understand this is how you think about AI's true cost. Every model released, every chatbot query, every image generated carries an invisible energy price tag. The industry is starting to face this, nuclear deals, renewable contracts, hardware innovations, but the trajectory is still upward. The grid is now, quietly, an AI infrastructure problem.


Originally published on SnackIQ

Top comments (0)