Key Takeaways
- Gartner projects AI agent inference costs will climb more than fivefold by 2028, even as per-token prices fall.
- A 2026 KAIST paper found AI agents consume up to 136.5 times more energy per query than standard chatbots, with GPUs idle more than half the time between steps.
- Gartner predicts organisations will use small, task-specific models three times as often as general-purpose LLMs by 2027, with hardware and cooling infrastructure becoming the least-visible cost layer. Token prices are falling. Total AI bills are rising anyway. Gartner‘s latest forecast puts inference costs for AI agent workflows more than five times higher by 2028, driven by the compounding compute demands of what agents actually do once they start running.
Why Agents Burn So Much Compute
A standard chatbot handles one question and returns one answer. Agents work differently: they plan, reason across multiple steps, call external tools and repeatedly check back with the underlying model as they work through a task. Each of those check-ins costs compute, and the costs stack quickly.
A 2026 paper from the Korea Advanced Institute of Science and Technology (KAIST) quantified the gap. AI agents consume up to 136.5 times more energy per query than standard models and take up to 153.7 times longer to respond. That extended processing time produces a counterintuitive inefficiency: GPUs sit idle for up to 54.5% of agent runtime, waiting between steps rather than running continuously.
The Inference Paradox
Gartner describes this as the “inference paradox.” Per-token prices for large language models are expected to fall by more than 90% by 2030, according to Gartner. Total spending rises regardless, because agents don’t just use more tokens than a chatbot, they use vastly more, across longer reasoning chains, replanning loops and tool calls. Cheaper tokens multiplied across far more steps still produces a larger bill.
Frontier model training runs can cost tens of millions of dollars in compute, with electricity accounting for a significant share, but the harder cost miscalculation for most enterprises sits at inference, not training, and agents make that gap wider.
Grid Pressure Behind the Scenes
Global data centre power demand is projected to reach 132 gigawatts by 2026 and could exceed 290 GW by 2030, according to Gartner, with AI-optimised servers accounting for a growing share of that total. Modern AI chips draw between 700W and 1,200W per processor, against 150W to 200W for a traditional server CPU. That gap compounds at scale when thousands of processors run continuously.
For teams focused on controlling agent deployment costs the infrastructure layer is where some of the least visible expenses accumulate, cooling, power draw and the idle-GPU overhead the KAIST paper identified all sit below the per-token line on most vendor invoices.
Where Efficiency Gains Are Coming From
Smaller, task-specific models are one of the more practical levers available. A model fine-tuned on a specific business task can be materially more efficient than a general-purpose model, consuming fewer tokens and returning faster responses. Gartner predicts organisations will run small, task-specific models three times as often as general-purpose LLMs by 2027.
Hardware is improving in parallel. Chips designed for AI inference use less power per operation than the general-purpose GPUs that dominated earlier deployments. Liquid cooling is displacing air cooling in newer facilities, and renewable energy is reaching more data centres, though the pace varies. Teams tracking LLM inference costs at the workload level are finding that profiling individual agent steps surfaces savings that aggregate billing obscures.
Transparency remains a structural obstacle. Major AI providers rarely publish detailed energy figures for their proprietary models, which makes it difficult to assess actual environmental cost. Open tools like CodeCarbon can measure the carbon footprint of locally-run models, but that covers only part of the picture. Full accounting, tracking inference costs across retrieval, embeddings, tool calls and the primary model, is something most organisations have not yet attempted.
Originally published at https://autonainews.com/gartner-ai-agent-inference-costs-to-rise-fivefold-by-2028/
Top comments (0)