The AI industry has spent the past several years discussing one dominant infrastructure problem:
There is not enough compute.
Companies are racing to secure GPUs. Cloud providers are expanding capacity. Governments are funding new data centres. Startups are competing for credits, reservations, and access to the latest accelerators.
But the shortage narrative hides another problem.
A significant amount of the compute we already have is not being used effectively.
In 2026, Gartner forecasts worldwide AI spending will reach $2.59 trillion, representing 47% year-over-year growth. Goldman Sachs’ baseline model separately estimates approximately $765 billion in annual AI capital expenditure during 2026.
This is an infrastructure build-out of historic proportions.
Yet buying more GPUs does not automatically produce more useful AI work.
Compute only creates value when the correct workload reaches the correct hardware, starts successfully, remains healthy, produces the expected output, and does so at a reasonable cost.
That execution chain is still surprisingly fragile.
The 5% utilization statistic needs attention — and context
CAST AI’s 2026 State of Kubernetes Optimization Report measured GPU utilization across tens of thousands of non-optimized production Kubernetes clusters running on AWS, Microsoft Azure, and Google Cloud.
Average GPU utilization across the clusters analyzed was just 5%.
CAST AI noted that, at that utilization level, organizations had roughly 20 times as much GPU capacity as their workloads were consuming at a given moment.
That is an alarming result, but it must be interpreted carefully.
It does not mean that 95% of every GPU purchased worldwide is permanently wasted. The measurement covers a particular population of non-optimized Kubernetes environments. Some unused capacity may also be deliberate headroom for traffic spikes, reliability requirements, or scheduling constraints.
Nevertheless, the result exposes something important:
Provisioned compute and productive compute are not the same thing.
An organization may technically possess substantial GPU capacity while still struggling to execute workloads efficiently.
The issue is not simply hardware availability. It is the software and operational machinery surrounding that hardware.
Compute waste occurs at multiple layers
When people hear “idle GPU,” they often imagine a server doing absolutely nothing.
In practice, AI-compute inefficiency is more complicated. Waste appears at several stages of the workload lifecycle.
1. Provisioning waste
Teams frequently reserve more capacity than their applications consume.
This happens because infrastructure is usually provisioned against requested resources, anticipated peaks, or broad hardware categories rather than the workload’s observed behaviour.
An inference service may require large amounts of GPU memory but relatively little continuous compute. A batch workload may alternate between intense GPU processing and CPU-bound data preparation. A training job may request several accelerators but use them unevenly.
The hardware is allocated, but its components are not necessarily saturated.
2. Workload-configuration waste
Even after a job reaches a GPU, it may use that GPU poorly.
A 2024 Microsoft Research study examined 400 industrial deep-learning jobs with average GPU utilization of 50% or less. Researchers identified 706 separate low-utilization issues.
Approximately 46% of those issues involved data operations, while roughly 45% were related to deep-learning models. The researchers found that low utilization commonly arose from insufficient GPU computation or interruptions caused by work being performed elsewhere in the system.
Crucially, they concluded that about 85% of the identified issues could be addressed through a relatively small number of code or script modifications.
This means low utilization is not always an unavoidable hardware limitation.
It can be caused by:
- inefficient data loading;
- inappropriate batch sizes;
- CPU bottlenecks;
- excessive synchronization;
- model configurations that do not fit the selected hardware;
- slow storage or network transfers;
- workloads that should not have been assigned an entire accelerator.
The GPU may be active, but it is still not being used efficiently.
3. Failure waste
The most obvious form of compute waste is a workload that runs, consumes resources, and eventually fails.
Microsoft researchers previously studied 4,960 failed deep-learning jobs from an internal production platform. They found that 48% of the failures occurred in interactions with the platform rather than in the core program logic.
Many were connected to differences between local development environments and the environment in which jobs were ultimately executed.
This distinction matters.
A developer can have correct model code and still lose compute because of:
- an incompatible runtime;
- a missing package;
- an invalid container entrypoint;
- incorrect storage permissions;
- insufficient GPU memory;
- an unsupported accelerator architecture;
- a malformed environment variable;
- a provider-specific configuration mismatch.
When these problems are detected only after provisioning begins, the organization pays to discover an error that could potentially have been caught before execution.
4. Operational waste
There is also a category of waste that does not appear directly on the cloud invoice.
It appears in engineering time.
AI teams routinely spend hours moving between provider consoles, comparing GPU types, checking quotas, locating logs, downloading artifacts, restarting failed jobs, and trying to determine whether a failure came from their code or the infrastructure.
A workload that costs $20 in failed compute may also consume several hours of a highly paid engineer’s time.
That operational burden becomes more severe when teams use multiple providers. Each platform has its own terminology, APIs, capacity behaviour, authentication model, logs, storage conventions, and failure states.
The result is a fragmented execution process held together by scripts, dashboards, and institutional knowledge.
Why adding more GPUs will not solve this
The natural response to capacity pressure is to increase supply.
More GPUs are certainly needed. AI demand is growing quickly, and many organizations genuinely face capacity constraints.
But additional supply alone does not fix poor workload placement, failed execution, inefficient configurations, fragmented observability, or idle reservations.
Without better execution infrastructure, increasing capacity can simply increase the amount of infrastructure available to be misallocated.
The industry therefore needs to ask a different question.
Not only:
Where can we find more compute?
But also:
How do we ensure every workload receives appropriate compute and successfully converts it into useful output?
That is an orchestration and execution problem.
The missing abstraction in AI infrastructure
Application developers generally do not want to become experts in every GPU, cloud provider, scheduler, container runtime, and regional capacity market.
They want to describe a workload and receive a predictable result.
A stronger execution layer should be able to:
Understand the workload before submission
Inspect resource requirements, container configuration, runtime limits, storage needs, and expected outputs before expensive infrastructure is provisioned.Match workloads to appropriate compute
Not every AI workload requires the fastest or most expensive GPU. The correct decision depends on memory, architecture compatibility, latency, duration, availability, and cost.Separate execution from individual providers
Developers should not have to rebuild their operational workflow whenever capacity moves from one provider to another.Make failure states understandable
Logs, status changes, retries, cancellations, and artifacts should be exposed through a consistent interface.Preserve execution history
A job should have a durable record showing what was requested, where it ran, what resources were used, why it failed or succeeded, and what it produced.Estimate before committing
Teams should know the likely cost and whether suitable capacity exists before launching a workload.
This does not eliminate every failure or guarantee perfect utilization.
It does, however, move important decisions earlier in the execution lifecycle — before they become expensive.
From infrastructure access to execution intelligence
The first generation of AI infrastructure focused on access.
Could a developer obtain a GPU?
The next generation must focus on execution intelligence.
Which GPU should be used? Is the workload compatible with it? Is there a cheaper appropriate option? Is capacity genuinely available? Can the job recover from failure? Are its outputs verifiable? Can the execution move between providers without forcing the developer to redesign the workflow?
These are becoming essential infrastructure questions.
At Jungle Grid, this is the problem we are working on.
We are building an execution layer through which AI workloads can be submitted, screened, routed to available compute, monitored, retried, and returned with consistent logs and artifacts.
The objective is not merely to expose more GPUs.
It is to make heterogeneous compute behave like dependable execution infrastructure.
That distinction is important because the future of AI will not be built on one provider, one GPU type, or one deployment model.
Workloads will increasingly move across hyperscalers, specialized GPU clouds, regional providers, private clusters, and distributed infrastructure.
The winning abstraction will not be the provider dashboard.
It will be the execution layer that determines how workloads move through all of them.
The real opportunity
The AI-compute conversation has been dominated by scarcity.
Scarcity is real, but it is only one side of the problem.
The other side is that expensive capacity is frequently idle, poorly matched, incorrectly configured, operationally fragmented, or consumed by jobs that never produce a valid result.
The organizations that solve this will create value in two ways:
They will expand access to compute, and they will make existing compute substantially more productive.
In an industry preparing to spend trillions of dollars on AI, even modest improvements in execution efficiency could have enormous consequences.
The next infrastructure breakthrough may not simply be a faster GPU.
It may be the system that ensures we stop wasting the GPUs we already have.
Top comments (0)