The Allocation vs. Yield Gap in Inference
Most engineering teams face the same harsh reality when moving AI models from batch training to real-time inference: You provision expensive accelerated infrastructure on AWS, your Kubernetes dashboard reports one hundred percent utilization, but your actual hardware yield is practically zero. Inference workloads are inherently bursty and latency-sensitive, which completely breaks traditional monolithic GPU allocation.
In our production EKS cluster, every pod requested an entire GPU, meaning a single low-intensity microservice locked an entire expensive device. If a workload only required 15% of a GPU's compute capacity to serve requests, the remaining 85% simply went to waste. That massive inefficiency was a major problem for our cloud economics.
But an even greater problem was that, initially, we couldn't even see the waste. The standard monitoring tools offered false comfort. Relying on the default utilization metric is like checking if a stove's pilot light is on to measure how productive a kitchen is. It only confirms that at least one kernel ran during a brief sampling window. We had to deploy DCGM to see the truth.
To truly monitor GPU usage, you must look beyond surface-level metrics and supplement your standard tools with DCGM-backed Prometheus instrumentation. You must track specific execution metrics, primarily SM Active, which reveals the exact ratio of clock cycles where a Streaming Multiprocessor has at least one warp assigned. You must also monitor SM Occupancy to verify if your workload is supplying enough parallel data to saturate the hardware. Finally, combining these with DRAM Active and Tensor Pipeline metrics confirms whether your models are bottlenecked by memory transfers or successfully hitting the hardware fast path.
Our Streaming Multiprocessor active rate hovered around ten percent, leaving the vast majority of our compute power completely untapped.
Standard exclusive allocation showing 100% K8s allocation, but DCGM SM Active hovering at a dismal 10% with severe tail latency spikes.
Architectural Paradigms for GPU Concurrency
To reclaim this stranded capacity, we evaluated the three standard sharing architectures to see what would actually survive our production traffic.
| Sharing Strategy | Execution Model | Hardware Pool | Memory Isolation | Best Enterprise Fit |
|---|---|---|---|---|
| Time-Slicing | Sequential | Extremely Broad | None | Development, non-critical batch |
| Multi-Instance GPU (MIG) | Parallel | Highly Limited | Strict Hardware Level | Multi-tenant SaaS, strict QoS |
| Multi-Process Service (MPS) | Parallel | Broad | Limited (Client-Server Bounds) | Trusted internal inference fleets |
Time-slicing divides access into sequential intervals. While easy to configure, it offers no memory isolation, and a heavy query can block other pods sharing the device, destroying our strict latency SLAs.
MIG provides flawless hardware-level isolation, but the partitions are rigidly fixed to specific profiles. If a workload needs slightly more memory than a standard slice provides, you have to round up to the next available size, permanently stranding expensive VRAM. Crucially, MIG drastically limits your potential GPU instance pool because it is not supported by all instances. Relying on MIG forces you into a much narrower and costlier segment of the cloud hardware catalog, eliminating the ability to scale out on more cost-effective options.
MPS operates as a client-server architecture that acts as a transparent, binary-compatible implementation of the CUDA API. Unlike time-slicing and MIG, MPS allows multiple distinct processes to share a single CUDA context. This enables their individual kernels to execute concurrently across the same Streaming Multiprocessors, dynamically sharing compute and memory resources on the fly.
We ultimately chose MPS for our fleet because it represents the optimal sweet spot for inference economics. Since real-time inference requests are intermittent, individual models rarely saturate a GPU's full parallel processing power. MPS allows the hardware to overlap kernel launches from completely different pods onto the same compute cores simultaneously. This completely circumvented the latency jitter of time-slicing and bypassed the partition fragmentation of MIG, allowing us to maximize raw hardware throughput for our trusted internal workloads.
Overcoming the Kubernetes Integer Trap with Karpenter
Choosing MPS was only the first step. Actually deploying this at scale across an auto-scaling EKS cluster introduced a massive scheduling headache. Kubernetes natively treats a GPU as an indivisible whole integer resource. When four pending inference pods each requested a GPU, our provisioning engine, Karpenter, assumed four physical instances were required. Without intervention, Karpenter scaled out and launched four separate nodes. This stranded massive amounts of premium CPU and memory across our EKS fleet, completely defeating the economic purpose of resource sharing.
To solve this integer trap, we paired the container-optimized Bottlerocket operating system with Karpenter NodeOverlay. First, we configured the node's underlying operating system using a declarative layout inside our custom resource definitions. This securely initialized the host-level MPS control daemon and commanded the device plugin to expose virtualized replicas to the cluster.
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: mps-bottlerocket
spec:
amiFamily: Bottlerocket
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: production-cluster
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: production-cluster
userData: |
[settings.kubelet-device-plugins.nvidia]
device-sharing-strategy = "mps"
[settings.kubelet-device-plugins.nvidia.mps]
replicas = 4
To prevent Karpenter from over-provisioning physical instances based on this demand, we deployed a NodeOverlay. This custom resource intercepted Karpenter's simulation logic, instructing it to treat single-GPU instance shapes as possessing four schedulable virtual allocations. Karpenter immediately recalculated that a single physical instance could satisfy the aggregate demand of four virtual devices, launching exactly one node instead of four.
apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
name: mps-gpu-virtual-packing
spec:
weight: 10
requirements:
- key: karpenter.k8s.aws/instance-gpu-count
operator: In
values: ['1']
- key: karpenter.k8s.aws/instance-gpu-manufacturer
operator: In
values: ['nvidia']
capacity:
nvidia.com/gpu: '4'
The impact on our infrastructure economics was immediate. By overlapping kernel launches from completely different pods onto the same compute cores simultaneously, our Streaming Multiprocessor active rate skyrocketed from a dismal ten percent to nearly ninety percent. We collapsed our cloud footprint and drastically reduced our hourly compute spend.
MPS enabled with NodeOverlay virtual packing. DCGM SM Active saturated at 87% with tightly compressed P95/P99 latency.
The Bottom Line
The results of this architectural shift were immediate and transformative for our production environment. By successfully configuring the system so that four pods can share a single GPU, an instance equipped with 8 GPUs can now seamlessly serve 32 inference pods simultaneously.
This massive density consolidation drastically reduced our hourly GPU compute costs, as we are finally extracting the full value out of the hardware we pay for. Beyond direct cost savings, our cluster's responsiveness improved significantly; we slashed overall bootstrap times because Karpenter no longer needs to provision and boot a brand new machine for every pending pod. Furthermore, this setup allows us to run multiple different inference models concurrently on the exact same underlying hardware.
Ultimately, surviving the transition to large-scale inference requires abandoning static allocation. This MPS-backed architecture, combined with Karpenter's virtual packing, proves that teaching your orchestrator to understand true concurrent capacity is the ultimate key to unlocking maximum GPU yield.
Top comments (1)
I was facing the same issue. Thank you for the blog post