Originally published on AI Tech Connect.
What you need to know Kubernetes has quietly become the default place teams run LLM inference. According to CNCF's 2025 Annual Cloud Native Survey, 66% of organisations hosting generative AI models use Kubernetes to manage some or all of their inference workloads — the majority, and rising. That matters because the autoscaler most teams reach for first, the standard Horizontal Pod Autoscaler wired to CPU and memory, is built for the wrong bottleneck. A GPU inference pod can sit at effectively 100% GPU while its CPU idles in single digits, so a CPU-based autoscaler either never fires or fires on noise that has nothing to do with whether users are waiting. CPU and memory are the wrong signal. GPU inference is bound by GPU compute and memory bandwidth, not CPU cycles — scale on what the GPU…
Top comments (0)