Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving

In Part 1 of AI Infrastructure for Cloud Engineers, we looked at why Kubernetes is becoming an important foundation for production AI systems.

Read Part 1: Why Kubernetes Is Becoming the Operating System for AI Infrastructure

Now the question is: what actually changes when we start running AI workloads on Kubernetes?

Traditional applications are usually scheduled around familiar resources such as CPU, memory, storage, and network capacity.

AI workloads introduce another resource that changes the infrastructure equation:

GPU

GPUs are powerful, expensive, and limited. Once teams start running model inference, embedding services, fine-tuning jobs, or other AI workloads at scale, simply deploying a container is no longer enough.

The platform also needs to decide:

• Which workload gets which GPU?
• How should those workloads be scheduled?
• When should inference capacity scale?
• How do we avoid leaving expensive accelerators idle?
• How should models be served reliably in production?

This is where Kubernetes scheduling, autoscaling, and model-serving patterns start to matter.

Let's look at how these pieces fit together.

1. How Kubernetes Sees a GPU

Kubernetes normally schedules Pods based on resources such as CPU and memory.

A basic application might request:

resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "1"
    memory: "2Gi"

GPU-enabled nodes add specialized resources to the cluster.

A workload can then request a GPU:

resources:
  limits:
    nvidia.com/gpu: 1

Conceptually, the cluster might look like this:

Kubernetes Cluster

Node A
├── CPU
├── Memory
└── No GPU

Node B
├── CPU
├── Memory
└── GPU

Node C
├── CPU
├── Memory
└── GPU

If an AI workload requests a GPU, Kubernetes needs to place it on a node where that resource is available.

Hardware vendors commonly expose devices such as GPUs to Kubernetes through mechanisms including device plugins.

That sounds straightforward.

At larger scale, however, GPU scheduling becomes much more interesting.

2. Not Every GPU Is the Same

Imagine a cluster with multiple accelerator types.

Node A → NVIDIA T4
Node B → NVIDIA A100
Node C → NVIDIA H100
Node D → CPU only

Now imagine three workloads:

Small embedding model
Large language model
Distributed training job

Placing all three randomly would be inefficient.

The embedding workload may not need the most powerful GPU, while the large model may require significantly more accelerator memory and compute.

This means AI platforms often need to consider:

GPU type
GPU memory
Workload size
Topology
Availability
Cost
Priority

Modern Kubernetes scheduling is evolving specifically for these kinds of workloads.

Kubernetes v1.36, for example, introduced further workload-aware scheduling capabilities including PodGroups, topology-aware scheduling, workload-aware preemption, and integration with Dynamic Resource Allocation. These features are particularly relevant to tightly coupled AI/ML and batch workloads.

3. Why Workload-Aware Scheduling Matters

Traditional Kubernetes scheduling largely thinks about individual Pods.

AI workloads may need Kubernetes to think about a group of Pods together.

Imagine distributed training that requires four workers:

Training Job

Worker 1
Worker 2
Worker 3
Worker 4

Scheduling only two workers while the others remain pending may not be useful if the job requires all four before it can start.

This is the idea behind gang scheduling.

Enough resources for all workers?

        Yes
         ↓
Schedule workload

        No
         ↓
Wait for capacity

Topology can also matter.

If several workers constantly exchange large amounts of data, placing them far apart across the infrastructure may introduce unnecessary network overhead.

Workload-aware and topology-aware scheduling allow Kubernetes to make placement decisions using more context about the complete workload rather than treating every Pod independently.

4. GPU Utilization Matters

GPUs can represent a significant portion of the infrastructure cost behind self-hosted AI.

That makes low utilization expensive.

Imagine:

GPU Capacity
████████████████████ 100%

Actual Workload
██████               30%

The remaining capacity is still being paid for.

This can happen when:

• Models are oversized for the hardware
• Traffic is inconsistent
• Each workload reserves an entire accelerator
• Requests are not batched efficiently
• Too many replicas are running
• GPUs are fragmented across teams

One goal of an AI platform is therefore not just:

Make the model run.

It is:

Keep the model responsive while using expensive compute efficiently.

The cloud-native ecosystem is increasingly developing GPU-sharing and accelerator-aware scheduling approaches for this reason. For example, HAMi focuses on sharing and scheduling heterogeneous accelerator resources, while Kubernetes Dynamic Resource Allocation provides a more flexible mechanism for requesting specialized devices.

5. Inference Is Different From Training

AI infrastructure discussions often combine training and inference, but they have different operational characteristics.

Training

Training commonly looks like:

Dataset
   ↓
Training Job
   ↓
Many GPUs
   ↓
Hours / Days
   ↓
Model

The workload may require several accelerators simultaneously and run for a long period.

Inference

Inference looks more like:

User Request
      ↓
Model Server
      ↓
GPU
      ↓
Generated Response

Inference is usually much more sensitive to:

Latency
Throughput
Availability
Queue depth
Concurrent requests

For a user-facing AI application, a model that eventually returns the correct answer is not enough.

It also needs to respond within an acceptable amount of time.

That changes how we think about scaling.

6. CPU Is Not Always the Right Scaling Metric

For many web applications, Kubernetes autoscaling might use CPU utilization.

CPU > 70%
     ↓
Add Pods

That can work well for traditional services.

AI inference may need different signals.

Imagine an inference server where:

CPU = 35%
GPU = 92%
Waiting requests = 120

From CPU alone, the application may appear healthy.

From the user's perspective, it may already be overloaded.

Better AI scaling signals may include:

GPU utilization
Requests waiting
Concurrent requests
Inference latency
Tokens per second
KV cache utilization
Queue depth

KServe, for example, supports autoscaling inference workloads using external LLM metrics through technologies such as KEDA, Prometheus, and OpenTelemetry. Its documentation includes examples based on active or waiting inference requests rather than relying only on CPU.

A simplified scaling flow could look like this:

Request Queue
     ↓
Waiting requests increase
     ↓
Autoscaling signal
     ↓
Create more inference replicas
     ↓
More capacity available

7. What Model Serving Actually Means

A trained model is essentially an artifact.

Users still need a service capable of loading the model and accepting requests.

That layer is commonly called model serving.

Conceptually:

Application
     ↓
Model Endpoint
     ↓
Inference Server
     ↓
Model
     ↓
GPU

A production model-serving layer may need to handle:

• Model loading
• Request routing
• Batching
• Autoscaling
• Health checks
• Model versions
• Metrics
• GPU allocation
• Failure recovery

Instead of application developers building all of this independently, model-serving frameworks can provide reusable infrastructure.

One Kubernetes-native example is KServe, which provides abstractions for deploying and operating inference workloads on Kubernetes.

The wider cloud-native ecosystem is also building more specialized inference infrastructure. Kubernetes' former WG Serving helped advance inference-oriented capabilities including request scheduling and gateway patterns before concluding its work in 2026.

8. A Simplified Kubernetes AI Architecture

Putting the pieces together, an inference platform might look like this:

                    Users
                      ↓
               API / AI Gateway
                      ↓
                Request Router
                      ↓
          ┌───────────┼───────────┐
          ↓           ↓           ↓
     Model Pod    Model Pod    Model Pod
          ↓           ↓           ↓
        GPU         GPU         GPU

              Kubernetes Cluster
                      ↓
        ┌─────────────┼─────────────┐
        ↓             ↓             ↓
   Autoscaling    Monitoring     Scheduling

Kubernetes handles the infrastructure layer.

The model-serving layer handles inference-specific concerns.

Together, they allow the platform to respond to changing demand.

9. Model Routing Is Becoming Smarter

Basic load balancing assumes that multiple application replicas are roughly interchangeable.

AI inference can be different.

The best place to route a request may depend on:

• Which model is already loaded
• Available GPU memory
• Current request queue
• Cached model state
• Accelerator type
• Current inference load

So instead of:

Request
   ↓
Random Pod

AI-aware routing can move toward:

Request
   ↓
Inference Gateway
   ↓
Best available model server

Modern cloud-native inference projects are increasingly exploring model-aware and state-aware routing.

For example, llm-d focuses on capabilities such as inference scheduling, KV-cache-aware behavior, and separating prompt processing from token generation to improve resource utilization and inference performance.

This is one of the clearest examples of Kubernetes infrastructure adapting specifically to AI workloads.

10. Scaling the Pod Does Not Scale Everything Else

There is an important limitation to remember.

Suppose Kubernetes scales an inference application:

2 replicas
    ↓
4 replicas
    ↓
8 replicas

Those eight replicas may now generate much more traffic toward:

Vector database
Object storage
External APIs
Model storage
Network
GPU nodes

Scaling one component can simply move the bottleneck somewhere else.

For example:

Inference Pods
████████████████  Healthy

        ↓

Vector Database
████████████████  Overloaded

Capacity planning needs to consider the complete request path.

This is the same lesson cloud engineers already know from distributed systems.

AI does not remove bottlenecks.

It introduces some new ones.

11. Failure Handling Still Matters

Suppose a GPU node fails while running an inference workload.

The platform needs to detect that condition and recover.

A production architecture should consider:

Pod failures
Node failures
GPU failures
Model loading failures
Provider failures
Network failures
Out-of-memory conditions

Kubernetes can restart or reschedule workloads, but AI platforms also need visibility into accelerator health and inference behavior.

Recent Kubernetes Dynamic Resource Allocation work includes exposing device health information to workloads and controllers, which can help operators understand failures involving specialized hardware.

The key point is simple:

Process running
        ≠
AI service healthy

Infrastructure health and application health both matter.

12. What Should We Monitor?

For an AI workload running on Kubernetes, I would separate metrics into three layers.

Kubernetes

Pod availability
Pod restarts
Node health
CPU
Memory
Network

GPU

GPU utilization
GPU memory
Accelerator availability
Device health

Inference

Request latency
Queue depth
Requests running
Tokens per second
Time to first token
Inference errors

Looking at only one layer can hide the real problem.

For example:

Kubernetes
Pods healthy ✓

GPU
Utilization 100%

Inference
Latency increasing ↑
Queue growing ↑

The cluster is technically running.

The service is still degrading.

What Cloud Engineers Should Take Away

You do not need to become a machine-learning researcher to work with AI infrastructure.

The infrastructure problems remain very familiar:

Scheduling
Scaling
Networking
Capacity
Observability
Security
Reliability
Cost

The difference is the resource being managed.

Instead of only asking:

How much CPU?
How much memory?

we now also ask:

Which GPU?
How much GPU memory?
Which model?
How many concurrent requests?
How many tokens per second?
Where should this inference request run?

That is the bridge between traditional cloud engineering and AI infrastructure.

Practical Architecture Checklist

Before running AI workloads on Kubernetes, think about:

• Which workloads actually require GPUs
• Which accelerator types each workload needs
• Whether workloads need single or multiple GPUs
• How GPU nodes are isolated from general workloads
• How models are served
• Which metrics trigger scaling
• How requests are routed
• How GPU utilization is measured
• Whether several workloads can share accelerator capacity
• What happens when a GPU or node fails
• Whether downstream dependencies can handle additional replicas
• How much idle accelerator capacity you are paying for

What's Next?

GPUs, scheduling, and model serving solve only part of the production problem.

Once the application is running, the next question becomes:

How do we know whether the AI system is actually healthy?

Traditional infrastructure monitoring gives us CPU, memory, and Pod health.

AI workloads introduce another set of signals including inference latency, token throughput, GPU utilization, queue depth, model failures, and cost.

That is what we will cover next.

Part 3: Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory

Final Thoughts

Running AI workloads on Kubernetes is not simply a matter of adding a GPU to a Pod.

Production systems need to think about the complete lifecycle:

GPU Allocation
      ↓
Scheduling
      ↓
Model Serving
      ↓
Request Routing
      ↓
Autoscaling
      ↓
Observability
      ↓
Failure Recovery

Kubernetes gives us a strong orchestration foundation.

But AI introduces new constraints around expensive accelerators, workload placement, inference latency, and resource utilization.

The interesting shift is that Kubernetes is beginning to understand more about these workloads directly, while projects around it are adding the inference-specific capabilities required to operate AI efficiently.

For cloud engineers, this is where existing Kubernetes knowledge starts becoming directly useful in the AI infrastructure world.

Thanks for Reading

This article is Part 2 of my AI Infrastructure for Cloud Engineers series:

1. Why Kubernetes Is Becoming the Operating System for AI Infrastructure
2. Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving
3. Coming next: Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory
4. FinOps for AI: Understanding GPU, Token, and Inference Costs
5. Building a Production AI Platform: Kubernetes, GitOps, IaC, Security, and Observability

I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, and the engineering behind production AI systems.

Looking forward to connect, learn and grow together 😄

LinkedIn: Connect with me on LinkedIn

If you're running AI workloads on Kubernetes, what has been harder in practice: GPU allocation, autoscaling, model serving, or keeping the GPUs efficiently utilized?