The Self-Aware GPU Fleet: Closed-Loop Autoscaling with KEDA, MCP, and OKE

How two open-source projects turn OKE GPU nodes into infrastructure that observes itself, scales itself, and tells your AI agents what's happening in real time.

---

The Missing Feedback Loop

Most GPU infrastructure on Kubernetes has a blind spot. Your inference servers run on expensive GPU nodes. Something scrapes metrics. Something else displays dashboards. A human looks at the dashboard, realizes GPUs are overloaded (or idle), and manually adjusts replica counts or node pools.

That's not automation. That's a human in the middle of a feedback loop that should be closed.

I wanted two things for GPU workloads on OKE:

1. Autoscaling that reacts to actual GPU load not CPU proxies, not PromQL queries that lag by 30 seconds, but direct NVML reads that drive scaling decisions in under 5 seconds.
2. AI agents that can see GPU state so when Claude or Goose is helping me troubleshoot an inference deployment, it knows the GPU is at 94% memory before I have to check a dashboard and paste the number.

These are different problems solved by different tools. But when you deploy them together on OKE, you get something more interesting: a GPU fleet that observes itself and scales itself, with AI agents as first-class participants in the loop.

Two Projects, One NVML Foundation

Both tools read GPU metrics from the same source. NVIDIA's NVML library via Go's cgo bindings, but serve them to completely different consumers:

Project	Consumer	Protocol	Purpose
keda-gpu-scaler	KEDA operator	gRPC	Autoscale deployments based on GPU load
gpu-mcp-server	AI agents (Claude, Goose, Cursor)	MCP (stdio/HTTP)	Give agents real-time GPU observability

Both run as DaemonSets on GPU nodes. Both are lightweight under 128MB of memory each. Neither requires Prometheus, dcgm-exporter, or any metric pipeline. They go straight to the hardware.

┌────────────────────────────────────────────────────────────────────────┐
│                    OKE Cluster — GPU Node Pool                         │
│                                                                        │
│  ┌─────────────────┐                                                   │
│  │  KEDA Operator   │ ◀── gRPC ──┐                                    │
│  │  (scales HPAs)   │            │                                    │
│  └────────┬────────┘            │                                    │
│           │ HPA                  │                                    │
│           ▼                      │                                    │
│  ┌─────────────────┐    ┌───────┴──────────────────────────────────┐  │
│  │  vLLM Inference  │    │  GPU Worker Node (VM.GPU.A10.1)         │  │
│  │  Deployment      │    │                                          │  │
│  │  (0-8 replicas)  │    │  ┌──────────────┐  ┌────────────────┐  │  │
│  └─────────────────┘    │  │keda-gpu-scaler│  │gpu-mcp-server  │  │  │
│                          │  │(DaemonSet)    │  │(DaemonSet)     │  │  │
│  ┌─────────────────┐    │  │               │  │                │  │  │
│  │  AI Agent Pod    │    │  │  NVML ──┐     │  │  NVML ──┐     │  │  │
│  │  (Claude/Goose)  │◀── MCP ──────────────│──│         │     │  │  │
│  │  "Why is latency │    │  │        ▼     │  │        ▼     │  │  │
│  │   spiking?"      │    │  │    GPU 0     │  │    GPU 0     │  │  │
│  └─────────────────┘    │  └──────────────┘  └────────────────┘  │  │
│                          └───────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────────────┘

The keda-gpu-scaler handles the scaling loop it reads GPU utilization every few seconds and tells KEDA whether to scale up, scale down, or scale to zero. The gpu-mcp-server handles the observability loop — it answers agent queries about GPU state so that AI assistants have hardware context for their reasoning.

Together, they make the GPU fleet self-aware in two senses: it automatically reacts to load (scaling), and it can describe its own state to anyone who asks (observability).

The Closed Loop in Practice

Here's what this looks like in a real OKE deployment serving LLM inference.

Morning: Traffic Ramp-Up

8 AM. Users start hitting your vLLM endpoint. The keda-gpu-scaler sees GPU memory climbing past 80% on the vllm-inference profile threshold. KEDA scales the deployment from 1 to 3 replicas. OKE's cluster autoscaler provisions additional VM.GPU.A10.1 nodes to fit the new pods. Nobody intervened.

Midday: Agent-Assisted Troubleshooting

11 AM. Inference latency spikes. You ask Claude: "What's happening with the GPU fleet?"

Claude calls gpu_summary via MCP and gets:

{
  "device_count": 3,
  "avg_gpu_utilization": 91.2,
  "avg_memory_utilization": 88.7,
  "total_memory_used_mib": 65536,
  "total_memory_total_mib": 73728,
  "max_temperature_celsius": 78,
  "total_power_draw_watts": 405
}

Claude sees 88.7% average memory utilization across 3 GPUs and says: "Your vLLM KV cache is nearly full on all replicas. The keda-gpu-scaler should be triggering a scale-up. Check the ScaledObject your maxReplicaCount might be capped at 3."

That's the closed loop. The agent didn't need a Grafana dashboard. It queried the GPUs directly, correlated the data with the scaling config, and identified the bottleneck.

Evening: Scale-to-Zero

9 PM. Traffic drops. GPU memory falls below the 5% activation threshold on the vllm-inference profile. keda-gpu-scaler tells KEDA the deployment is inactive. KEDA scales to zero replicas. OKE's cluster autoscaler eventually drains and removes the idle GPU nodes. Your bill stops accumulating.

Next morning, the first request triggers a scale-from-zero. The cold start takes 30-60 seconds (model loading into VRAM), then inference is live again.

Deploying Both on OKE

Prerequisites

• OKE cluster with GPU node pool (VM.GPU.A10.1 or BM.GPU.A10.4)
• KEDA v2.10+ installed
• NVIDIA GPU drivers and device plugin (standard on OKE GPU node pools)

1. Install KEDA

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

2. Deploy keda-gpu-scaler

helm install keda-gpu-scaler \
  oci://ghcr.io/pmady/charts/keda-gpu-scaler \
  --namespace keda --create-namespace

3. Deploy gpu-mcp-server

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-mcp-server
  namespace: gpu-monitoring
spec:
  selector:
    matchLabels:
      app: gpu-mcp-server
  template:
    metadata:
      labels:
        app: gpu-mcp-server
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: mcp
          image: iad.ocir.io/mytenancy/infra/gpu-mcp-server:v1.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "100m"
              memory: "64Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: gpu-mcp-server
  namespace: gpu-monitoring
spec:
  selector:
    app: gpu-mcp-server
  ports:
    - port: 8080

4. Create the ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-gpu-autoscaler
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 0
  maxReplicaCount: 8
  triggers:
    - type: external
      metadata:
        scalerAddress: "keda-gpu-scaler.keda.svc.cluster.local:6000"
        profile: "vllm-inference"

That's the full stack. keda-gpu-scaler handles automated scaling decisions. gpu-mcp-server handles agent-driven observability. Both read from NVML. Both run as lightweight DaemonSets. Total overhead: under 256MB of memory per GPU node.

OCI FinOps: What This Saves

The financial case is straightforward. Without GPU-aware scaling, teams overprovision because they can't trust CPU-based autoscaling for GPU workloads.

Scenario	OCI Shape	Replicas	Hours/Day	Daily Cost
Fixed (no scaling)	VM.GPU.A10.1	4	24	$145.92
GPU-aware + scale-to-zero	VM.GPU.A10.1	1-4 (avg 2.5)	14 active	$53.20
Preemptible + scale-to-zero	VM.GPU.A10.1	1-4 (avg 2.5)	14 active	$16.10

The GPU-aware scaling path saves ~63% over fixed provisioning on on-demand instances. Add preemptible shapes and it's ~89% savings. That's real money the difference between running inference as a line-item expense and running it as a scalable service.

The MCP observability layer doesn't save money directly, but it prevents the alternative: a platform engineer spending 20 minutes in Grafana every time someone asks "why is inference slow?" An agent that can answer that question in 3 seconds changes the operational cost model.

AAIF Alignment

Both projects align with the Agentic AI Foundation's mission. AAIF - a Linux Foundation initiative is building the open-source ecosystem for production AI agent infrastructure. The core idea: agents need standard protocols and tools to interact with infrastructure.

• gpu-mcp-server speaks MCP, the protocol AAIF has aligned around for agent-tool communication.
• keda-gpu-scaler provides the autoscaling layer that agent-driven inference deployments need.

Together, they demonstrate what AAIF calls "agentic infrastructure" systems that are both managed by AI agents and aware of their own hardware state. It's not a demo concept; it's running today on Kubernetes.

Connection to Oracle University Training

This architecture maps cleanly to multiple OCI learning paths:

• OCI AI Infrastructure - GPU shapes (VM.GPU.A10, BM.GPU.H100), NVML, and compute provisioning
• OCI Cloud Native Professional - OKE cluster management, DaemonSets, Helm, service discovery
• OCI FinOps - GPU cost optimization, preemptible instances, right-sizing
• OCI Security Professional - OCIR image management, namespace isolation, pod security

If you're studying for OCI certifications and want a project that touches GPU compute, Kubernetes orchestration, cost optimization, and AI infrastructure simultaneously — deploying this stack is a strong hands-on exercise.

Try It

1. Provision an OKE cluster with a VM.GPU.A10.1 node pool.
2. Install KEDA, keda-gpu-scaler, and gpu-mcp-server.
3. Deploy a vLLM inference server with the vllm-inference scaling profile.
4. Send inference requests and watch keda-gpu-scaler scale up replicas.
5. Connect Claude via MCP and ask: "What's my GPU fleet's status?"
6. Stop sending requests. Watch it scale to zero. Check your OCI bill the next day.

GPU infrastructure shouldn't need a human in the loop to scale or a dashboard to explain itself. The feedback loop should be closed. On OKE, with these two projects, it is.

GitHub: keda-gpu-scaler | gpu-mcp-server

---

Pavan Madduri is a Senior Cloud Platform Engineer at W.W. Grainger, a CNCF Golden Kubestronaut, and CNCF TAG Workloads Foundation Tech Lead. He maintains keda-gpu-scaler and gpu-mcp-server and contributes to CNCF projects including KEDA, Volcano, and Dragonfly. Find him on GitHub.