Self-hosting open-source Large Language Models (LLMs) like Llama 3, DeepSeek, or Mistral on dedicated On-Demand cloud GPU instances gets expensive rapidly. A typical 4x GPU setup easily exceeds 2,000 USD/month on AWS or GCP.
Spot/Preemptible instances offer steep 65% to 75% discounts, but cloud providers can reclaim them with only a 30 to 120-second warning. Without an automated eviction mechanism, node preemptions cause dropped TCP connections, broken streaming completions, and severe HTTP 502/504 spikes.
In this guide, we break down a production-grade Kubernetes architecture that achieves a 0.0% request failure rate during spot node evictions while automatically falling back to On-Demand nodes when spot capacity evaporates.
💰 FinOps & Cost Breakdown
Strategy
Compute Nodes (4x GPUs)
Monthly Cost (est.)
Preemption Failure Rate
Reliability Risk
Standard On-Demand
g4dn.xlarge (AWS) / g2-standard-4 (GCP)
~$2,050 / mo
0%
None (High Cost)
Naive Spot Instances
Spot / Preemptible equivalents
~$615 / mo
12% - 25% dropped reqs
High (HTTP 502/504 spikes)
Spot + Resilient Architecture
Spot with On-Demand Fallback
~$635 / mo
0.0% (Zero dropped reqs)
Zero-Downtime
🏗️ Architecture & Failover Mechanics
[ Ingress / In-Cluster Service Proxy ]
|
+-----------------------+-----------------------+
| |
v v
[ Node Pool: Spot (Worker 2) ] [ Node Pool: On-Demand (Worker 1) ]
+----------------------------+ +---------------------------------+
| [Spot Termination Daemon] | | |
| - Watches metadata API | | |
| - Triggers cordon & drain | | |
| | | |
| vLLM Pod 1 | --Evict--> | vLLM Pod 1 |
| vLLM Pod 2 | --Evict--> | vLLM Pod 2|
+----------------------------+ +---------------------------------+
The 4 Pillars of Resilient Failover
Autonomous Preemption Handler (DaemonSet):
Runs strictly on nodes labeled node-type: spot. It accesses the host node name via the Kubernetes Downward API (spec.nodeName) and monitors cloud preemption metadata. Upon receiving a warning, it cordons the node and initiates an asynchronous kubectl drain.
Pod Disruption Budget (PDB):
Configured with minAvailable: 1. When the node drain fires, Kubernetes rejects simultaneous evictions. It forces sequential evacuation, keeping active inference pods alive until fallback pods pass readiness probes.
Graceful Traffic Handover (preStop Hook):
Inference pods delay shutdown using a lifecycle.preStop sleep hook (5–15 seconds). This grace period allows kube-proxy and ingress controllers to deregister the pod endpoint before the container process receives SIGTERM.
Soft Node Affinity:
Inference workloads use preferredDuringSchedulingIgnoredDuringExecution to prioritize Spot instances during standard scheduling cycles, but fall back seamlessly to On-Demand nodes when Spot capacity is exhausted.
🧪 Validating the Failover Locally with Kind
You can validate this entire failover flow locally without renting cloud GPU instances by using kind (Kubernetes in Docker).
Step 1: Provision the Multi-Node Cluster
Create a cluster with heterogeneous node labels representing Spot and On-Demand pools:
kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker labels: node-type: on-demand
- role: worker labels: node-type: spot
Apply the configuration:
kind create cluster --name spot-lab --config kind-config.yaml
Step 2: Deploy Workloads & Preemption DaemonSet
Deploy the inference service, PodDisruptionBudget, and the preemption listener:
kubectl apply -f https://raw.githubusercontent.com/riyankhamdani/spot-llm-k8s/main/k8s/vllm-resilient.yaml
kubectl apply -f https://raw.githubusercontent.com/riyankhamdani/spot-llm-k8s/main/k8s/spot-handler.yaml
Step 3: Run the Chaos Test
In your first terminal, simulate continuous live inference requests from an in-cluster client locked to the on-demand worker:
kubectl run traffic-tester --rm -it --image=curlimages/curl --restart=Never \
--overrides='{"spec":{"nodeSelector":{"node-type":"on-demand"}}}' -- sh -c '
SUCCESS=0; FAILED=0;
while true; do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 2 -X POST http://vllm-service:8000/v1/completions -H "Content-Type: application/json" -d "{\"prompt\": \"Test\"}")
[ "$STATUS" -eq 200 ] && SUCCESS=$((SUCCESS+1)) || FAILED=$((FAILED+1))
echo "[$(date +%T)] Status: $STATUS | Success: $SUCCESS | Failed: $FAILED"
sleep 0.3
done'
In a second terminal, trigger a simulated spot termination signal:
POD_NAME=$(kubectl get pods -l app=spot-termination-handler -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -- touch /tmp/preempt_signal
Observation:
The DaemonSet detects the signal, immediately marks the spot worker as SchedulingDisabled, and initiates sequential pod eviction.
The PodDisruptionBudget maintains an active replica while the displaced replica spins up on the fallback on-demand node.
The traffic generator maintains continuous 200 OK responses with a 0.0% failure rate.
☁️ Production Cloud Migration
To run this in live production environments (AWS EKS or GCP GKE):
Metadata Endpoint Polling:
Replace the local file trigger in the DaemonSet with HTTP calls pointing to cloud provider metadata:
AWS IMDSv2:
TOKEN=$(curl -s -S -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/spot/instance-action
GCP Metadata:
curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/preempted
Production Image & GPU Limits:
Switch the mock inference container image to official vllm/vllm-openai:latest and declare explicit GPU limits (nvidia.com/gpu: 1).
📦 Source Code
All Kubernetes manifests, kind configuration files, and chaos testing scripts are available in the GitHub repository:
👉 github.com/riyankhamdani/spot-llm-k8s
Top comments (0)