DEV Community

Suresh Kumar Pallapothu
Suresh Kumar Pallapothu

Posted on Originally published at sureshpallapothu.in

Day 14: Week 2 Capstone — Deploying a Highly Available Local AI Application on K3d

This capstone ties Week 2 together. You will run a small but realistic platform on one machine: a multi-node cluster, persistent storage, a clustered vector database, scalable inference and a Helm release, then break it on purpose to see it heal.

The architecture

Diagram: FIG 1 · Traffic enters through the k3d proxy and ingress, reaches a FastAPI orchestrator, which calls Qdrant for embeddings and vLLM for tokens. Each tier has its own storage. See the animated version.

Traffic enters through the k3d proxy on port 80 and reaches an ingress controller. The ingress sends it to the ai-gateway-service, a ClusterIP in front of a FastAPI orchestrator. The orchestrator calls the Qdrant StatefulSet (with its headless service and a dedicated PVC per pod) for embeddings, and the vLLM Deployment (with a ReadOnlyMany shared-weights volume) for tokens.

One choice to reconcile: this diagram uses an ingress controller, so do not disable Traefik for the capstone. Day 8 disabled it to make room for Envoy later, but here we use K3s's built-in Traefik so the stack works end to end.

The steps

Diagram: FIG 2 · Create the cluster, prepare storage, install the Helm chart, then verify the pods. Keep the ingress controller enabled if your architecture depends on it. See the animated version.

# 1. Create the cluster (keep the built-in Traefik this time)
mkdir -p ~/ai-storage/qdrant ~/ai-storage/models
k3d cluster create ai-capstone --servers 1 --agents 2 \
  -p "80:80@loadbalancer" \
  --volume ~/ai-storage/models:/opt/ai/models@agent:*

# 2. Install the chart into its own namespace
helm upgrade --install enterprise-ai ./helm-ai-stack \
  --set global.namespace=ai-system --create-namespace \
  -f ./helm-ai-stack/values-staging.yaml

# 3. Check where everything landed
kubectl get nodes -o wide
kubectl get pods -n ai-system -o wide
kubectl get pvc -n ai-system
Enter fullscreen mode Exit fullscreen mode

A note on permissions. You will often see sudo chmod -R 777 in tutorials to get a volume working. That is a lab shortcut that makes a folder writable by everyone, and you should not carry it into production. Prefer creating the folders as your own user, as above, or set the right owner and a tight mode such as 755.

If your machine has no GPU, replace the vLLM image in the chart with an Ollama-based one. The manifests keep the same shape, and you can still exercise every Kubernetes concept in the lesson.

Break it on purpose

Fault injection is how you learn whether the design holds.

kubectl delete pod -n ai-system -l app=local-ai-llm --now
kubectl get pods -n ai-system -w
Enter fullscreen mode Exit fullscreen mode

Diagram: FIG 3 · Deleting a model pod triggers an immediate replacement. Because the weights live on a shared volume, the new pod only has to load them, so it recovers far faster than it would after a re-download. See the animated version.

The Deployment controller notices the missing pod and creates a replacement at once. Because the weights live on a ReadOnlyMany volume that is already on the nodes, the new pod does not re-download anything. It passes its startup probe as soon as the model has loaded, which is much quicker than a fresh download.

Verify the whole system

Diagram: FIG 4 · Verify nodes, pod placement, volume binding, database clustering, autoscaling and recovery. On a machine without a GPU, run a CPU-friendly engine such as Ollama in place of vLLM. See the animated version.

  1. Nodes show Ready.
  2. Pods are spread across the agents, with Qdrant and vLLM on different nodes.
  3. PVCs show Bound.
  4. Qdrant peers can see each other (check the cluster endpoint on port 6333).
  5. The HPA can read its metric.
  6. Deleting a pod recovers, and you can time how long that takes.

That completes Week 2. You now have a cluster you can create in seconds, storage that outlives pods, a database that keeps its identity, inference that scales on the right signal and a Helm chart that ships it all. Week 3 turns to the network and security layer: eBPF, Cilium and service meshes.

Run it yourself

This is the exact example from the video. Run against a real K3d cluster. A small nginx image stands in for the model server, and the video deletes one of the two pods and counts what comes back.

apiVersion: apps/v1
kind: Deployment
metadata: {name: llm}
spec:
  replicas: 2
  selector: {matchLabels: {app: llm}}
  template:
    metadata: {labels: {app: llm}}
    spec:
      containers:
        - name: llm
          image: nginx:1.27-alpine
          readinessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 1
Enter fullscreen mode Exit fullscreen mode

Run it:

kubectl apply -f deploy.yaml
kubectl delete pod <one of the two pods>
kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Output I got while recording the video:

before: 2/2 ready
deleted: 1 pod
after: 2/2 ready
new pods created: 1
deleted pod gone: yes
Enter fullscreen mode Exit fullscreen mode

Coming Up Next

Day 15: AI network bottlenecks: moving from iptables to eBPF.

Kubernetes #MLOps #SystemDesign #CapStone #CloudNative #DevOps #DataScience #AIEngineering


Originally published at https://sureshpallapothu.in/blog/ai-int-day-14-week-2-capstone, where this post includes animated diagrams.

Top comments (0)