Day 20: Persistent Storage in Kubernetes — Managing State, Weights and Vector Indices

#ai #kubernetes #30dayaiseries

Containers are ephemeral. When a Pod crashes or Kubernetes moves it, everything on its local disk is gone. For a web server that is fine. For AI it is costly: vector indices take time to rebuild, and model weights are many gigabytes that you do not want to download again every restart.

In plain terms: A container's disk is a whiteboard that gets wiped whenever the room is reassigned. A persistent volume is a filing cabinet in the corridor that every new room can open.

Diagram: A container's local filesystem is discarded when its Pod crashes or is rescheduled. Vector indices and multi-gigabyte model weights must outlive the Pod, so they need persistent volumes. See the animated version.

1. The storage abstraction

Kubernetes separates the storage from the application with two resources.

PersistentVolume (PV): a piece of real storage in the cluster, such as local NVMe disks, an NFS share or a SAN. It records the size, access modes and backing hardware.
PersistentVolumeClaim (PVC): the application engineer's request for storage. The Pod references the claim, and Kubernetes binds it to a matching volume. Many clusters also have a StorageClass that creates a volume on demand when a claim arrives.

Diagram: A PersistentVolume is a piece of real storage in the cluster. A PersistentVolumeClaim is the application's request for storage. The Pod references the claim, which binds to a matching volume, or a StorageClass can create one on demand. See the animated version.

2. Access modes for AI workloads

The access mode decides how Pods may use the volume.

ReadWriteOnce (RWO): mounted read-write by a single node. This is the usual choice for a stateful vector database such as Milvus or Qdrant, where uncoordinated writers could corrupt the index.
ReadOnlyMany (ROX): mounted read-only by many nodes at once. This suits model weights: keep one copy of, say, 15 GB of quantized weights on a shared, fast network mount, and let many stateless inference Pods attach to it instead of each holding its own copy.

Check that your storage backend actually supports the mode you ask for. Shared read-only mounts typically come from NFS or a similar network filesystem, while plain block disks are usually single-node.

Diagram: ReadWriteOnce mounts a volume read-write on a single node, which suits a vector database that must not have uncoordinated writers. ReadOnlyMany mounts read-only on many nodes at once, so one copy of the model weights can serve many inference Pods. The storage backend has to support the mode you request. See the animated version.

3. StatefulSets for databases

For a vector database, a standard Deployment is usually the wrong tool. A StatefulSet gives each Pod a stable name (db-0, db-1), starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That keeps replication between the database Pods consistent through cluster restarts.

Diagram: A StatefulSet gives each Pod a stable name such as db-0 and db-1, starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That is why stateful databases use StatefulSets instead of Deployments. See the animated version.

Here is an example. It assumes your cluster has a default StorageClass, and the Qdrant image and sizes are illustrative.

# 1. Shared, read-only model weights (needs a backend that supports ReadOnlyMany)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-weights
spec:
  accessModes: [ReadOnlyMany]
  resources:
    requests: { storage: 20Gi }
---
# 2. A vector database with its own volume per Pod
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: qdrant
spec:
  serviceName: qdrant
  replicas: 1
  selector:
    matchLabels: { app: qdrant }
  template:
    metadata:
      labels: { app: qdrant }
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.9.0        # pin a version you have tested
          volumeMounts:
            - { name: data, mountPath: /qdrant/storage }
  volumeClaimTemplates:                       # one PVC is created per Pod: data-qdrant-0, ...
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests: { storage: 50Gi }

Done well, this eliminates repeated weight downloads, preserves your database indices and protects the system's memory across cluster updates.

Coming Up Next

Day 21: High-performance networking: ingress and egress for AI clusters.

KubernetesStorage #DataEngineering #CloudNative #VectorDatabase #StatefulSet

Originally published at https://sureshpallapothu.in/blog/day-20-kubernetes-storage, where this post includes animated diagrams.

DEV Community