DEV Community

Yash Sonawane
Yash Sonawane

Posted on

๐Ÿ’€ I Deleted My Kubernetes Podโ€ฆ And Lost My Database Data! Understanding Kubernetes Storage

Your Pod gets deleted. Kubernetes creates a replacement. Your application starts again. But where did your data go?

Imagine deploying a database on Kubernetes.

Everything works perfectly.

You create a database, insert 10,000 records, and verify that everything is running.

Then someone deletes the Pod.

kubectl delete pod mysql-pod
Enter fullscreen mode Exit fullscreen mode

Kubernetes creates a replacement Pod.

You connect to the database.

And suddenly...

Database is empty.
No records found.
Enter fullscreen mode Exit fullscreen mode

๐Ÿ˜ญ What just happened?

You assumed Kubernetes would automatically preserve your database data.

But here's the problem:

Containers are designed to be replaceable. Your data needs a storage strategy of its own.

Let's understand Kubernetes storage, PersistentVolumes, PersistentVolumeClaims, StorageClasses, and how to protect application data.


๐Ÿง  1. Why Kubernetes Needs Persistent Storage

Containers are generally designed to be disposable.

When a container is removed, its writable container filesystem does not automatically survive as persistent application storage.

Consider this:

        Kubernetes Pod
              |
              v
        Application
              |
              v
       Container Filesystem
              |
              v
        Container Deleted
              |
              v
      Writable Data Lost
Enter fullscreen mode Exit fullscreen mode

This creates a problem for stateful applications such as:

  • PostgreSQL
  • MySQL
  • MongoDB
  • Redis with persistence enabled
  • Applications that store uploaded files

These applications often need data to survive container restarts and Pod replacement.

That's where Kubernetes storage resources become important.

๐Ÿ’พ 2. What Is a PersistentVolume (PV)?

A PersistentVolume is a cluster resource representing storage available for use by workloads.

Think of it as storage that exists independently of an individual Pod.

Examples of possible storage backends include:

  • AWS EBS
  • Google Persistent Disk
  • Azure Disk
  • NFS
  • Local storage
  • CSI-compatible storage systems

A simplified architecture:

       Kubernetes Cluster
               |
               v
        PersistentVolume
               |
               v
        External Storage
               |
               v
          Application
Enter fullscreen mode Exit fullscreen mode

A PersistentVolume describes the storage resource, including its capacity, access modes, and storage configuration.

Example:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: demo-pv
spec:
  capacity:
    storage: 2Gi

  accessModes:
    - ReadWriteOnce

  persistentVolumeReclaimPolicy: Retain

  hostPath:
    path: /mnt/data
    type: DirectoryOrCreate
Enter fullscreen mode Exit fullscreen mode

Save it as pv.yaml.

Apply it:

kubectl apply -f pv.yaml
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pv
Enter fullscreen mode Exit fullscreen mode

Important This hostPath example is intended for local learning. It is not a production storage design. The data is tied to the node's filesystem and does not provide reliable cluster-wide storage.

๐Ÿ”— 3. What Is a PersistentVolumeClaim (PVC)?

Here's an important question.

Should every application developer need to understand the details of the underlying storage infrastructure?

Not necessarily.

That's why Kubernetes provides PersistentVolumeClaims.

A PVC is a request for storage.

The application requests storage, and Kubernetes attempts to find or provision suitable storage.

Think of it like this:

       Application
            |
            v
    PersistentVolumeClaim
            |
            v
     PersistentVolume
            |
            v
      Storage Backend
Enter fullscreen mode Exit fullscreen mode

Example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc
spec:
  accessModes:
    - ReadWriteOnce

  resources:
    requests:
      storage: 1Gi
Enter fullscreen mode Exit fullscreen mode

Save it as pvc.yaml.

Apply:

kubectl apply -f pvc.yaml
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pvc
Enter fullscreen mode Exit fullscreen mode

A PVC can remain Pending if no suitable PV exists and no compatible dynamic provisioner is available.

This is one of the most common Kubernetes storage troubleshooting scenarios.

๐Ÿ”ฅ 4. Connect Persistent Storage to a Pod

Now let's use the PVC inside a Pod.

Create pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: storage-test
spec:
  containers:
    - name: nginx
      image: nginx:1.27

      volumeMounts:
        - name: storage
          mountPath: /usr/share/nginx/html

  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: demo-pvc
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f pod.yaml
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Now create a test file inside the mounted volume:

kubectl exec storage-test -- sh -c \
  'echo "Kubernetes Persistent Storage Works!" > /usr/share/nginx/html/index.html'
Enter fullscreen mode Exit fullscreen mode

Read the file:

kubectl exec storage-test -- cat /usr/share/nginx/html/index.html
Enter fullscreen mode Exit fullscreen mode

You should see:

Kubernetes Persistent Storage Works!
Enter fullscreen mode Exit fullscreen mode

Now delete the Pod:

kubectl delete pod storage-test
Enter fullscreen mode Exit fullscreen mode

Create it again:

kubectl apply -f pod.yaml
Enter fullscreen mode Exit fullscreen mode

Read the file again:

kubectl exec storage-test -- cat /usr/share/nginx/html/index.html
Enter fullscreen mode Exit fullscreen mode

If the PVC remains bound to storage that survives Pod replacement, the file should still be present.

That's the purpose of persistent storage.

The Pod can be replaced without automatically destroying the separately managed storage.

โš™๏ธ 5. What Is a StorageClass?

Imagine you manage hundreds of applications.

Manually creating PersistentVolumes for every application would be difficult.

Kubernetes supports dynamic volume provisioning through StorageClasses.

A StorageClass describes how storage should be provisioned.

For example, a cloud provider may offer different storage classes for different performance or cost requirements.

A PVC can request a StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-pvc
spec:
  accessModes:
    - ReadWriteOnce

  storageClassName: standard

  resources:
    requests:
      storage: 10Gi
Enter fullscreen mode Exit fullscreen mode

When a compatible StorageClass and provisioner are configured, Kubernetes can dynamically provision storage for this claim.

Check available StorageClasses:

kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

Inspect one:

kubectl describe storageclass <storage-class-name>
Enter fullscreen mode Exit fullscreen mode

Remember that standard is only an example name. Your cluster may use a different StorageClass or may not have a default provisioner configured.

๐Ÿ†š 6. PV vs PVC vs StorageClass

This is an important comparison to remember.

Resource Meaning
PersistentVolume Represents available storage
PersistentVolumeClaim Requests storage
StorageClass Defines dynamic storage provisioning
VolumeMount Mounts storage into a container

A simple way to remember:

  • PV = Storage resource
  • PVC = Storage request
  • StorageClass = Storage provisioning mechanism

The relationship looks like this:

       StorageClass
            |
            v
     Dynamic Provisioning
            |
            v
    PersistentVolume
            |
            v
    PersistentVolumeClaim
            |
            v
          Pod
            |
            v
       Container
Enter fullscreen mode Exit fullscreen mode

๐Ÿšจ 7. What Happens When You Delete a PVC?

This is where things become particularly important.

Deleting a PVC does not have one universal outcome for the underlying storage.

The result depends on the PersistentVolume's reclaim policy and the storage provisioner.

Two important reclaim policies are:

Retain

The underlying storage is retained after the claim is released. An administrator may need to handle the volume manually before it can be reused.

Delete

The provisioner may delete the underlying storage asset when the claim is released.

Check the reclaim policy:

kubectl get pv
Enter fullscreen mode Exit fullscreen mode

Example:

NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY
demo-pv   2Gi        RWO            Retain
Enter fullscreen mode Exit fullscreen mode

For important workloads, understand your storage backend's behavior and configure backups independently.

A PersistentVolume is not a backup.

If someone deletes the storage or the underlying infrastructure fails, persistent storage alone may not protect your data.

โ˜ธ๏ธ 8. StatefulSet vs Deployment: Which One Should You Use?

Another important Kubernetes concept is the difference between Deployments and StatefulSets.

A Deployment is commonly used for stateless applications.

A StatefulSet is designed for applications that need stable identities and associated storage.

Feature Deployment StatefulSet
Pod identity Generally interchangeable Stable ordinal identity
Pod names Generated names Predictable names
Storage Can use PVCs Supports stable per-Pod PVCs
Typical use Web applications Stateful workloads
Scaling Replica-based Ordered scaling behavior

For example, a StatefulSet might create:

mysql-0
mysql-1
mysql-2
Enter fullscreen mode Exit fullscreen mode

Each Pod can have its own persistent storage claim.

Example StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
spec:
  serviceName: database
  replicas: 2

  selector:
    matchLabels:
      app: database

  template:
    metadata:
      labels:
        app: database

    spec:
      containers:
        - name: database
          image: mysql:8.4

          volumeMounts:
            - name: database-storage
              mountPath: /var/lib/mysql

  volumeClaimTemplates:
    - metadata:
        name: database-storage

      spec:
        accessModes:
          - ReadWriteOnce

        resources:
          requests:
            storage: 5Gi
Enter fullscreen mode Exit fullscreen mode

This example demonstrates the storage pattern, not a complete production MySQL configuration. A real deployment needs secure credentials, appropriate database configuration, backup procedures, and a properly configured storage backend.

๐Ÿ› ๏ธ 9. Kubernetes Storage Troubleshooting Commands

When storage stops working, these commands are extremely useful.

List PersistentVolumes:

kubectl get pv
Enter fullscreen mode Exit fullscreen mode

List PersistentVolumeClaims:

kubectl get pvc
Enter fullscreen mode Exit fullscreen mode

Describe a PVC:

kubectl describe pvc <pvc-name>
Enter fullscreen mode Exit fullscreen mode

List StorageClasses:

kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

Inspect Pod volume configuration:

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

Check namespace events:

kubectl get events --sort-by=.metadata.creationTimestamp
Enter fullscreen mode Exit fullscreen mode

Common problems include:

  • PVC stuck in Pending.
  • No matching PersistentVolume.
  • StorageClass not found.
  • Provisioning failures.
  • Volume attachment problems.
  • Permission errors when mounting storage.
  • Node or CSI driver issues.

A simple troubleshooting flow

PVC Not Working
      |
      v
Is PVC Bound?
      |
   No | Yes
      |   |
      v   v
Check PV  Check Pod
and SC    Events
      |   |
      v   v
Check     Check Volume
Provisioner Mount
      |
      v
Investigate Storage Backend
Enter fullscreen mode Exit fullscreen mode

Follow the evidence instead of randomly deleting storage resources.


๐Ÿงช 10. Your Weekend Challenge: Build a Kubernetes Storage Lab

Want to become confident with Kubernetes storage?

Build this project in Minikube or another local Kubernetes environment.

Project: Kubernetes Persistent Storage Lab

Your tasks:

  1. Create a PersistentVolume.
  2. Create a PersistentVolumeClaim.
  3. Mount the PVC into a Pod.
  4. Write a file into the mounted directory.
  5. Delete and recreate the Pod.
  6. Verify that the data remains.
  7. Test what happens when a PVC cannot be bound.
  8. Explore Retain and Delete reclaim policies in a disposable test environment.
  9. Create a StatefulSet with volumeClaimTemplates.

Suggested repository structure:

kubernetes-storage-lab/
โ”‚
โ”œโ”€โ”€ pv/
โ”‚   โ””โ”€โ”€ pv.yaml
โ”‚
โ”œโ”€โ”€ pvc/
โ”‚   โ””โ”€โ”€ pvc.yaml
โ”‚
โ”œโ”€โ”€ pods/
โ”‚   โ””โ”€โ”€ storage-test.yaml
โ”‚
โ”œโ”€โ”€ statefulset/
โ”‚   โ””โ”€โ”€ database.yaml
โ”‚
โ””โ”€โ”€ README.md
Enter fullscreen mode Exit fullscreen mode

Document your commands, screenshots, errors, and solutions.

This project will help you understand storage concepts much more deeply than simply reading definitions.


๐ŸŽฏ Final Thoughts

Kubernetes makes it easy to replace failed Pods and maintain application availability.

But your application data needs a different level of protection.

Understanding PersistentVolumes, PersistentVolumeClaims, StorageClasses, and StatefulSets is essential when working with stateful applications.

And remember these three lessons:

  1. Containers are replaceable.
  2. Persistent storage must be managed separately.
  3. Persistent storage is not a substitute for backups.

A successful Kubernetes deployment isn't just about getting Pods into the Running state. It's about ensuring your applications and their data behave correctly when failures happen.

๐Ÿ‘‡ Let's discuss!

Have you ever lost application data because of an incorrect Kubernetes storage configuration?

Which topic do you find most confusing?

  • PersistentVolumes and PVCs
  • StorageClasses
  • StatefulSets
  • Reclaim policies
  • CSI drivers

Share your experience in the comments!


๐Ÿ“˜ Preparing for the Certified Kubernetes Administrator (CKA) Exam?

Explore my CKA Complete Study Guide โ€” Certified Kubernetes Administrator:

โ˜ธ๏ธ Get the CKA Complete Study Guide

Build your Kubernetes foundation, strengthen your understanding of important concepts, and prepare with a structured learning resource.

Don't just memorize commands. Understand Kubernetes deeply. Practice consistently. Become confident.

Top comments (0)