Resizing Persistent Volumes in Kubernetes like Magic

Kubernetes Disk Expansion

So your Workload's Persistent Volume is almost full. How can you expand it? With Kubernetes 1.11 and above, this can now easily be done by just updating the Persistent Volume Claim storage specification.

My example in this post is tested in Google's Kubernetes Engine.

Let's get started!

Enabling Volume Expansion in Storage Class

To enable the disk resizing feature, ensure that the Storage Class that your Workload's Persistent Volume Claim uses is configured with allowVolumeExpansion: true.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
parameters:
  type: pd-standard
provisioner: kubernetes.io/gce-pd
allowVolumeExpansion: true # ensure that this is true

A Side Note for StatefulSet Workloads

If you want to resize a disk for a StatefulSet with a configured volumeClaimTemplates like the following,

  volumeClaimTemplates:
  - metadata:
      name: postgresql-storage
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: standard

you will only have to modify the Persistent Volume Claim that it has provisioned.

Resizing a Persistent Volume Claim

Find the Persistent Volume Claim to resize.

kubectl get pvc -l app=postgresql

Let's say you want to update postgresql-storage-postgresql-02-1 from 1Gi to 10Gi

NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgresql-storage-postgresql-01-0   Bound    pvc-613c74c3-66b5-11e9-a602-42010a8400c0   10Gi       RWO            standard       6m12s
postgresql-storage-postgresql-01-1   Bound    pvc-88f897b1-60d4-11e9-a602-42010a8400c0   5Gi        RWO            standard       7d11h
postgresql-storage-postgresql-02-0   Bound    pvc-727c167c-6089-11e9-a602-42010a8400c0   10Gi       RWO            standard       7d20h
postgresql-storage-postgresql-02-1   Bound    pvc-80421dab-6089-11e9-a602-42010a8400c0   1Gi        RWO            standard       7d20h

kubectl edit pvc postgresql-storage-postgresql-02-1

From your text editor, change 1Gi to 10Gi

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
  creationTimestamp: "2019-04-16T20:52:09Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: postgresql
  name: postgresql-storage-postgresql-02-1
  namespace: default
  resourceVersion: "29892621"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/postgresql-storage-postgresql-02-1
  uid: 80421dab-6089-11e9-a602-42010a8400c0
spec:
  accessModes:
  - ReadWriteOnce
  dataSource: null
  resources:
    requests:
      storage: 1Gi # change this to 10Gi
  storageClassName: standard
  volumeName: pvc-80421dab-6089-11e9-a602-42010a8400c0
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi # change this to 10Gi
  phase: Bound

After successfully modifying the Persistent Volume Claim, observe its status.

kubectl get pvc postgresql-storage-postgresql-02-1 -o yaml

This should be the example output. Observe the status at the very end of the YAML.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
  creationTimestamp: "2019-04-16T20:52:09Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: postgresql
  name: postgresql-storage-postgresql-02-1
  namespace: default
  resourceVersion: "33540657"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/postgresql-storage-postgresql-02-1
  uid: 80421dab-6089-11e9-a602-42010a8400c0
spec:
  accessModes:
  - ReadWriteOnce
  dataSource: null
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
  volumeName: pvc-80421dab-6089-11e9-a602-42010a8400c0
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-04-24T17:30:51Z"
    message: Waiting for user to (re-)start a pod to finish file system resize of
      volume on node.
    status: "True"
    type: FileSystemResizePending
  phase: Bound

Restart the Pod

The next step is to restart the Pod as recommended by Kubernetes.

kubectl delete pod postgresql-02-1

Then wait until the Pod starts running again.

Observe the Changes

After a successful Pod restart, check the status of the Persistent Volume Claim again and you should see that the storage has been updated.

kubectl get pvc postgresql-storage-postgresql-02-1 -o yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
  creationTimestamp: "2019-04-16T20:52:09Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: postgresql
  name: postgresql-storage-postgresql-02-1
  namespace: default
  resourceVersion: "33541444"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/postgresql-storage-postgresql-02-1
  uid: 80421dab-6089-11e9-a602-42010a8400c0
spec:
  accessModes:
  - ReadWriteOnce
  dataSource: null
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
  volumeName: pvc-80421dab-6089-11e9-a602-42010a8400c0
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  phase: Bound

Do kubectl exec to the Pod and verify that the disk has been resized!

Run df -h.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         5.7G  4.7G  1.1G  83% /
tmpfs           1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1       5.7G  4.7G  1.1G  83% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/sdc        9.9G  1.3G  8.6G  13% /var/lib/postgresql
tmpfs           1.9G   12K  1.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           1.9G     0  1.9G   0% /sys/firmware

Magic!