DEV Community

Muhammad Awais Zahid
Muhammad Awais Zahid

Posted on

How I Fixed a 100% Disk Usage Issue Caused by Stuck etcd Backups in Kubernetes

#kubernetes #etcd #linux #devops

The Incident: When a Routine Backup Fills the Disk

In DevOps, some of the most challenging incidents begin with something that looks simple.

One day, while working on our Kubernetes infrastructure, I investigated a master node whose root filesystem had reached 100% disk usage.

At first, it looked like a typical disk-space problem. Maybe logs had grown too large, container images were consuming storage, or some application was generating unexpected data.

But after investigating the filesystem, I discovered that the problem was related to something we depend on for disaster recovery: etcd backups.

Our backup process was generating large snapshots, but multiple etcdctl snapshot save processes had become stuck. Meanwhile, completed backup files had accumulated over time without an effective retention policy.

The result was a nearly full root filesystem and a backup process that needed to be redesigned.

In this article, I'll walk through how I investigated the issue, identified the root cause, recovered disk space, and improved the backup automation.

1. Understanding the Environment

Our Kubernetes environment consisted of five master nodes.

The affected node was:

k8s-master-node-5
Enter fullscreen mode Exit fullscreen mode

It was being used to store etcd snapshots in:

/home/k8-node5/etcd-backups
Enter fullscreen mode Exit fullscreen mode

Each completed snapshot occupied approximately 693 MB, according to the snapshot output. The backup job was scheduled through root's crontab to run daily at 3:00 AM.

The original cron job looked like this:

00 03 * * * /usr/bin/etcdctl \
  --endpoints=https://<ip>:2379 \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  snapshot save /home/k8-node5/etcd-backups/etcd-bkp-$(date +\%F-\%H\%M\%S).db \
  >> /home/k8-node5/etcd-backups/etcd-backup-log-$(date +\%F).log 2>&1
Enter fullscreen mode Exit fullscreen mode

The job had a schedule, but it lacked important safeguards against long-running or overlapping backup operations.

2. Checking Disk Usage

The first step was to confirm the state of the filesystem.

df -h
Enter fullscreen mode Exit fullscreen mode

The output showed that the root filesystem was completely full:

Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  194G  194G     0  100% /
Enter fullscreen mode Exit fullscreen mode

When a Linux root filesystem reaches 100%, normal operations can start failing. Commands may be unable to create temporary files, services may have trouble writing logs, and scheduled jobs may fail.

I needed to identify which directory was consuming the space.

I ran:

sudo du -xhd1 /
Enter fullscreen mode Exit fullscreen mode

The results showed approximately:

/home    180G
/var     7.2G
/usr     2.9G
Enter fullscreen mode Exit fullscreen mode

This immediately narrowed the investigation to /home.

Rather than deleting files blindly, I continued investigating the directory responsible for most of the usage.

3. Finding the Actual Cause

I inspected the etcd backup directory:

sudo du -sh /home/k8-node5/etcd-backups
sudo ls -lh /home/k8-node5/etcd-backups
Enter fullscreen mode Exit fullscreen mode

The directory contained hundreds of completed snapshot files dating back to previous months. It also contained numerous empty .db.part files.

A .db.part file is a temporary file used while a snapshot is being written. Its presence alone does not prove that a backup is stuck, but in this case it was one of the clues.

Next, I checked the running processes for backup commands.

I found multiple etcdctl snapshot save processes associated with the backup directory.

This revealed two problems:

  • Stuck backup processes: Some snapshot operations were not completing normally.

  • Uncontrolled backup retention: Old snapshots continued accumulating, consuming disk space.

The original backup command did not enforce a maximum execution time or prevent overlapping runs.

That meant a backup could remain active while another scheduled run attempted to start.

The problem was no longer just disk capacity. It was a reliability issue in the backup automation.

4. Stopping Stuck Processes Safely

Before cleaning up files, I needed to stop the stuck backup operations.

I used a targeted command to send a termination signal only to matching backup processes:

sudo pkill -TERM -f '^/usr/bin/etcdctl .* snapshot save /home/k8-node5/etcd-backups/'
Enter fullscreen mode Exit fullscreen mode

Then I checked whether any matching processes remained:

pgrep -af 'etcdctl.*snapshot save'
Enter fullscreen mode Exit fullscreen mode

I verified that the old backup jobs were no longer running before removing the empty partial files.

This order matters. Deleting a temporary snapshot file while its process is still writing to it is not a safe cleanup strategy.

5. Recovering Disk Space

With the stuck backup operations stopped, I inspected the backup directory again.

I identified older completed snapshots that could be removed while preserving a recent set of backups.

I removed the selected older snapshots rather than deleting Kubernetes runtime directories or system data.

After cleanup, I checked the filesystem:

df -h /
Enter fullscreen mode Exit fullscreen mode

The result was a major improvement:

Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  194G   20G  165G   11% /
Enter fullscreen mode Exit fullscreen mode

Disk usage dropped from 100% to approximately 11%, restoring around 165 GB of available space.

This resolved the immediate storage pressure.

However, deleting old files was only the recovery step. Without fixing the backup process, the same issue could happen again.

6. Replacing the Cron Command With a Safer Script

I moved the backup logic into a dedicated shell script:

/usr/local/sbin/etcd-backup.sh
Enter fullscreen mode Exit fullscreen mode

This made the process easier to maintain and gave me a place to implement safeguards.

A. Adding a timeout

A backup process should not run indefinitely.

I used timeout to impose a 30-minute execution limit:

/usr/bin/timeout 30m /usr/bin/etcdctl \
  --endpoints="$ENDPOINT" \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  snapshot save "$BACKUP"
Enter fullscreen mode Exit fullscreen mode

If the operation exceeds the limit, it is terminated and the script records the failure.

The timeout should be chosen according to snapshot size, cluster activity, and observed backup duration.

B. Preventing overlapping executions

I added a lock using flock:

exec 9>/run/lock/etcd-backup.lock
/usr/bin/flock -n 9 || exit 0
Enter fullscreen mode Exit fullscreen mode

This prevents another instance of the script from starting a concurrent backup while the existing instance holds the lock.

For scheduled jobs, this is an important safeguard because overlapping operations can increase resource usage and make failures harder to diagnose.

C. Introducing a retention policy

Keeping every snapshot forever is not a sustainable backup strategy.

I introduced a retention policy to keep the seven newest completed, timestamped daily snapshots and remove older ones after a successful backup and snapshot verification.

This keeps storage usage under control.

Important: Retention logic must be tested carefully before production use. A backup should not be deleted simply because a command returned success; verification should happen first, and the retention filter should exclude manual or specially protected backups.

D. Improving logging

I configured the script to redirect standard output and errors to a daily log:

LOG="$BACKUP_DIR/etcd-backup-log-$(date +%F).log"
exec >> "$LOG" 2>&1
Enter fullscreen mode Exit fullscreen mode

This makes it easier to investigate backup failures without relying on terminal output.

The log records when a backup starts, whether it completes, and whether verification succeeds.

7. Updating the Cron Schedule

After creating the script, I updated root's crontab to run it daily at 3:00 AM:

0 3 * * * /usr/local/sbin/etcd-backup.sh
Enter fullscreen mode Exit fullscreen mode

I commented out the old direct etcdctl cron entry so that both mechanisms would not run independently.

I also checked the cron service:

sudo systemctl is-active cron
Enter fullscreen mode Exit fullscreen mode

The service was active.

8. Testing the Backup Instead of Assuming It Works

A scheduled job is not proof that a backup works. I manually executed the new script:

sudo /usr/local/sbin/etcd-backup.sh
Enter fullscreen mode Exit fullscreen mode

Then I inspected the log:

sudo tail -30 /home/k8-node5/etcd-backups/etcd-backup-log-$(date +%F).log
Enter fullscreen mode Exit fullscreen mode

The log showed that the snapshot completed in approximately 13 seconds.

The saved snapshot was approximately 693 MB.

I then checked the snapshot status:

sudo /usr/bin/etcdctl snapshot status \
  /home/k8-node5/etcd-backups/etcd-bkp-2026-10-23-122933.db \
  --write-out=table
Enter fullscreen mode Exit fullscreen mode

The output showed:

+----------+------------+------------+------------+
|   HASH   |  REVISION  | TOTAL KEYS | TOTAL SIZE |
+----------+------------+------------+------------+
| 8e5fa6c7 | 1352100800 |      16829 |     693 MB |
+----------+------------+------------+------------+
Enter fullscreen mode Exit fullscreen mode

This confirmed that etcdctl could read the snapshot metadata and report its revision, hash, key count, and size.

I also verified the endpoint's health:

sudo /usr/bin/etcdctl \
  --endpoints=https://192.168.18.17:2379 \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  endpoint health
Enter fullscreen mode Exit fullscreen mode

The endpoint returned a healthy status.

At this point, the new backup had been created and its snapshot status verified. A full restore test would still be needed to demonstrate end-to-end recoverability.

9. Extending the Approach to Master Node 4

After addressing the problem on master node 5, I was asked to apply the same backup approach to master node 4.

I did not simply copy the configuration without checking the second node. Each node's etcd endpoint and certificate configuration must be confirmed independently.

First, I checked the IP addresses:

hostname -I
Enter fullscreen mode Exit fullscreen mode

The primary address was:

192.168.*.*
Enter fullscreen mode Exit fullscreen mode

Then I inspected the etcd manifest:

sudo grep -E 'listen-client-urls|advertise-client-urls|cert-file|key-file|trusted-ca-file' \
  /etc/kubernetes/manifests/etcd.yaml
Enter fullscreen mode Exit fullscreen mode

This confirmed that node 4's advertised client endpoint was:
https://192.168.*.*:2379

I also checked the filesystem:
df -h /
Node 4 had approximately 74 GB available and was using around 20% of its root filesystem.

Finally, I checked its root crontab:
sudo crontab -l

The next step was to configure a dedicated backup directory and script using node 4's own endpoint and certificates, following the same timeout, locking, logging, verification, and retention principles.

The node 4 setup should only be considered complete after its backup has been created, verified, and its daily schedule confirmed.

10. What I Learned From This Incident

This incident reinforced several practical lessons for me as a DevOps engineer.

Disk usage needs investigation, not guesswork

Checking df -h tells us that a filesystem is full. Using du helps identify where the space is going. Inspecting processes and files then helps uncover why.

Cron jobs need operational safeguards

A cron entry only schedules a command. It does not automatically prevent hangs, overlapping executions, or uncontrolled storage growth.

Backup retention is part of backup design

A backup strategy needs a defined retention policy. Otherwise, the storage used by backups can eventually threaten the same infrastructure those backups are intended to protect.

*A successful command is not the same as a tested recovery
*
Snapshot-status verification is useful, but a separate restore test is needed to confirm that the backup can support disaster recovery.

Local backups are not enough

Storing etcd snapshots on the same node protects against some data-loss scenarios, but not against every server or storage failure. Verified snapshots should also be copied to a separate, secure destination.

Conclusion

What initially appeared to be a disk-space problem turned out to be a reliability issue in the etcd backup process.

By checking filesystem usage, identifying stuck snapshot processes, safely removing old backup files, and introducing timeout, locking, retention, and verification safeguards, I addressed the immediate incident and improved the backup workflow.

I then began applying the same approach to master node 4, using its own verified etcd endpoint rather than blindly reusing node 5's configuration.

The biggest takeaway is simple: a reliable backup system is more than a scheduled command. It needs monitoring, failure handling, verification, retention, and a tested recovery plan.

Top comments (0)