DEV Community

Cover image for Incident note: No space left on device despite free space
Daniele Michelin
Daniele Michelin

Posted on Originally published at daniele-michelin.com

Incident note: No space left on device despite free space

#linux #sysadmin #troubleshooting

Read the signal

Start with the exact wording of the error and its timestamp in the log. Then narrow down the cause before changing anything.

cp: cannot create regular file '/var/www/nextcloud/data/upload.tmp': No space left on device
Enter fullscreen mode Exit fullscreen mode

Narrow down possible causes

  • All inodes used up by millions of small files (PHP sessions, cache, mail queue, Maildir).
  • Deleted but still open files (e.g. logs) keep using space until the process closes them.
  • Blocks reserved for root (5% by default on ext4): regular users already see the disk as full.
  • A different filesystem is full, e.g. /tmp on tmpfs or a Docker volume.

Check before changing anything

1.

Space per filesystem

df -h
Enter fullscreen mode Exit fullscreen mode

2.

Inode usage per filesystem

df -i
Enter fullscreen mode Exit fullscreen mode

3.

Find deleted files that are still open

sudo lsof +L1
Enter fullscreen mode Exit fullscreen mode

4.

Find directories with the most files

sudo du --inodes -x /var 2>/dev/null | sort -n | tail -n 15
Enter fullscreen mode Exit fullscreen mode

Fix and test again

  1. Delete masses of small files in a targeted way, e.g. old sessions: find /var/lib/php/sessions -type f -mmin +1440 -delete.
  2. Restart the process holding a deleted file, or truncate it via : > /proc/<PID>/fd/<FD>.
  3. Configure logrotate properly (postrotate with reload or copytruncate) so open logs are not deleted.
  4. On pure data disks, lower the reservation: tune2fs -m 1 /dev/sdb1.

Then repeat the original action and compare the result with the initial error.

Keep a note for the next ticket

Record the exact error, the check result, the change and the retest. That gives the team a useful starting point next time.

Can I increase the inode count afterwards?

Not on ext4 without reformatting. XFS and Btrfs allocate inodes dynamically. Long term, remove the cause of all the small files.

Full error guide: No space left on device although df shows free space

Originally published at daniele-michelin.com. More guides, free checks (SPF, TLS, headers) and an error-message lexicon: daniele-michelin.com/en/.

Top comments (0)