DEV Community

Big Mazzy
Big Mazzy

Posted on

Fix No Space Left on Device on Ubuntu 24.04 Server

#linux #ubuntu #tutorial #devops

TL;DR: Find the known oversized leftover file on the full lab volume, remove it, then retry the write and check that free space has returned.

The problem

A write can fail with “No space left on device” even when you’re working on a separate lab volume rather than the server’s root disk. In this example, the lab volume is 64 MB. Filling it with a large file leaves no room for a normal write.

The failed write is a symptom, not a diagnosis. First confirm which filesystem is full, then identify what is using its space. The commands below follow that path and remove only the known lab leftover.

Why it happens

Filesystems track both data capacity and inodes. Bytes measure the space used by file contents; inodes track how many files and directories can be created. Either limit can prevent a new file from being written.

Here, the volume is at 100% usage with zero bytes available. The large file accounts for 57 MB. The inode limit is fine: this is a bytes-full problem, not an inode shortage. The lost+found permission message is separate; it means the current user cannot read that directory, and does not prevent du from reporting the large file.

Step-by-step fix

  1. Reproduce the failure by writing zeroes to the lab file. The command stops when the volume runs out of space: 
dd if=/dev/zero of=/srv/lab/export.csv bs=1M
Enter fullscreen mode Exit fullscreen mode

Example error:

dd: error writing '/srv/lab/export.csv': No space left on device
Enter fullscreen mode Exit fullscreen mode
  1. Try a normal write so you can see whether the filesystem also rejects a small file: 
echo test | tee /srv/lab/new.txt
Enter fullscreen mode Exit fullscreen mode

Example error:

tee: /srv/lab/new.txt: No space left on device
Enter fullscreen mode Exit fullscreen mode
  1. Check the capacity of the lab mount: 
df -h /srv/lab
Enter fullscreen mode Exit fullscreen mode

Example output:

Filesystem Size Used Avail Use% Mounted on
/dev/loop0 59M 59M 0 100% /srv/lab
Enter fullscreen mode Exit fullscreen mode
  1. Find the large entry under the lab directory: 
du -sh /srv/lab/*
Enter fullscreen mode Exit fullscreen mode

Example output:

57M /srv/lab/export.csv
du: cannot read directory '/srv/lab/lost+found': Permission denied
Enter fullscreen mode Exit fullscreen mode
  1. Remove the known lab leftover: 
rm /srv/lab/export.csv
Enter fullscreen mode Exit fullscreen mode

The target is specific: /srv/lab/export.csv. Check that you have identified the intended file before removing anything on a real server.

How to verify

Retry the same small write, then check the mount’s capacity again:

echo test | tee /srv/lab/new.txt
Enter fullscreen mode Exit fullscreen mode 
df -h /srv/lab
Enter fullscreen mode Exit fullscreen mode

The write should print test. The filesystem should show available space again:

test
/dev/loop0 59M 2.0M 57M 4% /srv/lab
Enter fullscreen mode Exit fullscreen mode

The IP, PID, and session numbers used in examples are placeholders where relevant; this walkthrough’s sample output does not include them. On a real server, confirm the mount and the file before removing anything, since deleting the wrong file can disrupt a service or remove needed data.

Common mistakes

  • Treating every “No space left on device” error as an inode problem. Check filesystem usage first; in this case, bytes are full and inodes are fine.
  • Assuming the root disk is full. The example concerns /srv/lab, a separate volume, so inspect the filesystem that contains the affected path.
  • Deleting a large file just because it appears in a usage report. Confirm it is a known leftover before removing it.
  • Misreading the lost+found permission message as the cause. It only reports that du could not read that directory.
  • Expecting the original large file to remain after removal. The verification checks a new write and confirms that space is available again.

If you want a clean Ubuntu box to practice on, PowerVPS is a reasonable choice: https://powervps.net/?from=32

Watch more fixes on Server Lab: https://www.youtube.com/channel/UCov7rfWAnS4c2wF5E9V1Yag

Top comments (0)