Why Disk Space Can Stay Full After a Huge Log File Is Deleted

A server runs out of disk space. The largest log file is found, deleted, and confirmed absent from its directory. Yet df still reports the filesystem as full.

The common misconception is simple: deleting a file immediately releases all of its disk space.

On Linux, deletion usually removes a directory entry—a name that points to a file’s inode. The file’s stored data can be reclaimed only when two conditions are satisfied:

1. No directory entries refer to the inode.
2. No process still has the file open.

If a running application opened a log file before it was deleted, its file descriptor remains connected to the same inode. The pathname may have disappeared, but the underlying file still exists for that process. The application can continue writing to it, and the supposedly deleted file may even keep growing.

This behavior is intentional. Open file descriptors remain valid when files are renamed or unlinked, allowing programs to use files safely without depending on their names remaining unchanged.

The mismatch between common disk tools makes the problem especially confusing. du walks the directory tree and totals files it can reach by pathname. An unlinked file has no pathname, so du cannot see it. df, by contrast, reports allocated filesystem blocks. Those blocks are still allocated while a process holds the file open. As a result, du may account for far less data than df says is in use.

The usual diagnostic command is:

lsof +L1

This lists open files whose link count is below one, including files marked (deleted). The process name, PID, file descriptor, and file size help identify which application is retaining the space.

The safest remedy is generally to make that application close and reopen the file. Depending on the program, that may mean requesting a log reopen, performing a graceful reload, restarting a service, or restarting a container. Once the final open file descriptor is closed, the kernel can reclaim the blocks.

Killing a process is not always necessary. Many daemons support log rotation explicitly: the old log is renamed, a signal or reload command asks the daemon to reopen its log path, and the rotated file is then compressed or removed. Without the reopen step, log rotation can leave the daemon writing through its old descriptor.

There is also an emergency technique: truncating the open descriptor through /proc/<pid>/fd/<fd>. That can release space without terminating the process, but it should be used cautiously. Writing to another process’s file descriptor can interact badly with buffered output, file offsets, or application assumptions.

The key distinction is between a filename and an open file. rm removes the filename. Storage is reclaimed only after the filesystem has neither a remaining name nor a process reference to the file.