Your root partition is at 100%, you ran rm on a 50GB log file, and df still reports zero free bytes. Here is how inodes, unlinked file descriptors, and kernel references keep ghost data alive, and how to recover disk space without restarting production.
A critical disk alert fires on your phone.
A busy database node or API host just crossed 99% disk capacity on /var. Applications begin failing. MySQL refuses incoming transactions with disk-full errors. Docker containers start getting evicted by the kubelet. System loggers choke on write operations.
You SSH into the struggling server and run a quick scan to find the space hog:
ls -lhS /var/log/app/
-rw-r--r-- 1 appuser appuser 52G Oct 16 19:40 debug.log
-rw-r--r-- 1 appuser appuser 1.2G Oct 16 00:00 access.log
-rw-r--r-- 1 appuser appuser 210M Oct 15 00:00 app.log.1
A rogue debug flag was left enabled in production, and debug.log has blown up to 52 gigabytes.
Relieved that you found the culprit so fast, you run the obvious command:
rm -f /var/log/app/debug.log
The shell prompt returns instantly. You run ls -lh /var/log/app/debug.log to make sure it is gone. The terminal confirms the file no longer exists:
ls: cannot access '/var/log/app/debug.log': No such file or directory
Problem solved, right? You run df -h to verify how much breathing room you just recovered:
df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 60G 60G 0 100% /var
The partition is still at 100% capacity. Available space is still 0 bytes.
Confused, you run du to inspect actual directory usage:
du -sh /var
4.8G /var
du says /var is only consuming 4.8 gigabytes. df insists all 60 gigabytes are completely full.
Where did the other 55 gigabytes go? Did the filesystem corrupt itself? Is the storage hardware failing?
Neither. You just walked straight into one of the most common traps in Linux systems administration: the open unlinked file descriptor.
rm Does Not Delete Files
To understand where your disk space went, you have to unlearn what the rm command actually does.
On Linux, rm does not mean "delete file". It stands for remove, which invokes the unlink() or unlinkat() system call inside the Linux kernel.
Unix filesystems like ext4, XFS, and btrfs do not store file data inside filenames or directories. Storage architecture in Linux is split into three separate layers:
First, Directory Entries (dentries). A directory in Linux is simply a special structured file that maps human-readable names to numbers. When you look at /var/log/app/debug.log, the string debug.log is just a pointer sitting in a directory table. That pointer references an integer called an inode number.
Second, Inodes (Index Nodes). The inode is the actual metadata structure of the file. It stores file ownership (UID and GID), access permissions, file size, modification timestamps, and pointers to the physical data blocks or extents on the storage drive. Crucially, the inode also stores an internal counter named i_nlink (the hard link count). This tracks how many directory entries point to this specific inode.
Third, Data Blocks. These are the raw 4KB blocks on the physical drive where the file contents live.
When you execute rm -f /var/log/app/debug.log, the kernel only performs two specific actions:
// Simplified kernel logic during unlink()
unlink("/var/log/app/debug.log");
// 1. Remove the dentry "debug.log" from the directory
// 2. Decrement the inode link counter: inode->i_nlink--;
The directory entry is erased, so ls cannot find it. The i_nlink counter drops from 1 to 0.
However, the kernel will not free the allocated data blocks on disk back to the filesystem free block bitmap just because i_nlink reached zero.
The kernel tracks a second reference counter: the open file table references.
Whenever an active process opens a file with the open() system call, the kernel creates a struct file entry in its internal system-wide open file table and increments an in-memory reference counter (f_count).
The storage driver adheres to a strict rule:
Disk blocks and inode allocations are only released when both conditions are met:
- The hard link count reaches zero (
i_nlink == 0). - Every active process closes its file descriptor pointing to that inode (
f_count == 0).
If an application (like your Node.js backend, a Java worker, Python daemon, or Nginx) opened /var/log/app/debug.log when it booted up and is still running, that process holds an active file descriptor pointing directly to the inode.
When you ran rm, you severed the directory entry. But the application still holds the open file handle.
The inode remains alive in kernel memory. The data blocks remain marked as active and allocated on your NVMe or SSD drive. The process keeps writing fresh bytes into those unlinked blocks.
You did not delete the file. You only removed its name from the phone book while the phone call was still active.
The Feud Between df and du
Now the conflicting numbers between df and du make complete sense.
These two utilities measure disk space from two completely different perspectives in the kernel.
du stands for Disk Usage. It is a directory tree crawler. When you run du -sh /var, the utility executes opendir("/var") and starts walking through every subdirectory with readdir(). For every filename it encounters, it calls the lstat() or stat() system call to ask the filesystem for the block count allocated to that path.
Because you ran rm, the directory entry for debug.log is gone. There is no filename for du to discover. It skips right past it. As far as du is concerned, those 52 gigabytes do not exist because they have no presence in the directory hierarchy.
df stands for Disk Free. It does not look at directories, filenames, or paths at all.
When you run df -h /var, df issues a single system call directly against the filesystem: statvfs() or statfs().
struct statvfs buf;
statvfs("/var", &buf);
// Reads total blocks, free blocks, and available blocks directly from the superblock
This system call queries the filesystem superblock and block group allocation bitmaps in kernel memory. The superblock tracks the real physical state of the storage partition:
- Total block count on the partition
- Total allocated blocks currently marked as "in use" in the allocation bitmap
- Total free blocks available for new writes
- Reserved blocks reserved strictly for the root user
Because your background daemon still has the file open, the filesystem driver continues to mark all 13 million 4KB blocks of that 52GB file as allocated. The superblock counters do not change.
df tells you the physical reality of your storage device. du tells you the visible contents of your directory tree.
When df and du disagree by gigabytes, it almost always means an active process is holding open a deleted file.
Tracking Down the Ghost Files
Finding these unlinked files does not require guessing which daemon is responsible. The Linux kernel exposes every open file handle in userspace.
There are two primary ways to track down these ghost files.
The fastest tool is lsof (List Open Files). You can instruct lsof to filter specifically for files whose hard link count is less than 1:
lsof +L1
The +L1 flag tells lsof to look for open files where link count < 1. If you want to restrict the search to a specific mount point like /var, add the filesystem path:
lsof +aL1 /var
Here is what the output looks like on a production host:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
node 24810 appuser 14w REG 8,2 55834574848 0 144182931 /var/log/app/debug.log (deleted)
The columns lay out the exact crime scene. Under COMMAND and PID, you get the executable (node) and process ID (24810). The FD column shows descriptor 14w, meaning file descriptor 14 opened for writing. Most tellingly, NLINK shows 0, confirming the directory entry was unlinked from disk while SIZE/OFF reveals all 55.8GB sitting in limbo under /var/log/app/debug.log (deleted).
If lsof is not installed on a minimal container or hardened production bastion, you do not need to install anything. You can query the /proc virtual filesystem directly.
Every running process in Linux has a directory at /proc/<PID>/fd/. Inside this directory, the kernel creates symbolic links for every open file descriptor assigned to that process.
When a file is unlinked while still open, the kernel automatically appends the string (deleted) to the symlink target.
You can sweep the entire system in one command:
find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'
144182931 54525952 -rw-r--r-- 1 appuser appuser 55834574848 Oct 16 19:40 /proc/24810/fd/14 -> /var/log/app/debug.log\ (deleted)
The output reveals everything: Process ID 24810 holds file descriptor 14, which points to your unlinked 55GB log file.
To inspect what the process is currently doing with that descriptor, you can check its descriptor status in /proc:
cat /proc/24810/fdinfo/14
pos: 55834574848
flags: 0102001
mnt_id: 34
ino: 144182931
The pos field shows the current write offset in bytes. It is sitting at 55.8GB and steadily growing. The daemon is actively writing to a file that you thought you deleted twenty minutes ago.
Freeing the Space Without Killing the Process
Once engineers discover the PID holding the deleted file, their immediate reflex is to kill the process:
kill -9 24810
Do not do this unless you have no other choice.
Killing a production process drops active user websocket connections, terminates inflight database transactions, invalidates local cache pools, and can trigger cascading failure alerts in your monitoring stack. If the process is a core database or monolithic service, restarting it might take ten minutes of cache warm-up time.
There is a clean, zero-downtime way to reclaim every single byte immediately:
Truncate the open file descriptor directly through /proc.
Run this command as root or as the user owning the process:
> /proc/24810/fd/14
Alternatively, you can use the truncate utility:
truncate -s 0 /proc/24810/fd/14
Or redirect from /dev/null:
cat /dev/null > /proc/24810/fd/14
Why does this work?
/proc/<PID>/fd/14 is a magic symlink maintained by the Linux kernel. When you redirect output into this path using >, your shell opens the underlying inode referenced by descriptor 14 with the O_TRUNC flag.
The kernel receives the truncate request, bypasses the missing directory entry, and executes do_truncate() directly against the active inode.
The filesystem driver immediately deallocates the extents and data blocks mapped to that inode and writes them back into the free block bitmap. The file size resets to 0.
Now check df -h /var:
df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 60G 4.8G 53G 9% /var
Instantly, 53 gigabytes of storage return to the available pool. The disk-full alerts resolve. And best of all, the application never stopped running.
The O_APPEND Caveat
There is an important technical nuance you must understand before truncating a running file descriptor.
How the application behaves after truncation depends on how it originally opened the file.
Most production logging libraries, syslog daemons, and web servers open log files with the O_APPEND flag:
int fd = open("/var/log/app/debug.log", O_WRONLY | O_CREAT | O_APPEND, 0644);
When a file descriptor has O_APPEND enabled, the Linux kernel guarantees that before every single write() call, the file offset is automatically reset to the current end of the file.
When you truncate the file to 0 bytes, the next write goes to offset 0. The file continues logging normally from byte zero.
However, if an unusual program opened the file without O_APPEND and maintains its own internal byte counter using lseek():
lseek(fd, current_offset, SEEK_SET);
write(fd, buffer, length);
If the internal offset was at 55GB, truncating the inode sets the length to 0, but the application's next write might seek forward to byte 55,000,000,000 and write 100 bytes there.
This creates a sparse file. On modern filesystems, sparse files do not consume physical disk space for unwritten zero-blocks, so your available disk space will still remain free. But reading the file will return gigabytes of null bytes (\0) before reaching the new log lines.
In practice, virtually all standard daemons (Nginx, Apache, Node.js Winston/Bunyan, Python logging handlers, Logstash) use append mode or standard streaming writes, making descriptor truncation completely safe.
Other Ways Linux Hides Deleted and Used Space
Open unlinked file descriptors account for roughly 90% of df vs du discrepancies. But when lsof +L1 returns completely empty and your disk is still full, one of three other subtle filesystem mechanisms is stealing your storage.
1. The Over-Mounted Directory Shadow
This is the sneakier cousin of the open file descriptor problem.
Suppose an automated backup script runs early during server initialization before external storage volumes are attached:
tar -czf /mnt/backups/system-backup.tar.gz /var/data
The backup script dumps 40 gigabytes into /mnt/backups/. At this moment, /mnt/backups/ is just a regular directory residing on your primary / root partition.
Twenty minutes later, a systemd automount unit or storage script mounts an external EBS volume, SAN LUN, or NFS share on top of that mount point:
mount /dev/sdb1 /mnt/backups
What happened to the 40GB archive?
The mount operation did not delete it. It shadowed it.
When a filesystem is mounted on a directory, the kernel redirects all directory lookups to the root inode of the newly mounted device. The original directory contents on the underlying filesystem become completely invisible to ls, du, and find.
du /mnt/backups now measures the secondary disk /dev/sdb1.
Meanwhile, your root filesystem /dev/sda1 is still holding those 40 gigabytes of data blocks, and df -h / reports that root is almost full.
How do you check for shadowed files without unmounting active storage?
Use a bind mount to inspect the underlying root filesystem in an unmasked state:
mkdir -p /mnt/root_inspect
mount --bind / /mnt/root_inspect
A bind mount creates an alternate view of the root filesystem without re-triggering the mount points stacked on top of it.
Now you can inspect the shadowed directory directly:
ls -lh /mnt/root_inspect/mnt/backups/
There is your missing 40GB file. You can safely remove it from /mnt/root_inspect/mnt/backups/, and df on your root partition will instantly drop.
When you are done, clean up the inspection mount:
umount /mnt/root_inspect
rmdir /mnt/root_inspect
2. The Hard Link Survival Trap
Another common misconception is assuming rm frees disk blocks when a file has hard links.
If another engineer or an automated snapshot script created a hard link to your log or data file:
ln /var/log/app/debug.log /opt/archive/debug_backup.log
Both filenames point to the exact same inode number. You can verify this with ls -l or stat:
stat -c "Links: %h Inode: %i File: %n" /var/log/app/debug.log /opt/archive/debug_backup.log
Links: 2 Inode: 8912345 File: /var/log/app/debug.log
Links: 2 Inode: 8912345 File: /opt/archive/debug_backup.log
Notice the link count is 2.
When you run rm /var/log/app/debug.log, the kernel decrements the inode link count from 2 to 1.
Because i_nlink is still greater than zero, the inode is not marked for deletion. Not a single data block is freed. The 50GB file remains fully allocated on disk under the name /opt/archive/debug_backup.log.
If you run rm on a large file and space is not reclaimed even though no process holds it open, check if the file had multiple hard links before deletion:
find / -inum 8912345 2>/dev/null
3. Reserved Filesystem Blocks (The 5% Buffer)
Have you ever seen df -h show that a filesystem is 100% full, but the math does not add up?
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 100G 95G 0 100% /
Total size is 100G. Used space is 95G. Why is Available space sitting at 0 instead of 5G?
By default, the mke2fs utility reserves 5% of total filesystem blocks exclusively for the root user when creating ext2/ext3/ext4 partitions.
The reasoning dates back to early multi-user Unix systems:
- It prevents unprivileged user processes from completely filling the disk, which would prevent system daemons (like
sshd,syslog, orpostfix) from writing state. - It mitigates filesystem fragmentation by ensuring the allocation engine always has contiguous blocks available.
On a 20GB disk created in 2005, a 5% reserve was only 1GB. On a modern 4TB NVMe drive, 5% is a staggering 200 gigabytes of wasted storage that regular service accounts cannot use.
When your application service account hits this 5% wall, its writes fail with ENOSPC: No space left on device, even though physical blocks are empty.
You can inspect and adjust this reserved percentage on a live, mounted ext4 filesystem with tune2fs:
tune2fs -l /dev/sda1 | grep "Reserved block count"
To reduce the reserve from 5% to 1% (or even 0% on a dedicated non-root data partition), run:
tune2fs -m 1 /dev/sda1
This immediately releases 4% of the partition's total capacity back to general users and background services without a reboot.
Exactly One Surprising Fact
The fact that unlink() keeps open files alive until process termination is not an accident, a bug, or an oversight of early filesystem design.
It was an intentional, foundational security and design pattern invented by Ken Thompson and Dennis Ritchie in the early 1970s for Unix.
In Unix, programs frequently needed temporary scratchpad files for sorting, staging intermediate compiler output, or buffering pipeline streams. If a program crashed or was terminated with kill, temporary files littered /tmp and required manual cleanup scripts.
Thompson and Ritchie designed unlink() so that a program could create a file, immediately call unlink() on it, and continue reading and writing to its file descriptor in private memory.
Because the file had no directory entry, no other user on the shared multi-user mainframe could see the file, guess its name, or tamper with its data. And the microsecond the program finished or crashed, the kernel automatically cleaned up every single allocated disk block.
Modern Linux filesystems formalized this exact pattern in Linux kernel 3.11 with the O_TMPFILE flag passed to open():
int fd = open("/tmp", O_TMPFILE | O_RDWR, 0600);
This creates an anonymous, completely invisible inode that never exists in any directory tree from the moment of creation. It consumes disk blocks while open, and vanishes without a trace the moment the descriptor closes.
How to Prevent the Ghost File Trap in Production
Now that you understand the mechanics, here is how to make sure this never catches your team off guard again.
1. Never rm an Active Log File
If a service is currently running, never use rm on its active log files. Always truncate them instead:
# Good: Frees disk space immediately while keeping the file descriptor valid
> /var/log/app/debug.log
# Also good:
truncate -s 0 /var/log/app/debug.log
Truncation preserves the directory entry, resets the file length to zero, and forces the filesystem to reclaim the physical disk blocks instantly. The application continues logging without missing a write or needing a restart.
2. Configure logrotate Correctly
Most ghost file incidents occur because of misconfigured log rotation.
There are two primary paradigms in logrotate:
Approach A: Copy and Truncate (copytruncate)
/var/log/app/*.log {
daily
rotate 7
compress
missingok
copytruncate
}
The copytruncate directive instructs logrotate to copy the live log file to a backup name (e.g., debug.log.1), and then truncate the original file in place to 0 bytes.
This is ideal for applications that do not support reloading log file handles (such as third-party binaries or proprietary Java apps). Because the original file is never unlinked, its descriptor remains intact.
Approach B: Move and Signal (create + reload signal)
/var/log/nginx/*.log {
daily
rotate 14
missingok
compress
delaycompress
sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
endscript
}
In this model, logrotate renames access.log to access.log.1 and creates a fresh empty access.log.
At this exact instant, Nginx is still writing to descriptor pointing to access.log.1.
The postrotate block sends the USR1 signal to Nginx. Nginx intercepts USR1, closes its existing file descriptors, and reopens /var/log/nginx/access.log.
If someone deletes the postrotate section or the signal fails because of an incorrect PID file path, the daemon will hold the rotated or unlinked file open forever.
3. Monitor Unlinked File Descriptors
Do not wait for a 100% disk outage to discover deleted files.
If you use Prometheus and the node_exporter, configure alerts on the divergence between filesystem size and available bytes:
# Alert when unlinked files or reserved blocks consume more than 15% of partition
(node_filesystem_size_bytes - node_filesystem_free_bytes) - (node_filesystem_size_bytes - node_filesystem_avail_bytes)
Alternatively, add a simple periodic cron check or Datadog custom metric that counts the volume of open unlinked files:
lsof +aL1 /var -F s | grep '^s' | cut -c 2- | awk '{s+=$1} END {print s/1024/1024 " MB"}'
The 60-Second Ghost File Triage Playbook
When df reports that a disk partition is full but du cannot find the files, follow this sequence:
First, identify all open files with zero directory links on the affected mount:
lsof +aL1 /mount_point
Second, if lsof is not available, sweep the process descriptors in /proc:
find /proc/*/fd -ls 2>/dev/null | grep '(deleted)' | sort -k7 -n
Third, note the Process ID (PID) and File Descriptor (FD) of the largest unlinked file.
Fourth, reclaim the disk space immediately without killing the process:
> /proc/<PID>/fd/<FD>
Fifth, verify that disk space has returned:
df -h /mount_point
Sixth, if the application supports signal reloading, notify it to close and reopen logs cleanly:
kill -HUP <PID>
# Or for Nginx:
kill -USR1 <PID>
Seventh, if lsof showed zero deleted files, check for shadowed files trapped under an active mount point:
mkdir -p /mnt/temp_root && mount --bind / /mnt/temp_root
du -sh /mnt/temp_root/* | sort -h
umount /mnt/temp_root && rmdir /mnt/temp_root
Wrapping Up
Storage in Linux is governed by reference counting, not desktop trash cans.
When you run rm, you are only removing a label from a directory index. As long as any running process keeps that file handle open, the Linux kernel protects those data blocks from being overwritten.
The next time df and du tell you two conflicting stories during a storage incident, do not reboot the server and do not reach for kill -9. Look at the file descriptors in /proc, truncate the open handle, and get your disk space back in seconds without taking your services down.
What is the largest ghost file you have ever chased down in production, and which daemon was stubbornly refusing to let it go?
If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.
About the Author
Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.
His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.
Connect with Me
- Portfolio: asepsayyad007.in
- Blog: asepsayyad007.in/blogs
- GitHub: github.com/asepsayyad007
- LinkedIn: linkedin.com/in/asepsayyad
- Medium: asepsayyad007.medium.com
- Support: buymeacoffee.com/asepsayyad007
Enjoyed this article?
If this guide saved you hours of debugging or gave you something practical for production, consider:
- Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
- Starring my open-source projects on GitHub.
- Sharing this article with fellow Linux and DevOps engineers.
You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!
© 2026 Asep Sayyad
Top comments (0)