rm Freed Nothing and df Stayed Full: Finding the Process Still Holding a Deleted Camera Clip

A cleanup job deletes last week's camera clips, ls shows the folder nearly empty, and df reports the disk exactly as full as before. Nothing is broken. Some process still has one of those clips open, and on Linux a file that is open does not give its space back when its name is removed. I wanted to see that happen on a disk I could measure, find the process holding the space, and work out which fixes actually return it.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it stores files. Everything here ran on one Linux machine (kernel 6.8, /bin/sh is dash, GNU coreutils 8.32, lsof 4.93.2, Python 3.10), on an ext4 disk, against files the scripts create themselves.

What the tests showed:

• Deleting an open 200 MiB clip freed nothing. df stayed 200 MiB up after rm, while du on the folder said 4 KiB.
• The space came back the moment the writing process exited. No other step was needed.
• lsof +L1 and a search of /proc/<pid>/fd both found the file, marked (deleted).
• Truncating the file through /proc freed the space without stopping the writer, but the writer kept its old position, so its next write made a sparse file with a 200 MiB hole in front.
• My first lsof check was wrong. +L1 and -p were joined with OR, so it listed a file that still had a name.

A stand-in for the recorder

The real culprit could be a recorder, a streaming tool, a sync client or a log writer. For the test, all that matters is a process that opens a file, writes to it, and keeps it open:

import sys, time
# Stand-in for a recorder: open a clip, write 200 MiB, keep the file open.
f = open(sys.argv[1], "wb")
chunk = b"\0" * (1024 * 1024)
for _ in range(200):
    f.write(chunk)
f.flush()
time.sleep(600)

It writes 200 MiB of zeros, flushes, and then sleeps with the file still open, which is what a recorder in the middle of a clip looks like from the outside.

Delete it and measure

This script starts the writer, deletes the file while it is open, and reads df at each step. It reports the change in used space against a reading taken before the writer started:

#!/bin/sh
# Reproduce "I deleted the clip and the disk did not get the space back".
DIR=/tmp/clips
mkdir -p "$DIR"
used() { df --output=used -k "$DIR" | tail -1 | tr -d ' '; }

BEFORE=$(used)
python3 holder.py "$DIR/clip_0001.mp4" &
PID=$!
sleep 3
echo "after writing: df grew by $(( ($(used) - BEFORE) / 1024 )) MiB"

rm "$DIR/clip_0001.mp4"
echo "after rm:      df grew by $(( ($(used) - BEFORE) / 1024 )) MiB"
echo "du of folder:  $(du -sk "$DIR" | cut -f1) KiB"

echo "--- lsof +L1 (open files with no name left)"
lsof -nP -a +L1 -p "$PID" | awk 'NR==1 || /clip_/'
echo "--- the same thing without lsof"
find /proc/"$PID"/fd -lname '*(deleted)' -printf '%p -> %l\n'

kill "$PID"; wait "$PID" 2>/dev/null
sleep 1
echo "after the process exits: df grew by $(( ($(used) - BEFORE) / 1024 )) MiB"

Output:

after writing: df grew by 200 MiB
after rm:      df grew by 200 MiB
du of folder:  4 KiB
--- lsof +L1 (open files with no name left)
COMMAND PID                USER   FD   TYPE DEVICE  SIZE/OFF NLINK  NODE NAME
python3  14 eager-stoic-maxwell    3w   REG    8,1 209715200     0 13774 /tmp/clips/clip_0001.mp4 (deleted)
--- the same thing without lsof
/proc/14/fd/3 -> /tmp/clips/clip_0001.mp4 (deleted)
after the process exits: df grew by 0 MiB

The unlink(2) manual page describes exactly this: "If the name was the last link to a file but any processes still have the file open, the file will remain in existence until the last file descriptor referring to it is closed." The name went away, so ls and du, which walk names, no longer count it. df asks the filesystem how many blocks are in use, and those blocks are still allocated to a file with no name.

The NLINK column is the useful part of the lsof line. Zero means no directory entry points at the file any more. The lsof manual page says a specification of the form +L1 "will select open files that have been unlinked." The /proc search finds the same thing without lsof installed, because the kernel shows each open descriptor as a symbolic link and appends (deleted) to the target of one whose file has lost its name.

When the process exited, the kernel closed its descriptors, the last reference went away, and the blocks were released. On a normal unlink with nothing holding the file, the same page says "the file is deleted and the space it was using is made available for reuse." That step had only been postponed.

The check that lied first

My first version of the lsof line was lsof +L1 -p $PID, and on a later test it printed a clip that had not been deleted at all. The lsof manual explains why: "Normally, list options that are specifically stated are ORed". So +L1 -p PID means every unlinked open file on the machine, plus every file that process has open. To get only the deleted files of one process, the options have to be joined with -a, which "causes list selection options to be ANDed".

#!/bin/sh
# lsof joins selection options with OR unless -a is given.
DIR=/tmp/clips
mkdir -p "$DIR"
python3 holder.py "$DIR/clip_0005.mp4" &
PID=$!
sleep 3
echo "named file, still on disk. Lines mentioning it:"
echo "  lsof +L1 -p PID     -> $(lsof -nP +L1 -p "$PID" | grep -c clip_0005)"
echo "  lsof -a +L1 -p PID  -> $(lsof -nP -a +L1 -p "$PID" | grep -c clip_0005)"
kill "$PID"; wait "$PID" 2>/dev/null
rm -f "$DIR/clip_0005.mp4"

named file, still on disk. Lines mentioning it:
  lsof +L1 -p PID     -> 1
  lsof -a +L1 -p PID  -> 0

The same file shows up once without -a and not at all with it. Every lsof line in the scripts here carries -a for that reason. Without a -p there is nothing to combine, so a plain lsof +L1 across the whole machine is fine as it is.

Getting the space back without stopping the writer

Stopping or restarting the process that holds the file is the clean fix. Sometimes that process is a recording you would rather not interrupt. The descriptor link under /proc can be opened like a file, and truncating it to zero frees the blocks even though the file has no name. I also wanted to see what happens when the writer keeps writing after that, so this writer adds one more MiB when told to:

import os, sys, time
# Like holder.py, but writes one more MiB when a go-file appears.
f = open(sys.argv[1], "wb")
chunk = b"\1" * (1024 * 1024)
for _ in range(200):
    f.write(chunk)
f.flush()
while not os.path.exists(sys.argv[2]):
    time.sleep(0.2)
f.write(chunk)
f.flush()
time.sleep(600)

#!/bin/sh
# Free the space of a deleted-but-open file without stopping the writer,
# then watch what happens when the writer writes again.
DIR=/tmp/clips
mkdir -p "$DIR"
used() { df --output=used -k "$DIR" | tail -1 | tr -d ' '; }

BEFORE=$(used)
python3 holder2.py "$DIR/clip_0002.mp4" "$DIR/go" &
PID=$!
sleep 3
rm "$DIR/clip_0002.mp4"
FD=$(find /proc/"$PID"/fd -lname '*clip_0002.mp4 (deleted)' -printf '%f\n')
echo "deleted, still open as fd $FD: df grew by $(( ($(used) - BEFORE) / 1024 )) MiB"

: > /proc/"$PID"/fd/"$FD"
sleep 1
echo "after truncating through /proc: df grew by $(( ($(used) - BEFORE) / 1024 )) MiB"

touch "$DIR/go"; sleep 2
echo "writer wrote 1 MiB more:"
echo "  apparent size $(stat -L -c %s /proc/"$PID"/fd/"$FD") bytes"
echo "  blocks on disk $(( $(stat -L -c %b /proc/"$PID"/fd/"$FD") * 512 )) bytes"
kill "$PID"; wait "$PID" 2>/dev/null
rm -f "$DIR/go"

Output:

deleted, still open as fd 3: df grew by 200 MiB
after truncating through /proc: df grew by 0 MiB
writer wrote 1 MiB more:
  apparent size 210763776 bytes
  blocks on disk 1048576 bytes

The truncation worked: df went back to where it started. Then the writer wrote its next MiB at the position where it had left off, 200 MiB into the file, not at the start. The file's apparent size jumped to 201 MiB while only 1 MiB of blocks were actually allocated on disk. The rest is a hole that reads back as zeros.

For a file that has already been deleted, the hole costs nothing, because nobody will open it by name again and the disk space really is free. The same behaviour is worse on a file that still has a name. Truncating a clip or log that a process is still writing, with > file or truncate -s 0, leaves that writer at its old offset too, and its next write produces a file that starts with a long run of zeros. Anything that reads that file from the start sees zeros where the first bytes of the clip should be. My writer opened its file in plain write mode. A writer that opens in append mode may behave differently, and I did not test that case.

A cleanup job that skips open files

The deleted-but-open file is usually created by the cleanup job itself: it deletes everything older than some cutoff, and one of those files is still being written or uploaded. Checking whether anything has a file open before deleting it avoids the problem rather than cleaning up after it. fuser -s exits with success when some process has the file open and says nothing:

#!/bin/sh
# Delete clips, but skip any clip some process still has open.
DIR=/tmp/clips
for f in "$DIR"/*.mp4; do
    [ -e "$f" ] || continue
    if fuser -s "$f" 2>/dev/null; then
        echo "skip (open): $f"
    else
        rm -- "$f" && echo "removed:     $f"
    fi
done

And a test with one closed clip and one held open by the stand-in writer:

#!/bin/sh
DIR=/tmp/clips
mkdir -p "$DIR"
head -c 1048576 /dev/zero > "$DIR/clip_0003.mp4"
python3 holder.py "$DIR/clip_0004.mp4" &
PID=$!
sleep 3
./prune.sh
echo "--- left in folder:"
ls "$DIR"
echo "--- lsof +L1 for the writer:"
lsof -nP -a +L1 -p "$PID" | awk '/clip_/' | wc -l
kill "$PID"; wait "$PID" 2>/dev/null
rm -f "$DIR"/clip_0004.mp4

removed:     /tmp/clips/clip_0003.mp4
skip (open): /tmp/clips/clip_0004.mp4
--- left in folder:
clip_0004.mp4
--- lsof +L1 for the writer:
0

The closed clip went, the open one stayed, and the writer was left with no unlinked files. On the next run, after the writer has finished with it, the skipped clip gets deleted like any other.

There is a race in that loop. A process could open a file in the moment between the fuser check and the rm. For a folder where files are written once and then only read occasionally, that gap is small. For a folder where a sync tool might pick up any file at any moment, the safer pattern is to have the writer close a clip and move it into a finished folder, and have cleanup touch only that folder.

Finding the holder on a real machine

On a box where df and du disagree, these are the checks I would run, from cheapest to most specific:

1. Compare df -h on the mount with du -sh on its top directory. A large gap is the first hint, though other things, such as reserved blocks and other mounts hidden under a directory, can also cause one.
2. Run lsof -nP +L1 as root to list every unlinked open file and the process holding it. I ran everything here as an ordinary user, against processes that user owned, so I have not checked how much a non-root run misses.
3. Without lsof, search /proc/*/fd for links ending in (deleted), again as root.
4. Decide what to do with the holder: let it finish, restart it, or, if neither is possible, truncate through /proc knowing what the writer will do next.

Things I did not test

• Other filesystems. Everything ran on ext4. Network filesystems handle files that are open on one machine and deleted on another in their own way, and the unlink(2) page has a note on NFS in its bugs section. I did not look at that.
• Writers in append mode after a truncation, mentioned above.
• Memory-mapped files. A process can keep a deleted file alive through a mapping rather than a descriptor. Those are listed under /proc/<pid>/maps rather than under fd, and none of the scripts here checks for them.
• Containers. I did not run anything inside a container or on a bind-mounted volume.

Sources. Quotes are from the unlink(2) manual page, description section, and from the lsof(8) manual page, on how selection options are combined, on -a, and on +L1. Test results come from the scripts above, each run once in a fresh folder on one Linux machine, with the output pasted unedited.

If you want an old phone to record in the background so there is a folder of clips to clean up, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Disk space belongs to whoever still holds the file open, whatever the folder listing says.