flock on a Cron Copy Job: A Background Helper Kept the Lock After the Script Exited

Cron starts a job on schedule whether or not the last run has finished. Give it a copy job that runs every fifteen minutes and sometimes takes longer, and two copies of that job end up working on the same folder at once. The usual fix is one word in the cron line, flock, and I wanted to know what that word does and does not cover before trusting it with a folder of camera clips. So I tested it with small scripts that stand in for the real job.

Disclosure: I make an Android camera app. Nothing here uses its code or describes how it stores files. Everything below runs on a Linux computer, with bash 5.1 and flock from util-linux 2.37.2, against files the scripts create themselves.

What the tests showed:

• Without a lock, the second run started while the first was still working. Both logged a start before either logged an end.
• flock -n refused the second run, with exit status 1. A job that ran and failed also exits 1, so the two looked the same until I set -E.
• A home-made lock file got stuck after kill -9. Every later run skipped itself and exited 0.
• flock's lock survived kill -9 of the flock process while its child was still running, and cleared the moment the child exited.
• A helper left running in the background kept the lock after the script finished. The next run was refused even though nothing was copying.

The stand-in job

The real job copies clips. For a test, all that matters is that it takes a while and says when it starts and stops:

#!/bin/bash
# Stand-in for a slow nightly copy: a start line, 3 seconds of work, an end line.
echo "start $1" >> run.log
sleep 3
echo "end   $1" >> run.log

It is saved as copy-job.sh. Each test below runs in a fresh folder with its own copies of the scripts.

No lock: two runs overlap

Cron does not know whether the last run finished. It starts the job on schedule. This script does the same thing by hand, with the second run one second behind the first:

#!/bin/bash
# Two runs, the second starting 1 second after the first. No lock.
rm -f run.log
./copy-job.sh A &
sleep 1
./copy-job.sh B
wait
cat run.log

start A
start B
end   A
end   B

Both runs were inside the job at once. For a copy job that means two processes reading the same source and writing the same destination, and whichever finishes second decides what the destination looks like.

flock -n: the second run is refused

The same two runs, each wrapped in flock -n on one lock file:

#!/bin/bash
# The same two runs, each wrapped in flock -n on one lock file.
rm -f run.log
flock -n copy.lock ./copy-job.sh A &
sleep 1
flock -n copy.lock ./copy-job.sh B
echo "B exit status: $?"
wait
cat run.log

B exit status: 1
start A
end   A

B did not start. Run A held the lock, B could not get it, and -n told flock not to wait. Without -n, flock queues instead. The manual says that by default, "if the lock cannot be immediately acquired, flock waits until the lock is available." For a job that runs every fifteen minutes, queueing is usually the wrong choice: if every run is slow, the queue grows.

The lock file itself is just an empty file. flock creates it if it is missing, and it stays on disk after the run. That is fine, and the next section shows why it matters that it is fine.

Exit status 1 means two different things

A refused run exits 1. So does a job that ran and failed. Side by side:

#!/bin/bash
# A job that ran and failed, next to a job that never ran.
flock -n copy.lock false
echo "ran and failed:          $?"
flock -n copy.lock sleep 2 &
sleep 0.5
flock -n copy.lock true
echo "never ran:               $?"
flock -n -E 75 copy.lock true
echo "never ran, with -E 75:   $?"
wait

ran and failed:          1
never ran:               1
never ran, with -E 75:   75

The manual describes -E as "The exit status used when the -n option is in use, and the conflicting lock exists", with "The default value is 1." I picked 75 because it is free in my scripts; any value from 0 to 255 that your own job does not use for its own errors will do. With it, a log or a monitoring check can tell a skipped run, where the last one was still going, from a run that broke. A run skipped once is normal. Skipped every time means the job takes longer than its interval, which a plain exit 1 would hide among the real failures.

The home-made lock file gets stuck

Before flock, the usual pattern is to check for a file, create it, and remove it on exit:

#!/bin/bash
# The do-it-yourself lock: skip if the file exists, remove it on exit.
if [ -e naive.lock ]; then echo "lock file exists, skipping"; exit 0; fi
touch naive.lock
trap 'rm -f naive.lock' EXIT
sleep 30

Saved as naive-job.sh. The test kills the first run with SIGKILL, which no trap can catch, then starts the next one:

#!/bin/bash
# Kill the first run with SIGKILL, then start the next one.
rm -f naive.lock
./naive-job.sh & P=$!
sleep 1
kill -9 $P; wait $P 2>/dev/null
./naive-job.sh
echo "next run exit status: $?"
ls naive.lock

lock file exists, skipping
next run exit status: 0
naive.lock

The file outlived the process that made it. From here on, every run sees the file, prints that it is skipping, and exits 0. The job has stopped, and its exit status says everything is fine. A power cut or an out-of-memory kill does the same thing as kill -9.

flock does not have this problem, because the lock is not the file. The flock(2) manual says the lock is held on the open file, and "the lock is released either by an explicit LOCK_UN operation on any of these duplicate file descriptors, or when all such file descriptors have been closed." When a process dies, the kernel closes its files, so the lock goes with it. The empty file left on disk means nothing.

But the lock belongs to every process holding the file

That last clause, about every duplicate descriptor, turned out to matter. In the command form, flock starts the job as a child process, and the child inherits the open lock file. This test kills only the flock process and leaves the child alone:

#!/bin/bash
# Kill only the flock process with SIGKILL, then its child. Check the lock after each.
flock -n copy.lock sleep 30 & P=$!
sleep 1
C=$(pgrep -P $P)
kill -9 $P; wait $P 2>/dev/null
flock -n copy.lock true; echo "flock killed, child alive: $?"
kill $C; sleep 0.2
flock -n copy.lock true; echo "child gone too:            $?"

flock killed, child alive: 1
child gone too:            0

With flock gone, the lock was still held, because sleep (the stand-in for the job) still had the file open. Once the child exited, the lock cleared. That is the behaviour you want while the job is really still running. It becomes a problem when the process holding the file is not the job.

A background helper keeps the lock after the script ends

A copy script that starts something in the background and exits, an upload, a thumbnail pass, a notification, hands that helper its open files, including the lock. Here the lock is taken inside the script, on file descriptor 9, which is the pattern the flock manual shows for use inside scripts:

#!/bin/bash
# Lock held on fd 9 for the whole script. Leaves a helper running in the background.
exec 9>copy.lock
flock -n 9 || { echo "busy, skipping"; exit 75; }
echo "copying"
sleep 20 &
echo "script done"

The second version is identical except for one redirection, 9>&-, which closes descriptor 9 for the helper only:

#!/bin/bash
# Same script, but the helper does not inherit fd 9.
exec 9>copy.lock
flock -n 9 || { echo "busy, skipping"; exit 75; }
echo "copying"
sleep 20 9>&- &
echo "script done"

Each one run twice in a row, saved as bg-job.sh and bg-job-fixed.sh:

#!/bin/bash
# Run each version twice in a row; wait out the helpers in between.
./bg-job.sh;       echo "exit $?"
./bg-job.sh;       echo "exit $?"
sleep 21
./bg-job-fixed.sh; echo "exit $?"
./bg-job-fixed.sh; echo "exit $?"

copying
script done
exit 0
busy, skipping
exit 75
copying
script done
exit 0
copying
script done
exit 0

The first script printed script done and exited 0, and the next run was still refused, because the 20-second helper held the lock. In a real job that helper could be anything slow, and every run in its shadow is skipped. With 9>&- on the helper, back-to-back runs both went through.

The flock manual names this exact case, in its entry for -u: dropping a lock by hand "may be required in special cases, for example if the enclosed command group may have forked a background process which should not be holding the lock." Closing the descriptor on the helper does the same thing without having to remember to unlock.

The command form has a switch for this: -o

When flock runs the command itself, -o does the same job. The manual says it will "Close the file descriptor on which the lock is held before executing command", and that it "is useful if command spawns a child process which should not be holding the lock."

#!/bin/bash
# Command form, with and without -o (close the lock's descriptor before running the command).
flock -n copy.lock sh -c 'sleep 20 & echo started helper'
flock -n copy.lock true; echo "without -o, next run: $?"
sleep 21
flock -n -o copy.lock sh -c 'sleep 20 & echo started helper'
flock -n copy.lock true; echo "with -o, next run:    $?"

started helper
without -o, next run: 1
started helper
with -o, next run:    0

-o has a cost, and the test shows it. With -o, only the flock process holds the lock while the command runs. Kill flock, and the lock is gone even though the command is still working:

#!/bin/bash
# With -o, kill only the flock process while its command is still running.
flock -n -o copy.lock sleep 5 & P=$!
sleep 1
flock -n copy.lock true; echo "while the command runs: $?"
C=$(pgrep -P $P)
kill -9 $P; wait $P 2>/dev/null
flock -n copy.lock true; echo "flock killed, command still running: $?"
kill $C

while the command runs: 1
flock killed, command still running: 0

So the choice is which failure you would rather have. Without -o, a stray helper can block the next run. With -o, a killed flock lets a second copy start beside a first one that is still running. For a copy job, I would rather close the descriptor on the helpers I start on purpose (9>&- as above) and leave the job itself holding the lock.

What the cron line looks like

Putting the pieces together, a line like this runs the copy every fifteen minutes, skips a run when the last one is still going, and marks a skip with 75 instead of 1:

*/15 * * * * flock -n -E 75 /home/me/locks/copy-clips.lock /home/me/bin/copy-clips.sh

I have not run this line under cron itself. Every result above comes from running the same flock commands by hand. The lock file should live on a local disk, somewhere the job's user can write, since opening it with > needs write permission.

Things I did not test

• Network filesystems. The manual warns: "Some filesystems (e. g. NFS and CIFS) have a limited implementation of flock(2) and flock may always fail." Every lock file here sat on a local disk. If the clips are on a network share, keep the lock file on a local disk.
• Jobs that ignore the lock. flock(2) says these locks are advisory only: "a process is free to ignore the use of flock() and perform I/O on the file." The lock only stops runs that also ask for it. A second script copying into the same folder without flock is not stopped. The same page lists one exception: on CIFS since Linux 5.5, flock() locks are emulated with SMB byte-range locks and "are not advisory anymore".
• systemd timers. I only looked at cron-style scheduling here.
• -w. A timeout instead of an instant refusal. The manual says a timeout of zero is the same as -n. I left it out of the tests above.

Sources. Quotes are from the flock(1) manual page, entries for the default behaviour, -E, -o, -u and the notes on NFS and CIFS, and from the flock(2) manual page, on how locks are released, on advisory locking and on CIFS. Test results come from the scripts above, run on one Linux machine with bash 5.1 and util-linux 2.37.2. Every test was run a second time in a fresh folder, with identical output.

This lock sits in front of the same nightly camera-clip copy I tested elsewhere: what the copy itself does when the source folder is empty is in rsync --delete emptied the backup and exited 0, and the cleanup job that deletes old clips is in find -mtime +7 keeps eight days.

If you want a phone that records in the background so there is something to copy, the app I make is Background Camera RemoteStream on Google Play. The scripts above do not depend on it. Developer site: superfunicular.com.

A lock file is only as honest as the list of processes that still have it open.