timeout Returned 124 After the Job Had Finished Its Work: Testing GNU timeout on a Nightly Clip Script

A common way to keep a nightly clip job inside its window is to wrap it in timeout and add the obvious check after it: if the exit status is 124, the job was cut off. I tested that check against a job that ignores the stop signal. It ran for about four seconds against a one-second limit, finished all its work, and timeout still reported 124:

#!/bin/bash
# stands in for a job that catches TERM and keeps going
trap '' TERM
sleep 4
echo "ignore_term.sh: finished its work"

start=$(date +%s)
timeout 1 ./ignore_term.sh
rc=$?
echo "exit status $rc after $(( $(date +%s) - start )) s (limit was 1 s)"

ignore_term.sh: finished its work
exit status 124 after 5 s (limit was 1 s)

(The 5 s is whole seconds from date +%s, so it includes rounding.)

So 124 does not mean the command was stopped. It means the limit was reached and a signal was sent. Whether the command obeyed is a separate question, and the exit status does not answer it.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it records. The jobs here are shell scripts and a desktop ffmpeg writing a test pattern, and every output is pasted from the scripts shown. Versions:

timeout (GNU coreutils) 8.32
GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)
ffmpeg version 4.4.2-0ubuntu0.22.04.1

What the manual promises about 124

The GNU manual's list of exit statuses starts with "124 if command times out, and --preserve-status is not specified". It does not say the command ended because of the timeout. One of the manual's own examples makes the point: it sends INT to a sleep that ignores INT, and the comment beside it says the sleep runs its full 20 seconds while timeout still returns 124.

A job that catches TERM to clean up, flush a buffer or finish a file is in the same position. A 124 tells you it was asked to stop, not when it stopped or whether it finished first.

Adding -k changes the number

-k (--kill-after) sends KILL some time after the first signal if the command is still running. The manual is specific about when that clock starts: at the first signal, "not from the beginning when the command is started."

With a one-second grace period the same script was gone after two seconds, and the status was no longer 124:

start=$(date +%s)
timeout -k 1 1 ./ignore_term.sh
rc=$?
echo "exit status $rc after $(( $(date +%s) - start )) s"
[ "$rc" -eq 124 ] && echo "a check for 124 sees a timeout" || echo "a check for 124 sees no timeout"

exit status 137 after 2 s
a check for 124 sees no timeout

That is 128 plus 9, the KILL signal. The manual lists 137 for a KILL sent to either the command or timeout itself, and says the two cases "cannot be distinguished." Bash also printed a Killed line to standard error naming the timeout process itself, which I left out of the output file.

So a wrapper that only tests for 124 reports no timeout exactly when the job was stopped hardest. Adding -k to make the limit stricter, without updating that check, makes the check wrong.

A shell function is not a command

Another case that looks harmless: moving a step such as an upload into a shell function and passing the function name to timeout:

upload() { sleep 3; }
timeout 1 upload; echo "function: exit status $?"
export -f upload
timeout 1 bash -c upload; echo "exported, run via bash -c: exit status $?"

function: exit status 127
exported, run via bash -c: exit status 124

timeout is a separate program, so it can only run something it can find as a file; a function exists only inside the shell that defined it. It printed failed to run command ‘upload’: No such file or directory to standard error and returned 127, which the manual lists as "127 if command cannot be found".

Exporting the function and running it through bash -c works, as the second line shows. Putting the step in its own script file is simpler and easier to test.

127 is worth a look in any wrapper. A typo in a crontab, a missing binary after an upgrade and a function name all produce it, and none of them is a timeout.

Background children go with it, unless you ask otherwise

A job script that starts a helper in the background raises a question: does the helper outlive the time limit? I tested a parent script that starts sleep 30 & and waits:

cat > parent.sh <<'P'
#!/bin/bash
sleep 30 &
echo $! > child.pid
wait
P
chmod +x parent.sh
for mode in "" "--foreground"; do
  timeout $mode 1 ./parent.sh; rc=$?
  sleep 0.5
  if kill -0 "$(cat child.pid)" 2>/dev/null; then s="still running"; kill "$(cat child.pid)"; else s="gone"; fi
  echo "timeout ${mode:-(default)}: exit status $rc, background sleep $s"
done

timeout (default): exit status 124, background sleep gone
timeout --foreground: exit status 124, background sleep still running

By default the background sleep went too. With --foreground it did not; the manual describes that option as not creating a separate background program group. The manual says so directly: in that mode "any children of command will not be timed out."

--foreground exists for commands that need the terminal, so it is easy to add while debugging a job by hand and forget it is there. If the job starts anything in the background, that one flag decides whether the time limit covers it.

--preserve-status hides the timeout

--preserve-status returns the command's own status instead of 124. For a sleep stopped by TERM that is 143 (128 plus 15). It also means a command can produce 124 by itself:

timeout --preserve-status 1 sleep 5; echo "--preserve-status, sleep stopped by TERM: $?"
timeout 5 bash -c 'exit 124'; echo "command that exits 124 by itself, no timeout: $?"

--preserve-status, sleep stopped by TERM: 143
command that exits 124 by itself, no timeout: 124

Neither case is a defect, but both show that the exit status is a weak signal. The manual's last rule is "the exit status of command otherwise", so a command that happens to exit 124 on its own looks exactly like a timeout.

What it does to a recording

This is the part that matters for a clip job. I recorded a three-second test pattern with ffmpeg to MP4 and stopped it with each signal at the limit:

for sig in INT TERM KILL; do
  rm -f "rec_$sig.mp4"
  timeout -s "$sig" 3 ffmpeg -nostdin -loglevel quiet -re \
    -f lavfi -i testsrc=size=320x240:rate=15 -c:v libx264 -pix_fmt yuv420p "rec_$sig.mp4"
  rc=$?
  if ffprobe -v quiet "rec_$sig.mp4"; then p="opens"; else p="does not open"; fi
  echo "$sig: exit status $rc, $(stat -c %s "rec_$sig.mp4") bytes, ffprobe: $p"
done

INT: exit status 124, 15064 bytes, ffprobe: opens
TERM: exit status 124, 15064 bytes, ffprobe: opens
KILL: exit status 137, 48 bytes, ffprobe: does not open

INT and TERM both gave a file that ffprobe opens. KILL left 48 bytes that it does not open; in my exploratory runs ffprobe named the reason as moov atom not found. That is the index an MP4 writer adds when it closes the file, and KILL gives a process no chance to close anything.

One honest gap: across thirteen TERM runs on this machine, the first left a 48-byte file, and ffmpeg printed Error writing trailer followed by Immediate exit requested. The twelve TERM runs after it all gave a file that opens, so I do not know what caused it. INT gave a file that opens in all thirteen of its runs in that batch. A later re-check on the same machine saw the very first run fail the same way with INT instead, so the failure looks tied to a cold first run rather than to one signal.

The practical reading: give the recorder a first signal it handles, and a grace period long enough for it to write the end of the file. -k with a few seconds is a backstop for a process that hangs, not the normal way to end a recording.

A wrapper that reads every status

#!/bin/bash
# usage: run_clip_job.sh LIMIT GRACE command [args...]
limit=$1; grace=$2; shift 2
timeout -k "$grace" "$limit" "$@"
rc=$?
case $rc in
  0)       echo "ok" ;;
  124)     echo "limit reached, first signal sent; the command may still have finished" ;;
  137)     echo "KILL sent after the grace period; treat output as incomplete" ;;
  125)     echo "timeout itself failed" ;;
  126|127) echo "command not run: not found, not executable, or a shell function" ;;
  *)       echo "command exited $rc on its own" ;;
esac
exit $rc

And a run across the five cases above (standard error discarded, so the Killed and failed to run command lines are not shown):

./run_clip_job.sh 5 1 sleep 1
./run_clip_job.sh 1 1 sleep 3
./run_clip_job.sh 1 1 ./ignore_term.sh
./run_clip_job.sh 1 1 nosuchcommand
./run_clip_job.sh 5 1 false

ok
limit reached, first signal sent; the command may still have finished
KILL sent after the grace period; treat output as incomplete
command not run: not found, not executable, or a shell function
command exited 1 on its own

The order of those lines matches the order of the calls: a quick success, a sleep that was stopped, the TERM-ignoring script, a missing command, and a command that failed by itself.

What I took from the tests:

• Read 124 as a sign the limit was reached, not that the job was stopped, and check the output separately.
• Handle 137 next to 124 the moment -k goes in.
• Treat 126 and 127 as configuration errors, not slow jobs.
• Do not pass shell functions to timeout; put the step in a script.
• Leave --foreground off unless the command needs the terminal, if background helpers should be covered.
• For a recorder, prefer INT or TERM with a grace period over KILL.

Things I did not test

• Newer coreutils. I ran 8.32. The quotes come from the current online manual, which documents newer releases. The exit-status list matched what I saw, but I did not run a newer build.
• Other implementations. BusyBox and the BSDs have their own timeout. Nothing here was run against them.
• systemd timers. Their own time-limit settings are a different mechanism and not covered here.
• Real camera footage. The recording test used a generated pattern at low resolution. A longer, larger recording may need more time to finish its file after the signal.

Sources. Quotes are from the GNU Coreutils manual page timeout invocation. Outputs are pasted unedited from the scripts above.

If you want an old Android phone recording in the background so there is footage for a job like this to process, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

timeout decides when to send a signal; what your job does with that signal, and what the exit status can tell you about it, is up to you to check.[shadow_engine] SHADOW_ENGINE_NO_EXPORT set — snapshot NOT written