A nightly script encodes four camera clips at once by putting each encoder in the background and then calling wait. One of the four fails. The script exits with status 0, nothing that checks its exit status notices, and the broken clip turns up whenever someone next opens it.
The four encoders here are a stand-in script that sleeps and then exits with whatever code it is given, so a failure can be planted on purpose:
#!/bin/bash
# Stand-in for an encoder: fake_encode.sh NAME SECONDS EXIT_CODE
sleep "$2"
echo "finished $1, exit $3"
exit "$3"
#!/bin/bash
# Four encodes in parallel. c.mp4 fails with exit 1.
./fake_encode.sh a.mp4 0.4 0 &
./fake_encode.sh b.mp4 0.1 0 &
./fake_encode.sh c.mp4 0.3 1 &
./fake_encode.sh d.mp4 0.2 0 &
wait
echo "wait returned $?"
finished b.mp4, exit 0
finished d.mp4, exit 0
finished c.mp4, exit 1
finished a.mp4, exit 0
wait returned 0
c.mp4 exited 1 and said so. wait returned 0 anyway.
Disclosure: I make an Android camera app. Nothing below uses its code or describes how it records or processes anything. The encoder is the fake one above, the clips are only names, and every output is pasted from the scripts shown. Versions:
GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)
Linux 6.8.0-138-generic
What the manual says plain wait returns
The Bash manual is direct about this case. With no arguments, wait "waits for all running background jobs and the last-executed process substitution, if its process id is the same as $!, and the return status is zero".
So the 0 is not a lucky race or a bug. Plain wait is a barrier. It holds the script until every background job is done, and it reports nothing about how any of them ended. The exit codes are thrown away unless something else asks for them.
That is easy to miss because the form with an argument behaves the way people expect from plain wait. The first sentence of the same entry: "Wait until the child process specified by each id exits and return the exit status of the last id." The status only comes back when you name the job.
set -e does not catch it either
A common reflex is to put set -e at the top and assume any failure will stop the script. I ran two versions with set -e, one with plain wait and one that waits on the failing job by PID:
#!/bin/bash
set -e
./fake_encode.sh a.mp4 0.2 0 &
./fake_encode.sh c.mp4 0.1 1 &
wait
echo "set -e did not stop the script; wait returned $?"
#!/bin/bash
set -e
./fake_encode.sh a.mp4 0.2 0 & pa=$!
./fake_encode.sh c.mp4 0.1 1 & pc=$!
wait "$pc"
echo "this line never prints"
#!/bin/bash
./set_e.sh; echo "set_e.sh exit status: $?"
./set_e_pid.sh; echo "set_e_pid.sh exit status: $?"
sleep 0.3 # let the orphaned a.mp4 job finish before the next command
finished c.mp4, exit 1
finished a.mp4, exit 0
set -e did not stop the script; wait returned 0
set_e.sh exit status: 0
finished c.mp4, exit 1
set_e_pid.sh exit status: 1
finished a.mp4, exit 0
Three things show up in that output:
-
With plain
wait,set -edid not fire. The script printed its last line and exited 0. Plainwaitreturned 0, and the failed job did not produce a nonzero status anywhere in the script itself. -
Waiting on
$pcby PID,set -edid fire. Thatwaitreturned 1,set -etreated it like any other failing command, and the script exited 1 before its last line. -
The other job kept running after the script exited.
finished a.mp4printed afterset_e_pid.sh exit status: 1. Exiting early does not stop background jobs. If the job is an encoder writing to a file, that file is still being written after the script has reported failure.
The last point matters if the script does cleanup at the end. A script that stops at the first failed wait and leaves a sibling running can have a half-written clip next to a finished one.
Keep every PID and wait for each one
The fix that needs nothing beyond plain Bash is to remember each PID as the job starts, then wait for them one at a time and collect the failures:
#!/bin/bash
# Keep every PID, then wait for each one by PID.
declare -A clip
./fake_encode.sh a.mp4 0.4 0 & clip[$!]=a.mp4
./fake_encode.sh b.mp4 0.1 0 & clip[$!]=b.mp4
./fake_encode.sh c.mp4 0.3 1 & clip[$!]=c.mp4
./fake_encode.sh d.mp4 0.2 0 & clip[$!]=d.mp4
failed=()
for pid in "${!clip[@]}"; do
wait "$pid" || failed+=("${clip[$pid]}")
done
if (( ${#failed[@]} )); then
echo "failed: ${failed[*]}"
exit 1
fi
echo "all encodes succeeded"
finished b.mp4, exit 0
finished d.mp4, exit 0
finished c.mp4, exit 1
finished a.mp4, exit 0
failed: c.mp4
$! holds the PID of the job just started, so saving it on the same line keeps the PID and the clip name together. The loop then waits for every PID, even after one has failed, so the script does not exit while an encode is still running. It ends by naming the failed clip and returning 1, which is what cron and anything else downstream needs to see.
The order the loop waits in does not matter. If it waits for a.mp4 first and c.mp4 has already finished, the result of c.mp4 is still there when the loop gets to it.
A job that already ended still has a status
That last claim is worth checking, because it is what makes the PID loop safe. Here the job exits with 3 immediately and nobody waits for it until a second later:
#!/bin/bash
# The job exits with 3 long before anyone waits for it.
./fake_encode.sh x.mp4 0 3 > /dev/null & p=$!
sleep 1
wait "$p"; echo "wait on a job that ended a second ago: $?"
wait "$p"; echo "the same PID a second time: $?"
wait on a job that ended a second ago: 3
the same PID a second time: 3
The status was still there after the job ended, and in this Bash it was still there on a second wait for the same PID. I would not build on the second result. The point that matters is the first one: a script can start its jobs, do other work, and collect the results afterwards without losing any.
wait -n for a limit on how many run at once
Four encodes at once is fine for four short clips. For a folder of a hundred, the usual goal is to keep a fixed number running and start the next one whenever one finishes. wait -n does that part. With no PIDs given, the manual says it waits for any job or process substitution "to complete and returns its exit status".
wait -n on its own says how a job ended but not which job it was. The -p option fills that gap by putting the PID in a variable, and the manual is clear about where it applies: "This is useful only when used with the -n option."
#!/bin/bash
# At most two encodes at once. wait -n returns as soon as any one finishes;
# -p names the PID it is reporting on.
declare -A clip
running=0 fails=0
reap() {
local who rc
wait -n -p who; rc=$?
echo "reaped ${clip[$who]}: $rc"
(( rc != 0 )) && fails=$((fails + 1))
running=$((running - 1))
}
for spec in "a.mp4 0.4 0" "b.mp4 0.1 0" "c.mp4 0.2 1" "d.mp4 0.2 0"; do
set -- $spec
(( running == 2 )) && reap
./fake_encode.sh "$1" "$2" "$3" > /dev/null & clip[$!]=$1
running=$((running + 1))
done
while (( running > 0 )); do reap; done
echo "failures: $fails"
reaped b.mp4: 0
reaped c.mp4: 1
reaped a.mp4: 0
reaped d.mp4: 0
failures: 1
Two encodes run at a time. b.mp4 finished first, so c.mp4 started in its place. c.mp4 finished next, and failed, so d.mp4 started. The script reports the failure by clip name and still waits for the last two.
The while loop at the end drains whatever is still running. Without it the script would leave the last jobs unreaped, which is the same problem as the plain wait version.
Two status values worth knowing on sight
127. The manual says wait -n returns 127 "if no arguments are supplied and the shell has no unwaited-for children", and that a PID which is not a child of the shell also gives 127:
#!/bin/bash
wait -n; echo "wait -n with no children: $?"
wait 99999; echo "wait on a PID that is not our child: $?"
wait -n with no children: 127
./nothing.sh: line 3: wait: pid 99999 is not a child of this shell
wait on a PID that is not our child: 127
A drain loop that counts 127 as one more failure will report a failure that did not happen. The loop above avoids that by counting the jobs it started, so it does not call wait -n with nothing left.
128 and up. A job stopped by a signal comes back with a status above 128, and kill -l turns that status back into a signal name:
#!/bin/bash
# A long job that gets stopped with the default signal of kill (SIGTERM).
sleep 30 & p=$!
sleep 0.2
kill "$p"
wait "$p"; rc=$?
echo "status $rc, signal $(kill -l "$rc")"
status 143, signal TERM
143 is SIGTERM, the signal kill sends by default. If an encode was killed by a timeout, a supervisor or a person at a terminal, this is the status the script sees, and it is worth logging as a stop rather than a failure inside the encoder.
What to change in a script like this
- Replace a bare
waitat the end of a parallel section with a loop over saved PIDs, or with await -nloop if the number of jobs running at once needs a limit. - Name the failed job in the output. A script that only exits 1 still leaves someone guessing which clip broke.
- Wait for every job before exiting, even after a failure, so nothing is still writing when the script reports.
- Treat 127 from
waitas a script bug, not an encode failure, and a status above 128 as a stop by signal. - Do not count on
set -eto notice a background job. It only acts on a status that reaches the script, and plainwaitreturns 0.
Things I did not test
- Other Bash versions. I ran 5.1. The quotes come from the current online manual. Everything I ran matched it, but I did not run a newer or older Bash.
-
Other shells.
sh,dash,zshand others have their ownwait. I did not test any of them, and the-nand-poptions above are taken from the Bash manual only. -
Job control. These are scripts, where job control is off by default. The
-foption and job specs like%1behave differently with job control on, and I did not test them. - Real encoders. The jobs here sleep and exit. A real encoder may trap signals or exit with codes of its own, so check what it returns before reading meaning into a status.
Sources. Quotes are from the GNU Bash Reference Manual page Job Control Builtins (the wait entry). Outputs are pasted unedited from the scripts above.
If you want an old Android phone recording in the background so there are clips for a job like this to process, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.
A bare wait returns 0 however its jobs ended, so save each $! and wait on every PID, or loop on wait -n -p when the count needs a limit, and read a 127 as a bug in the script rather than a failed clip.
Top comments (0)