A retry loop that couldn't tell the checker crashed from the task not being done

#bash #retry #debugging #automation

A retry loop that couldn't tell "the checker crashed" from "the task isn't done"

I run a completeness checker that a scheduler calls after every task to decide whether that task is finished. The checker returns a non-zero exit code in two completely different situations, and my retry loop was treating them the same.

Exit code 1: the checker ran fine, judged the task, and the task is missing something. Its own output always contains the literal string "not done: <what's missing>".
Exit code 2: the checker refuses to judge at all — usually because the caller forgot a required argument.

Here is an exit code 2, produced by leaving out a required --since flag:

$ python3 scripts/completeness_check.py my-task 2026-08-21
exit=2
[checker] refusing to judge: no --since <marker file> given.
  Without a defined window this can pick up output from an unrelated run
  and pass an idle pass as complete. The scheduler always passes it;
  a manual run needs a marker file touched first.

That message never contains the words "not done". It is the checker saying "I can't tell", not "you're missing something".

What the old retry loop did

The old logic in my scheduler script checked one thing: was the checker's exit code zero. Anything else got treated as "the task isn't finished, retry it" — including a checker that refused to run because of a missing flag. The retry fed the checker's own error message back into the task's next attempt as if it were a list of missing work. The task had nothing to fix. What was broken was the way the scheduler called the checker, not the task itself.

Root cause

An exit code only carries one bit: zero or not zero. "Failure" here actually has two distinct meanings — the thing being checked is not good enough, or the checker itself couldn't render a verdict. Collapsing both into the same non-zero signal removes the caller's ability to tell "retry the task" from "fix how you're calling the checker".

The same shape got caught three separate times in one review pass, and each fix just added another item to a list of "here's what a broken checker call looks like" — first "exit code 2", then "the output contains a traceback". Both missed a third shape: the checker calling a hard exit with code 1 and no traceback at all, which still isn't "not done".

The fix

Instead of enumerating every way the checker can fail to judge, check for the one positive signal that means it actually did judge and found something missing:

# the checker prints "not done:" from exactly one line in its own source,
# whenever it has actually rendered a not-done verdict
if [ "$rc" -ne 0 ] && ! printf '%s' "$why" | grep -q "not done:"; then
  # non-zero, but no not-done marker -> the checker itself is broken,
  # not the task. File an incident, don't retry the task.
  ...
fi

Anything with the marker string is a real "not done" and gets retried. Everything else — regardless of exit code — is "can't judge", and gets an incident filed instead of a retry. New failure shapes from the checker don't need a new branch; they fall into "can't judge" by default because they lack the one positive marker.

What I actually verified

I reproduced the exit-code-2 case for real — ran the checker without --since and got the output shown above. I then ran the new classification logic against that exact output by hand: no "not done:" marker present, so it classifies as "can't judge", not retried. Before the fix, the same output would have been read as "not done, retry" — I traced that branch by hand too, since I didn't want to actually re-trigger a real retry to prove a negative.

What this does not prove

I did not exhaustively enumerate every exit path the checker can take and confirm each one lands where I expect — only the one shape I had in hand (missing --since). The fix's design, positive matching instead of enumeration, is meant to make that unnecessary, but I have not verified it against a checker that has been running long enough to grow new failure shapes I have not seen yet.

What I check now

For any external checker or validator wrapped in a retry loop: does its output give me a way to tell "the subject is invalid" apart from "the checker itself couldn't run"? If the only signal is a bare exit code, that is the early warning sign — a retry loop built on exit code alone will eventually retry something that was never actually broken.

This article was written with the help of AI. The checker, the retry loop, and the exit codes described are real, from my own scheduler. The bash snippet above is translated out of the original — the real marker string is in Chinese — and I re-ran the classification logic by hand against the real reproduced output before writing this, rather than pasting a raw terminal transcript.

DEV Community