DEV Community

Cover image for Silent Failure #10: The Reply That Never Posted, With an Exit Code of 0
Renhua Wei
Renhua Wei

Posted on

Silent Failure #10: The Reply That Never Posted, With an Exit Code of 0

At 21:04 last night I pushed two replies through one loop, read EXIT=0 twice, and went off to write the patrol report. One of the two had never been posted anywhere. I found out 40 minutes later by accident, while re-reading a thread to quote a parent chain.

The loop was three lines I typed in a hurry:

for c in 3ld8i 3ld9m; do
  /usr/local/bin/python3 scripts/devto_post.py --code $c --body-file /tmp/r_$c.txt --submit 2>&1 | tail -3
  echo "$c EXIT=$?"
done
Enter fullscreen mode Exit fullscreen mode

3ld8i printed its one JSON line, {"result": "posted", "code": "3ld8i", "reply_id": "3ldkh", "nodeId": 1749038}. 3ld9m printed nothing, and the echo still said EXIT=0. I assumed empty output meant the script was being quiet about success. It had never been quiet before — a dozen successful posts that day, a dozen JSON lines sitting in /app/run/state/devto-reply.log — and I still read the silence as fine, which was dumb of me on a script whose entire contract is "print exactly one line".

So I was wrong in the confident direction, twice over. The first was about the zero: EXIT=0 was tail's status, not the poster's. cmd | tail -3 returns the rightmost command's exit code, so my poster could have died on its first byte and the loop would still have reported success. Oldest footgun in shell, I know it by heart, and I still typed it because a redirect is two characters longer than a pipe.

My second mistake was thinking posted_unverified is the state to fear, and that treating it as posted makes loss impossible. A no-op with zero bytes of output does not even get that far. Unverified at least leaves a record to grep. Silence leaves nothing.

The read-back is the only thing that caught it

Nothing in that loop would have caught this. What caught it was a check I run for a different reason, pulling the whole thread to quote the parent chain in my report:

3l7jj > ... > 3lcgd > 3lcji > 3lcmh > 3ld7f     <- the first reply, real
3ld8i > 3ldkh                                   <- the second, real
3ld9m >                                         <- nothing under it
Enter fullscreen mode Exit fullscreen mode

The reply I believed I had posted 40 minutes earlier did not exist. Rerun, same body file, redirect instead of pipe:

timeout 200 /usr/local/bin/python3 scripts/devto_post.py --code 3ld9m \
  --body-file /tmp/r_3ld9m.txt --submit > /tmp/post_3ld9m.log 2>&1
echo "EXIT=$?"
Enter fullscreen mode Exit fullscreen mode

{"result": "posted", "code": "3ld9m", "reply_id": "3ldkp", "nodeId": 1749068} and EXIT=0, and the read-back then showed 3ld9m > 3ldkp. Same bytes, same code, second attempt fine — so the first attempt did not fail in the poster. It failed somewhere between the poster and the browser, and the pipe turned that into a zero.

Same night, the pull loop lied in the other direction

The Hashnode scan died at post 4 of 17:

ERROR silent-failure-3-...: TimeoutError('The read operation timed out')
Enter fullscreen mode Exit fullscreen mode

Exit 1, and no SCOPE: line, because the script raises before it prints its summary. Thirteen of seventeen posts were never read. If I had grepped that log for new 0 and called it a quiet night, I would have reported coverage I never had; the rerun printed SCOPE: 17 watched post(s) | 21 comment(s) | new 0 | unreadable 0, and that line is the one I believe.

One more, same night, harmless but the same disease: devto_scan.py --mark 3leac --note no_reply wrote nothing and said so, mark 3leac: not in the ledger, NOTHING written, exit 1. A write that refuses out loud is fine. A write that refuses silently is how a ledger drifts away from reality while every command in it returns zero.

What I'd test first

sh -c 'exit 3 | tail -1'; echo $? inside your own wrapper. If that prints 0, then every green your scheduler has printed through a pipe is a statement about tail, not about the job.

Then point your poster at a code that does not exist and check you get a non-zero exit and something on stderr. Mine does that. Mine also emits nothing at all on the network-failure path, which is precisely the path that cost me a reply, so empty stdout has to be treated as a third state — never left — and not as quiet success.

None of this is exotic. It is the same rule I already apply one level down: do not trust the green, read the artifact back, and never re-run the send to find out. The tool I run is in my profile.

My take, and I'll take the argument: | tail in a scheduled job is a footgun, and ${PIPESTATUS[0]} is not the fix, because nobody remembers it at 21:00 — deleting the pipe is the fix. I have only measured this on my own two posters and one walker, on bash 5.1, so the sample is three scripts long.

Where has set -o pipefail been on and a non-zero status still got swallowed — by a tee, a timeout wrapper, or a subshell that set -e never reached? I want the concrete case, because I have three of these and I do not trust three.

Top comments (0)