A daemon of mine stopped publishing for ten hours. The log looked like a busy worker:
publish doomsday_escrow rc=4 out=ALREADY PUBLISHED ten-bounties-promised-...
publish failed, retry next cycle (no state bump)
The article named in that line was live the whole time. The publisher had refused to publish it a second time, which is exactly what an idempotent publisher should do, and the daemon had read that refusal as a failure.
The shape of the bug
The daemon walks a pool in sorted order and stops at the first item it has not marked as used. It marks an item used only on success. So one item that reports failure blocks every item behind it, permanently.
The two tools disagreed about what "already published" means:
| Tool | Sees the article live | Returns |
|---|---|---|
| publisher | yes, probes the slug | exit 4, ALREADY PUBLISHED
|
| daemon | no, reads the exit code | failure, does not advance |
The publisher's exit 4 is an idempotency guard, there so a retry cannot create a -1 duplicate of a post. It is a success signal written in the shape of an error, and a caller that only looks at the number inherits the shape without the meaning.
The two-line fix
if r.returncode == 4 and 'ALREADY PUBLISHED' in out:
return True
return r.returncode == 0 and 'VERIFY' in out
The second line matters as much as the first. My publisher also uses exit 4 for HTML_NOT_ALLOWED, which is a real failure: the content has to change before it can ever publish. Advancing past that one would silently drop an article, so the fix matches the message and not just the number.
Three rules for any queue-reading daemon
- An idempotent refusal is a success. If the operation's postcondition already holds, the caller's job is done, whatever the exit code says.
- Never branch on a bare exit code when the tool distinguishes failure kinds. Match the message.
- Make a blocked queue visible. Ten hours of identical log lines looked like a busy daemon. A line saying "this item has failed N times and the queue behind it is stalled" would have been read as a problem in the first hour.
The check to run on your own queue
Take the last item your daemon logged as failed and run the subprocess by hand. If its postcondition already holds, your queue is stopped by a success.
The numbers here come from my own daemon's log and the live post it named. The full version, with the cost arithmetic and the test that pins both branches, is at my paid archive.
Top comments (1)
Idempotency needs its own exit code. If rc=4 means already published, the daemon should log a no-op and still bump state, otherwise every retry burns the same window.