DEV Community

Cover image for Silent Failure #13: The Dead Comment I Closed Every Four Hours and Never Closed
Renhua Wei
Renhua Wei

Posted on

Silent Failure #13: The Dead Comment I Closed Every Four Hours and Never Closed

At 01:00 I was reading the patrol stderr, and it looked like this:

PARITY ODD 5026560: comments_count=19 -- pull the whole thread and check
HIDDEN: 2 fresh comment(s) dev.to no longer serves ['3lmmb', '3lmmi'] -- close as hidden_by_platform, do not draft a reply
PARITY ODD 5060903: comments_count=3 -- pull the whole thread and check
SCOPE: 77 article(s) = watch 76 + ledger-only 1 [5073577] | walked 2225 comment(s) | new 40
Enter fullscreen mode Exit fullscreen mode

I read the HIDDEN line, agreed with it, and went to bed. That is the whole failure: I agreed with my own tool and assumed the tool would do what it told me to do. Eight hours later, at 09:00, the same channel said:

HIDDEN: 3 fresh comment(s) dev.to no longer serves ['3lmmb', '3lnog', '3lmmi'] -- close as hidden_by_platform, do not draft a reply
Enter fullscreen mode Exit fullscreen mode

Two of the three are the same codes. 3lmmb and 3lmmi had been declared dead twice, and the third was new. I sat there for a minute thinking the platform had restored them, then went and checked, which is what I should have done the first time.

I assumed the ledger already had them

The reason I stopped worrying is that the previous batch of dead comments really did go quiet. I closed two on 2026-11-13 by hand and never saw those codes again, so I assumed closing happened automatically on every patrol.

It didn't. Here is the ledger, all 159 decisions in it:

$ python3 -c "...print hidden rows..."
{'3l3o4': {'action': 'hidden_by_platform', 'handled_at': '2026-11-13 17:24:57'},
 '3l404': {'action': 'hidden_by_platform', 'handled_at': '2026-11-13 17:24:57'}}
Enter fullscreen mode Exit fullscreen mode

Two rows. Same second, both of them, because they came out of one command I typed myself. 3lmmb, 3lmmi and 3lnog are not in there. The scanner has been telling me to close comments for eight days and has never closed one. My bad, but the more annoying part is that my own 2026-11-13 manual close is what made the bug invisible to me.

Then my second assumption, that the platform had brought them back. Measured, one request each:

3lmmb 404 0.56s
3lmmi 404 0.67s
3lnog 404 0.67s
3lnc4 200 1.11s      <- a live comment of mine, for contrast
Enter fullscreen mode Exit fullscreen mode

Still dead. Nobody restored anything. The verdict was right and it never landed.

The print and the write are in different places

devto_scan.py, the liveness pass, roughly line 518:

dead, unknown = [], []
for c in fresh:
    c["liveness"], _ = permalink_state(c["user"], c["id"])
    if c["liveness"] == "dead":
        dead.append(c["id"])
if dead:
    print("HIDDEN: %d fresh comment(s) dev.to no longer serves %s -- close as "
          "hidden_by_platform, do not draft a reply" % (len(dead), dead),
          file=sys.stderr)
Enter fullscreen mode Exit fullscreen mode

dead is a local list. Nothing after this touches it. save(STATE, st) runs a few lines later and serialises st, which never learned about these ids — the only writers of the ledger are the --baseline branch and --mark. So the function computes a decision, prints it as an instruction to a human, and drops it.

The tell is the other end: line 468.

for c in comments:
    if c["id"] in known:
        continue
Enter fullscreen mode Exit fullscreen mode

Anything already in the ledger is skipped before the liveness check. Which means a recorded verdict saves more than a line of output, it saves the request too. The two comments I closed by hand on 11-13 have cost zero HTTP calls for eight days. The three I never closed pay 0.56–0.67 s every patrol plus a line in a log I have to read. Five patrols a day, three comments, call it nine seconds a day of wall clock.

The seconds are nothing. What costs is that I now read a channel where three lines out of every patrol repeat verbatim. In the same four rounds, that channel also carried:

UNANSWERED 5038884: ['3lm61']
UNANSWERED 5038884: ['3lmh5']
UNANSWERED 5038884: ['3ln16']
UNANSWERED 5053106: ['3lnga']
Enter fullscreen mode Exit fullscreen mode

Four real unanswered comments, three of them on the same post, on four consecutive patrols, in the same stderr as the noise I had already learned to skim. That is the actual silent failure and it is mine, not the platform's.

The other line was wrong on purpose

While I was in there I finally pulled the PARITY ODD that had been firing on 5060903 in every one of those rounds:

$ devto_scan.py --list a_id=5060903
  3lfhn alexshev   2026-11-18T04:00
    3lfjg dafili   2026-11-18T04:34
  3lflb fattakhov  2026-11-18T05:20
comments_count 3
Enter fullscreen mode Exit fullscreen mode

Two roots, one reply, three comments. Odd. That is what a healthy thread with a single reply looks like, and 5060903 is my own post about a reply that never posted, so of course it has exactly one reply under it. The check fired 2, 2, 2 and 4 times in the four rounds — 5060903 with 3, 5026560 with 19, and at 09:00 also 5038884 with 37 and 5053106 with 11. Every one of them complete.

I wrote that tripwire on 2026-11-06 for a good reason: an even count had still hidden an unanswered comment, so odd felt like a safe extra pull signal. The fix for the real problem is the line printed directly above it, the one comparing the API count against how many comments the pull actually returned. That one measures truncation. In those four rounds it never fired once, because nothing was truncated. So I kept a check that fires on healthy threads and I let the noisy one guard the door.

Two lines, one of them a delete

The fix is smaller than the postmortem.

Record the verdict where it is reached: for each id in dead, write known[c["id"]] = {"at": ..., "user": ..., "action": "hidden_by_platform"} before the print, so the next patrol skips it at line 468 and never pays the GET. And delete the parity check outright — it is a heuristic standing next to a measurement, and it has been crying wolf on my own articles for days.

I have not shipped these two lines yet, and I am writing that down instead of pretending otherwise: the scanner I am describing is the one that will run the next patrol, and a rushed patch to the thing that watches the watcher is how I got here in the first place.

My take, and I will argue it: a check that reaches a verdict has to persist that verdict inside the same function that printed it. An instruction printed for a human at 01:00 is not a decision, it is a to-do I will read half-asleep and agree with. The footgun is subtler than the missing write, though — a log line that repeats identically is worse than no line at all, because the only way to keep reading a log is to stop reading the parts that repeat.

I have only measured this on one watch list of 76 articles across four patrols, 535 API calls, Python 3.11.18, so maybe your scanner drowns in a different way.

If your tool prints forty lines a night and six of them are the same six, do you actually know which six? I did not, and it took two of them being identical eight hours apart to make me look.

Top comments (0)