Six days ago I built a pipeline for a problem I'd found by accident: 35 published articles, 20 comments, none replied to. The DEV.to API can't help you post a reply — POST /api/comments returns a 404, and POST /api/reactions returns a 401 even with a valid key, both verified live. So the fix I shipped was a draft-only pipeline: a script that pulls unreplied comments, an agent that writes context-grounded replies into drafts/comment_replies.md, and a note at the top of that file telling a human to paste each reply at its link and then delete the entry.
That file has been running twice a day for six days. Today, going back through this repo for material, I actually checked whether any of it had worked.
It hadn't. I cross-referenced every id_code currently sitting in drafts/comment_replies.md against the live comment thread on DEV.to — walking each comment's children array looking for a reply authored by enjoy_kumawat, the same check reply_comments.py's own replied_by_me() already does for the pending side of this pipeline. Thirty-six drafted replies. Zero of them have a matching reply on-site.
def audit():
drafted_codes = set(re.findall(r"^## (\S+)", open(DRAFTS).read(), re.M))
unposted = []
for a in api(f"/articles?username={ME}&per_page=100"):
if not a["comments_count"]:
continue
for c in api(f"/comments?a_id={a['id']}"):
if c["id_code"] not in drafted_codes:
continue
if not any(ch["user"]["username"] == ME for ch in c["children"]):
unposted.append(c["id_code"])
return {"drafted": len(drafted_codes), "never_posted": unposted}
{"drafted": 36, "never_posted": 36}. Not "most." All of them.
The reason is obvious once you say it out loud, and I still missed it for six days: the paste step assumes a human periodically opens the file. Nobody does. This entire pipeline — the comment-drafter and the article-publisher both — runs as a scheduled routine with nobody watching. I wrote "paste manually via each comment link, then delete the entry" as an instruction to a person, then handed the whole loop to something that has no hands and isn't a person, and never noticed the contradiction because every run since then reported success. And it wasn't lying: the drafting step really does succeed every time. It just isn't the step anyone asked for. The comment was still unanswered; a well-written answer just now existed somewhere nobody would read it either.
What made this invisible for six days is that the pipeline's own dedup logic can't distinguish "replied" from "drafted." pending() skips a comment if its id_code shows up anywhere in drafts/comment_replies.md — which is correct for not drafting the same reply twice, but it means the file filling up looks identical, from inside the system, to the backlog actually shrinking. The only signal that would have caught it — checking the live thread for a reply from enjoy_kumawat — already existed in replied_by_me(). I had the check. I was just running it against the wrong question: "should I draft this" instead of "did drafting this do anything."
I can't fix the actual gap by writing more code — the API still won't let this account post a comment, so there's no way to close the loop end-to-end without a human in it somewhere. What I can fix is the pipeline's blindness to its own state. I added an audit command to reply_comments.py that runs exactly the check above:
if sys.argv[1:2] == ["audit"]:
load_env()
print(json.dumps(audit(), indent=2))
It doesn't paste anything and it doesn't delete anything. It just tells the truth: how many drafted replies exist, and how many of them still have no reply on-site. Six days ago I built a system that could report "I drafted a reply" every single run and be completely correct about it while the actual problem — comments sitting unanswered — kept growing underneath that report. audit is the difference between a pipeline that reports its own activity and one that reports the state of the thing it was supposed to change. Those aren't the same measurement, and I'd built a whole automation around only ever computing the first one.
There's a smaller version of this bug pattern I keep running into in this exact repo: a step that's easy to automate (draft a reply, generate a commit message, publish an article) sits right next to a step that isn't (paste it, sign it, verify a human wanted it published), and the automated half quietly starts reporting on behalf of the whole pipeline because it's the half that always succeeds. The unautomated half doesn't fail loudly — it just never happens, and nothing was ever watching the difference between "the easy part ran" and "the actual outcome occurred." replied_by_me() was sitting right there the whole time. I just never pointed it at my own drafts until a comment-count of thirty-six made it worth checking.
The next run of this pipeline is going to call audit before it drafts anything new, and log the unposted count next to the pending count in docs/project_notes/issues.md, the same work log this finding came from. Not because a number in a log file fixes anything on its own — it doesn't paste a single reply — but because a growing, unreported backlog is a worse failure mode than a growing, reported one. Six days of silent zero-posts only surfaced because I happened to go looking for article material and picked up a file I'd otherwise have kept trusting. The fix isn't cleverness, it's just making the pipeline check the one thing it was actually built to accomplish, instead of the one thing it always successfully does.
Top comments (0)