My Comment-Dedup Check Used "in" on a Whole Markdown File. A Date in a Sentence Broke It.

I have a small script, reply_comments.py, that keeps my DEV.to comment-reply backlog from turning invisible. It runs in an unattended pipeline twice a day: pull every comment on my articles, skip the ones I've already replied to, skip the ones I've already drafted a reply for, and print whatever's left as JSON so I can act on it. The dedup for "already drafted" is supposed to be simple: every drafted reply gets a markdown header in drafts/comment_replies.md like ## 3b908 — mads_hansen on "...", keyed by the comment's id_code. If a comment's id_code has a header, it's drafted. If not, it's pending.

Here's the line that actually did that check, until today:

def pending():
    try:
        drafted = open(DRAFTS, encoding="utf-8").read()
    except FileNotFoundError:
        drafted = ""
    out = []
    for a in api(f"/articles?username={ME}&per_page=100"):
        if not a["comments_count"]:
            continue
        for c in api(f"/comments?a_id={a['id']}"):
            if c["user"]["username"] == ME or replied_by_me(c):
                continue
            if c["id_code"] in drafted:
                continue
            out.append({...})

drafted is the raw text of the entire file. c["id_code"] in drafted is a substring check against that raw text, not a check against a set of actual header id_codes. I wrote it this way originally because it was the two-second version and it happened to work on every real case I tested it against.

The bug is that "already appears as a header" and "already appears anywhere in this file" are not the same claim, and the file is full of things that aren't headers. Every drafted reply in comment_replies.md quotes the commenter's original text and includes my own reply prose underneath the header. That prose is full of ordinary words, numbers, and dates. dev.to id_codes are short lowercase alphanumeric strings — things like 3b908, 3bdja, 39pk4 — and nothing stops a short alphanumeric token from showing up naturally inside a sentence.

I went looking for a live example instead of a hypothetical one. My own draft file has a reply that talks about FTS5 and BM25 scoring, and elsewhere just talks about the current year, 2026. 2026 is exactly the shape of a real id_code: four lowercase-alphanumeric characters, digits included. So I ran the actual check the function uses, against my real file, with a fake id_code that has never been a header anywhere in that file:

import re
text = open("drafts/comment_replies.md", encoding="utf-8").read()
headers = set(re.findall(r"^## (\S+)", text, re.M))

fake_new_code = "2026"
print(fake_new_code in headers)   # False — never actually drafted
print(fake_new_code in text)      # True — but the old check would skip it anyway

False
True

If a brand-new comment ever showed up on one of my articles with the id_code 2026, pending() would silently treat it as already handled and never print it. Nobody would see it in the JSON output. Nobody would draft a reply. And there'd be no error, no log line, nothing — the comment would just quietly not exist as far as the pipeline is concerned, forever, because a completely unrelated sentence somewhere else in the file happened to contain the same four characters.

What made this worth writing up rather than shrugging off as "unlikely" is that I already have the counter-example sitting right there in the same file, in the same function's neighbor. This script also has an audit() command, added a couple of days ago, that cross-checks drafted-but-never-actually-posted replies. audit() extracts id_codes the correct way:

def audit():
    ...
    drafted_codes = set(re.findall(r"^## (\S+)", drafted_text, re.M))

Anchored to the start of a line, matched against the exact ## <code> header format, collected into a set. That's the right way to ask "is this id_code one we've drafted." pending(), sitting forty lines above it in the same file, was asking a different and looser question — "does this string of characters occur somewhere in this document" — and getting away with it purely because no real comment had collided yet.

The fix is to make pending() use the same extraction audit() already uses:

def pending():
    try:
        drafted_text = open(DRAFTS, encoding="utf-8").read()
    except FileNotFoundError:
        drafted_text = ""
    drafted_codes = set(re.findall(r"^## (\S+)", drafted_text, re.M))
    out = []
    for a in api(f"/articles?username={ME}&per_page=100"):
        if not a["comments_count"]:
            continue
        for c in api(f"/comments?a_id={a['id']}"):
            if c["user"]["username"] == ME or replied_by_me(c):
                continue
            if c["id_code"] in drafted_codes:
                continue
            out.append({...})

Same shape, same cost, and now it can't be fooled by a coincidence in the prose. Reran the collision test against the fixed logic:

drafted_codes = set(re.findall(r"^## (\S+)", text, re.M))
print("2026" in text)            # True  (old check)
print("2026" in drafted_codes)   # False (new check)

What bugs me about this one is that it's not a rare edge case in some adversarial input — it's ordinary language sitting in a file this exact pipeline generates every single day. The file's job is to accumulate quoted human comments and my own replies about them, which means it will keep accumulating short alphanumeric tokens that look exactly like id_codes: years, version numbers, hex-ish abbreviations, whatever a commenter happens to type. The longer the file grows, the more surface area there is for a genuine collision, and the failure mode is total silence — a comment that never gets surfaced isn't a crash you notice, it's a gap you'd only find by manually re-diffing the live thread against the file, which is the exact manual step this pipeline exists to replace.

The audit() command I wrote two days ago was built to catch "drafted but never posted." It never occurred to me to point the same scrutiny at "pending but never even detected as pending," because that failure doesn't leave a header behind to audit against — it leaves nothing at all. If I hadn't happened to reread pending() right next to a helper that does the same lookup correctly, I don't think I'd have caught it from the outside.