My Similarity Check Let the Same Story Through 3 Times. Here's How I Killed It.

I run a content pipeline that picks trending topics and publishes articles automatically. Last week I found out it had published the same story three times. Not the same title — the same exact topic, reworded each time. My dedup check was supposed to stop that. It didn't. Here's why, and how I killed the check.

The Bug

My pipeline had a similarity gate. Every candidate title got compared against the last 30 published titles, and anything scoring 0.58 or higher was rejected. Straightforward, right?

from difflib import SequenceMatcher

def jaccard_bigram(a: str, b: str) -> float:
    def bigrams(s: str) -> set[str]:
        return {s[i : i + 2] for i in range(len(s) - 1)}
    x, y = bigrams(a), bigrams(b)
    return len(x & y) / len(x | y) if (x | y) else 1.0

def similarity(a: str, b: str) -> float:
    return max(SequenceMatcher(None, a, b).ratio(), jaccard_bigram(a, b))

THRESHOLD = 0.58

Here's the pair that slipped through. The candidate:

中国军队国际形象网宣片《当红》

And a title I had already published:

《当红》网宣片刷屏，普通人看到的中国军人是什么样

Same film. Same topic. Third time it was being covered. Watch what the algorithm did:

candidate = "中国军队国际形象网宣片《当红》"
published = "《当红》网宣片刷屏，普通人看到的中国军人是什么样"

print(similarity(candidate, published))
# SequenceMatcher: 0.187
# jaccard bigram:   0.185
# max:              0.187  < 0.58  ->  PASSED

0.187. The gate let it through with a five-fold margin to spare.

Why It Failed

The name 当红 is the same in both titles. That is the whole topic. But the algorithm does not care about that.

SequenceMatcher matches in order. In the published title, 当红 sits at position zero. In the candidate, it is at the end. Reordered tokens break the match, so the ratio collapses to the shared fragments — 网宣片 plus the generic words around it.

The bigram fallback does not save you either. Jaccard over character bigrams measures surface overlap, not meaning. Five shared bigrams out of twenty-seven total. 0.185. It "proves" the titles are unrelated because most of the characters happen to be different.

That is the whole failure mode in one line: it compares characters, not topics. Two headlines can share a single named entity — the actual subject — and still score like strangers.

The Fix

I did not tune the threshold. I did not add a better algorithm. I removed the algorithm from the decision entirely.

Human dedup is slower. It costs me an extra minute per article. I will take that trade forever, because the alternative shipped a duplicate to a live feed, and the only person who noticed was me, three articles later.

First, the selector script now declares dedup is a human step:

# 去重与选题是人工环节：agent 逐条对照候选——题材重复即弃。
# 禁止用相似度算法替代人工判断（相似度算法放行过同题材《当红》重复选题）。

Second — and this is the part I want you to steal — the pipeline's validation guard now asserts the old algorithm is gone. It runs on every change, and it fails the build if anyone ever sneaks a similarity check back in:

def require(condition: bool, message: str) -> None:
    if not condition:
        raise SystemExit(f"FAIL:{message}")

require(
    "similarity" not in select and "nearest >= 0.58" not in select,
    "obsolete similarity dedupe remains in select_topic",
)

The guard does not test whether the algorithm is correct. It tests whether the algorithm exists. That is a different job, and it is the one that actually failed us.

Why the Guard Works

Two lessons, both annoying.

First, a dedup algorithm you cannot explain to a human is a dedup algorithm you cannot trust. The similarity score was a black box. When it passed something wrong, nobody could look at the number and say "yes, that is a topic collision." The number said 0.187 and the editor said "same story again," and the number lost.

Second, automated gates need to be treated like the code they guard. Mine had no tests. It was not in the validation suite. The first time it was ever scrutinized was after the third duplicate went live. The fix was not a better dedup heuristic — it was moving the decision to a place where a human actually reads the titles, plus a contract test that makes the bad path un-buildable.

Gotchas

• This only works because my pipeline publishes a few articles a day. If you are doing thousands, "human checks every title" is not a solution. You would need real semantic dedup — embeddings, or entity extraction, not character n-grams.
• Removing the algorithm shifts the failure mode. Now a slip-up is a human miss, which is harder to catch in CI. The guard cannot stop a tired editor.
• Do not keep the old code around "for reference." The comment I left in the file is a warning, not a museum.

What about you?

Has an automated check ever let the exact thing it was supposed to block slip through? Did you fix it with a better algorithm, or with a worse but more honest one? I want to hear the failure mode — drop it in the comments.