De-duplicating Scraped Posts: The Two-Hash Trick That Replaced My Fuzzy Matching

The dedup step in my daily scraper used to be the clever part of the pipeline. It is now the boring part, and the pipeline is better for it.

Context: I pull public Telegram channel posts every morning (~a few hundred messages) and ship the day as a data bundle. Dedup matters twice: inside a run (the crawler revisits pages) and across runs (channels re-post, pins resurface, someone quotes a post verbatim in three channels).

The version that ate my weekends

Original design: normalize whitespace, lowercase, compare text similarity with a token overlap ratio, treat > 0.9 as duplicate. Fuzzy matching felt rigorous. It was a trap:

• Threshold tuning is endless. 0.9 caught re-posts; it also merged two posts that said "the update is out" and "the update is not out." Negation survives token overlap perfectly.
• It is O(n²) per comparison window with string-distance costs — invisible at 100 rows, ugly at 10,000.
• It is not explainable. "Why did you drop this row?" "Similarity 0.91." That answer does not survive a buyer's scrutiny.

The version that ships: two hashes, no cleverness

Every row now carries two cheap, exact, explainable keys:

1. Content hash — sha256 of the raw stored bytes. Catches byte-identical revisits, which is 95% of the duplication. Exact, O(1) lookups against a set of seen hashes, and "dropped as exact duplicate" is a sentence a buyer can verify by recomputing.
2. URL key — the canonical post URL. Channels syndicate: the same text appears at a new URL. The content hash misses it, so a second lookup checks "have I ever shipped this URL?" A re-post with a new URL and identical content is kept on purpose — syndication is signal, and I'd rather ship two rows with a seen_before: true flag than silently drop information.

Text near-duplicates (the fuzzy-matching dream) are not merged at all. They are flagged. A flag is reversible; a drop is not.

Why exactness wins at this scale

The honest accounting: fuzzy matching solved a problem I did not have (aggressive merging of paraphrases) and created one I did (silent false merges, opaque verdicts, quadratic cost). Exact hashes answer the two questions the product actually asks — "is this the same bytes?" and "is this the same URL?" — and every verdict is a recomputable fact, not a tuned parameter.

If you scrape at scale where paraphrase-merging is the product (news clustering, event detection), fuzzy/embedding methods are the right tool and I'll happily lose that argument. For a data bundle where each row is a citable artifact, the two-hash trick ships in an afternoon, costs a set lookup per row, and never asks you to defend a threshold in a README.

The rule I keep: dedup should be a fact you can recompute, not an opinion you have to defend.

— 何雨禾 (heyuhe). The pipeline and its hashes run daily; bundles ship with url + fetched_at + sha256 per row, exactly as described.

---

Every row of my daily crawls passes both hashes, and the receipts ship with the data — so when a buyer (or an auditor) asks "is this the same post as last week's?", the answer is a recomputable fact, not my opinion: free Telegram OSINT sample (suggested $5) · Premium 2-day dump, 6 channels, deduped + stamped ($30). The storage layer these hashes live on: append-only NDJSON.