Every corpus pipeline has a dedup stage. It sits after collection, near storage, and it quietly deletes the 30–60% of your fetched documents that are copies of something you already have. Everyone agrees it is there, nobody audits what it cost. Because in a proxy-mediated crawler, every one of those discarded copies was bought by the byte: a full request through a paid exit, a full body across the wire, a full TLS handshake, spent to rediscover that page 47 is the same article you fetched through page 4. Dedup placed after collection is not a data-hygiene detail. It is the line on your invoice marked "wasted," and the reason it exists is that the pipeline treats exits as sources of diversity without ever measuring whether different exits actually produce different content.
The thesis: deduplication in a proxied corpus has two layers, and most teams build only the cheap one. You need content-level near-dup detection on the ingest path, yes — but you also need unique-yield accounting per exit, because your proxy pool's job is to see the web from distinct vantage points, and half the exits in an average pool see exactly what the other half sees. Measure which exits actually produce new documents, stop paying the ones that do not, and dedup stops being a storage cleanup and becomes a pricing decision.
The duplicate taxonomy a proxy pool creates
Before code, the failure shapes, because each has a different fix.
Exact re-fetches. The same URL, queued twice, or reached through two links. URL canonicalization catches this; it is free and mandatory.
Template twins. The same article under /2024/03/story.html and /amp/story.html and ?print=true. Different URLs, byte-similar bodies. Needs content hashing, not URL logic.
The exit illusion. The one this article is about: the same content variant served through fifty different residential exits because all fifty terminate at the same CDN edge cache, or because geo-targeting is coarser than your exit granularity — twenty US residential IPs from three states all get the identical "national" edition. Your pool looks diverse; your corpus is not. A dedup index keyed by content collapses them to one row, but you paid for fifty.
Spurious non-duplicates. The mirror failure: exits that differ only in volatile chrome — a fresh CSRF token, a server-rendered timestamp, a personalized greeting — hash as new documents and pollute your "unique" count. Your dedup says 50 uniques; a human sees one page wearing 50 hats. This is why hash stability under volatile fields is half the battle.
The last two failure modes point at the same instrumentation: for any origin, the quantity you actually care about is distinct documents per exit purchased, and you can only compute that if your pre-hash normalization is good.
Normalize the volatility out before you hash
A content identity function for web pages is a small pipeline: parse, drop the elements that are per-request rather than per-page, collapse the rest to a bag of normalized text shingles. The drop list is empirical — hash the same page fetched through two exits, diff what changed, and add the noisy selectors until near-duplicates stop looking novel.
import re
from html.parser import HTMLParser
_NOISE_TAGS = {"script", "style", "noscript", "svg", "iframe"}
_VOLATILE_MARKERS = re.compile(
r"(csrf|nonce|authenticity|timestamp|session[-_]?id|greeting|welcome,)", re.I
)
class TextHarvester(HTMLParser):
def __init__(self):
super().__init__()
self._skip = 0
self.chunks = []
def handle_starttag(self, tag, attrs):
if tag in _NOISE_TAGS or tag == "time":
self._skip += 1
elif any(_VOLATILE_MARKERS.search(k) or _VOLATILE_MARKERS.search(str(v))
for k, v in attrs):
self._skip += 1
def handle_endtag(self, tag):
if self._skip:
self._skip -= 1
def handle_data(self, data):
if not self._skip and data.strip():
self.chunks.append(data.strip())
def page_text(html: str) -> str:
p = TextHarvester()
p.feed(html)
return " ".join(p.chunks)
def shingles(text: str, k: int = 8) -> set[int]:
words = re.findall(r"\w+", text.lower())
return {hash(" ".join(words[i:i + k])) for i in range(max(0, len(words) - k + 1))}
def jaccard(a: set, b: set) -> float:
return len(a & b) / len(a | b) if a | b else 1.0
(The production version swaps the raw hash() set for MinHash signatures — same interface, fixed memory. The concept that matters is the normalization stage; two pages that differ only in a session-scoped data-* attribute must collide.)
Unique yield: the metric your pool deserves
Now the accounting that turns dedup into a routing decision. For a sampled basket of URLs, fetch each one through several exits, run the pair through the similarity function, and ask the only question that matters: did this exit's copy contain information I did not already have from this origin? Score each exit as marginal unique documents per paid byte and use it to re-weight selection.
import random
from dataclasses import dataclass, field
@dataclass
class ExitLedger:
ip: str
fetched: int = 0
novel: int = 0 # docs not ≥-similar to anything seen for origin
bytes_paid: int = 0
@property
def unique_yield(self) -> float:
return self.novel / self.fetched if self.fetched else 0.0
class OriginView:
"""Keeps one representative shingle-set per ~unique doc seen for an origin."""
def __init__(self, sim_threshold: float = 0.85):
self.seen: list[set] = []
self.t = sim_threshold
def is_novel(self, sh: set) -> bool:
if sh and any(jaccard(sh, old) >= self.t for old in self.seen):
return False
if sh:
self.seen.append(sh)
return True
def probe_exits(session_factory, urls, exits, origin_view: OriginView,
ledgers: dict[str, ExitLedger]):
"""Sequential version for clarity; async it exactly like your real crawler."""
import asyncio
async def run():
async with session_factory() as session:
for url in urls:
got_novel_here = False
for ip in random.sample(exits, len(exits)):
async with session.get(url, proxy=ip) as r:
body = await r.text()
led = ledgers[ip]
led.fetched += 1
led.bytes_paid += len(body)
if not got_novel_here and origin_view.is_novel(
shingles(page_text(body))
):
led.novel += 1
got_novel_here = True # one copy per URL per round is enough
asyncio.run(run())
Run the probe on a small stratified sample — say 30 URLs × 20 exits, off-peak — and read the ledger. A typical real result looks like: five exits carrying 80% of the novel documents, eight exits contributing only template twins and volatile-chrome variants, seven exits contributing nothing the first five did not already return. Those last fifteen are not broken; your origin simply does not vary along the axis you paid to sample. The pool is diverse; the response is not, and you were billing against the pool.
Fold the ledger back into routing exactly like article one folded geography in: weight exit selection by recent unique yield for the target origin, not by round-robin faith. And let the weights decay — an exit that stops producing novelty for origin A may be the only one that breaks a cache pinning on origin B. Unique yield is per (exit, origin) pair, which is the same table shape the health scorecard folks use, populated by content rather than status codes.
Check the index before you spend the byte
The cheap half of dedup, and the one that pays back first: consult the near-duplicate index before dispatching the request. URL canonicalization is its first pass — strip utm_*, sort parameters, drop fragments, resolve tracking redirects at HEAD level — and catches the exact re-fetch class at zero wire cost. Content identity is the second pass and cannot run pre-fetch, which is why the pre-hash normalization above exists: the faster and stabler your identity function, the earlier in the funnel you can refuse work. In an image-heavy corpus the win is enormous, because a refused duplicate is 40 KB of metadata on text crawls and 8 MB of never-spent residential traffic on assets (see yesterday's bandwidth arithmetic).
The honest version
Dedup is where corpus teams quietly certify that a third of their collection was redundant — and in a proxied pipeline, redundancy has a per-GB price tag attached. So instrument the exit layer, not just the storage layer: normalize volatility out of your content identity, measure unique documents per exit per origin, and let that ledger — not your provider's IP count — define how diverse your pool actually is. An exit that never changes what the origin shows you is not a vantage point; it is a toll booth. And the day you prune against unique yield, your dedup index shrinks its own input, your invoice follows it down, and "we collected 2M documents" quietly becomes honest again, because now you can say how many of them were actually seen more than once.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)