News Scraping at Scale: Deduplicating Syndicated Stories and Extracting Clean Articles

News scraping is the automated collection of articles from news websites into structured records (headline, body text, publication, section, publication date and URL) for uses such as media monitoring, market and risk intelligence, research and training datasets. Fetching pages is the easy part. The hard part is that one story arrives many times: the same wire report republished by dozens of outlets, the same article at three different URLs, and every copy wrapped in navigation, related links and captions.

This post builds the pipeline that turns raw news pages into a clean, deduplicated dataset: URL canonicalisation, article extraction, reliable dates, and near-duplicate clustering with MinHash. Every snippet below was run against sample pages.

Where the duplicates come from

• Syndication. Agency and wire stories are republished by many outlets, often lightly edited, with a new headline or an extra paragraph.
• URL variants. The same article is reachable through mobile and print versions, tracking parameters and trailing-slash differences.
• Updates. Developing stories are revised during the day, sometimes at a new URL.
• Aggregation. Section pages, topic pages and feeds all link to the same articles, so a crawler discovers each one many times.

Counting raw pages overstates coverage dramatically. A dataset built for "how much coverage did this topic get?" has to count stories, not URLs.

Step 1: canonicalise URLs

Before fetching or storing anything, reduce every URL to one canonical form. Prefer the page's own rel="canonical" link when it has one; otherwise strip tracking parameters and known path variants:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

TRACKING = ("utm_", "fbclid", "gclid", "mc_", "ref", "cmpid")

def canonical_url(url: str, rel_canonical: str | None = None) -> str:
    """Prefer the page's own canonical link; otherwise strip tracking noise."""
    url = rel_canonical or url
    parts = urlsplit(url)
    query = [(k, v) for k, v in parse_qsl(parts.query)
             if not k.lower().startswith(TRACKING)]
    path = parts.path.rstrip("/") or "/"
    for suffix in ("/amp", "/print"):
        if path.endswith(suffix):
            path = path[: -len(suffix)] or "/"
    return urlunsplit((parts.scheme.lower(), parts.netloc.lower().removeprefix("www."),
                       path, urlencode(sorted(query)), ""))

https://www.Example-News.com/business/rates-hold/amp/?utm_source=x&id=5&fbclid=abc becomes https://example-news.com/business/rates-hold?id=5. This alone removes a large share of exact duplicates before they reach your storage.

Step 2: extract the article, not the page

A news page is mostly not the article. Read structured data first: many publishers include schema.org NewsArticle markup with headline, datePublished, dateModified, publisher and articleSection. Then extract the body from the

element, removing navigation, asides, figures and other boilerplate, and dropping very short paragraphs such as captions and one-line promos:

import json
from datetime import datetime, timezone
from bs4 import BeautifulSoup

BOILERPLATE = ["nav", "header", "footer", "aside", "form", "script", "style", "figure"]
ARTICLE_TYPES = {"NewsArticle", "Article", "ReportageNewsArticle", "AnalysisNewsArticle"}

def to_utc(value: str | None) -> datetime | None:
    if not value:
        return None
    try:
        dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
    except ValueError:
        return None
    if dt.tzinfo is None:
        return None  # a date without a timezone is ambiguous; flag it rather than guess
    dt = dt.astimezone(timezone.utc)
    return None if dt > datetime.now(timezone.utc) else dt

def extract_article(html: str, url: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")
    meta = {}
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        for node in data if isinstance(data, list) else [data]:
            if isinstance(node, dict) and node.get("@type") in ARTICLE_TYPES:
                meta = node
                break

    published = meta.get("datePublished") or (
        soup.find("meta", property="article:published_time") or {}).get("content")
    publisher = meta.get("publisher")
    publisher = publisher.get("name") if isinstance(publisher, dict) else publisher

    body_root = soup.find("article") or soup.body
    for tag in body_root.find_all(BOILERPLATE):
        tag.decompose()
    paragraphs = [p.get_text(" ", strip=True) for p in body_root.find_all("p")]
    body = "\n".join(p for p in paragraphs if len(p.split()) >= 8)  # drop captions and one-liners

    return {
        "url": url,
        "headline": meta.get("headline") or (soup.title.string.strip() if soup.title else None),
        "publication": publisher,
        "section": meta.get("articleSection"),
        "published_at": to_utc(published),
        "modified_at": to_utc(meta.get("dateModified")),
        "language": soup.html.get("lang") if soup.html else None,
        "body": body,
    }

On a sample page with navigation, a photo caption, a "related" box and a footer, the extractor returned the headline, publication, section, language and a 51-word body containing only the article's own paragraphs.

Step 3: get the dates right

Dates are where news datasets quietly go wrong, and timing is often the whole point of the analysis. Three rules:

Prefer datePublished from structured data, then the article:published_time meta tag, and only then anything parsed from visible text.
Normalise to UTC. A story published at 09:30 in a +01:00 timezone is 08:30 UTC, and comparing local times across publishers produces nonsense orderings.
Reject what you cannot trust. to_utc returns None for dates without a timezone and for dates in the future, so they can be flagged rather than silently mis-sorted.

Keep dateModified separately. A story updated this afternoon was still first published this morning.

Step 4: cluster syndicated copies with MinHash

Canonical URLs catch exact duplicates. Syndicated copies need near-duplicate detection, because outlets add a line, change the headline or append a credit. Comparing every article with every other is quadratic and impractical at scale. MinHash with locality-sensitive hashing (LSH) solves it: each article becomes a short signature, and only articles whose signatures collide in at least one band are compared.

import hashlib
import re
from collections import defaultdict

NUM_PERM, BANDS = 64, 16          # 16 bands x 4 rows
ROWS = NUM_PERM // BANDS

def shingles(text: str, k: int = 5) -> set[str]:
    words = re.findall(r"[a-z0-9]+", text.lower())
    return {" ".join(words[i:i + k]) for i in range(max(len(words) - k + 1, 1))}

def signature(text: str) -> list[int]:
    """MinHash signature: for each seeded hash, the minimum over all shingles."""
    sh = shingles(text)
    return [min(int.from_bytes(hashlib.blake2b(f"{seed}:{s}".encode(), digest_size=8).digest(), "big")
                for s in sh)
            for seed in range(NUM_PERM)]

def estimated_similarity(a: list[int], b: list[int]) -> float:
    return sum(x == y for x, y in zip(a, b)) / len(a)

def candidate_pairs(signatures: dict[str, list[int]]) -> set[tuple[str, str]]:
    """LSH banding: articles sharing any identical band become candidate pairs."""
    pairs = set()
    for band in range(BANDS):
        buckets = defaultdict(list)
        for doc_id, sig in signatures.items():
            buckets[tuple(sig[band * ROWS:(band + 1) * ROWS])].append(doc_id)
        for ids in buckets.values():
            pairs.update((a, b) for i, a in enumerate(ids) for b in ids[i + 1:])
    return pairs

In testing, a syndicated copy with "UPDATED:" prepended and a credit line appended had an estimated similarity of 0.86 to the original and was returned as a candidate pair. An unrelated story scored 0.0 and was never compared. With 16 bands of 4 rows, pairs above roughly 0.5 similarity are likely to become candidates; confirm each candidate with its estimated similarity (for example, at least 0.8) before merging. Tune the band and row split to move that threshold.

Step 5: keep stories, and keep the syndication

Group confirmed near-duplicates into story clusters. For each cluster, choose a canonical article (usually the earliest published, or the original source when it can be identified) and keep the full list of outlets and URLs that carried it.

Do not throw the copies away. Which publications carried a story, and how quickly, is one of the most useful signals in media monitoring: a story picked up by forty outlets in an hour is a different event from one carried by two.

Practical notes

1. Store the raw HTML for a sample of pages, so you can re-run extraction when publishers change their templates.
2. Monitor extraction quality per publisher. A sudden drop in average body length or in the share of articles with a valid date usually means a layout change. Treat these as quality checks, alongside the other measures in any framework for assessing data quality.
3. Discover through feeds and sitemaps where publishers provide them. They are lighter for publishers to serve and for you to process than repeated crawls of section pages.
4. Collect responsibly. Respect each site's robots.txt and terms, keep request rates low, and do not attempt to get around paywalls. Store text for analysis, and do not republish full articles.

Wrapping up

News scraping at scale is a deduplication problem as much as a collection problem. Canonicalise URLs before storing anything, extract the article rather than the page, normalise dates to UTC and reject the ones you cannot trust, then cluster syndicated copies with MinHash so that you count stories rather than pages. Keep the syndication list for every story, and you have both a clean dataset and a measure of how far each story travelled.

FAQ

How do you detect duplicate news articles?

First canonicalise URLs to remove tracking parameters and mobile or print variants. Then detect near-duplicates, such as syndicated copies with small edits, using shingling and MinHash with locality-sensitive hashing, and confirm candidate pairs with an estimated similarity threshold before grouping them into story clusters.

What fields should a news scraping dataset include?

Headline, body text, publication, section, publication date and modified date normalised to UTC, language, canonical URL, collection time and, for deduplicated datasets, a story cluster ID with the list of outlets that carried the story.

How do you extract clean article text from a news page?

Read structured data such as schema.org NewsArticle markup for the headline, dates and publisher, then extract body text from the article element after removing navigation, asides, figures, footers and very short paragraphs such as captions.