News Monitoring in Multiple Languages: Geographic Variants and Cross-Language Story Clustering

A few years ago I built a media-monitoring pipeline for a client with operations in six countries, and the spec sounded simple: "tell us what the news says about us in each market." The first version fell over in week one — not because scraping is hard, but because news on the internet is not one corpus. It's a set of parallel corpora: each language, each country's edition of global outlets, each regional aggregator, with different coverage of the same events and different silences. A story that's front-page news in Frankfurt's German press may appear in English-language German media a day later, softer, shorter, and six paragraphs of context lighter.

This post is about building multilingual news monitoring that accounts for all of that: collection across languages and geographic editions, entity and story matching across languages, and the pragmatic use of translation in the middle.

Step 1: Collect From the Right Shape of Sources

News is the friendliest scraping domain there is, because most of the industry still runs on infrastructure designed for machines: RSS and Atom feeds, sitemaps, and increasingly JSON APIs. Exploit that before you ever render a page.

• RSS/Atom for outlets that publish feeds (add ?lang=de or locale path variants where offered).
• Sitemaps (news.xml where available) for high-frequency change discovery — Google News sitemaps list fresh articles with publication timestamps, and many publishers maintain them even when their RSS is stale.
• Aggregator search pages (Google News, Bing News) for discovery of outlets you don't track directly. This is the layer where anti-bot lives — and where geography matters most, covered below.
• Direct article pages only as a last resort, for paywalled or JS-rendered outlets.

import feedparser, httpx

def fetch_feed(url: str, proxy: str | None = None) -> list[dict]:
    client_kwargs = {"timeout": 30}
    if proxy:
        client_kwargs["proxies"] = proxy
    with httpx.Client(**client_kwargs) as client:
        r = client.get(url)
        r.raise_for_status()
    feed = feedparser.parse(r.text)
    return [{
        "title": e.get("title", ""),
        "url": e.get("link", ""),
        "published": e.get("published", ""),
        "summary": e.get("summary", ""),
    } for e in feed.entries]

The Geography Dimension: Same Site, Different News

Here's the part that surprises people. Google News — and many global outlets — serve geographic variants: request the German Google News homepage from a US IP and you get a different result than requesting it from a German IP, in mix if not in language. Regional outlets may soft-block foreign traffic. Some editions are only reachable from in-country. If your monitoring is meant to answer "what does the news landscape look like in this market," you must collect it through exit IPs in that market.

This is a place where I've found geo-targetable residential proxies genuinely necessary rather than just convenient: route the German collection worker through German IPs, the Japanese worker through Japanese IPs, and so on. With Thordata I do this with a -geo-de, -geo-jp style suffix on the proxy username; one credential, per-market exit. Sessions should be sticky per outlet — a consistent reader looks like a reader.

def make_client(market: str) -> httpx.Client:
    geo = {"de": "de", "jp": "jp", "br": "br", "us": "us"}[market]
    return httpx.Client(
        proxies=f"http://thor-user-sessid-{market}-01-geo-{geo}:@proxy.thordata.com:24125",
        headers={"User-Agent": UA, "Accept-Language": LANG[market]},
        timeout=30,
    )

One more geographic subtlety: timezones. An article "published at 06:00" is ambiguous until you know the outlet's timezone, and for freshness ranking you need everything in UTC. Store both the raw published string and a normalized UTC timestamp.

Step 2: Language Handling — Detect, Don't Trust

You cannot rely on the feed's declared language. Feeds lie, multilingual outlets mix languages within a channel, and aggregators mix everything. Run language detection on the title plus first paragraph of every item:

import fasttext
_lang = fasttext.load_model("lid.176.ftz")

def detect_lang(text: str) -> str:
    text = " ".join(text.split())[:1000]
    if not text:
        return "und"
    return _lang.predict(text)[0][0].replace("__label__", "")

For the translation layer, my honest advice after living with several architectures: translate late and cheaply, translate once, and cache. You don't need full-text translation of every article for most monitoring use cases. The pipeline that works:

1. Detect language; route into per-language processing.
2. Translate titles and summaries only (they're short and cheap) for cross-language story clustering.
3. Translate full text on demand — when a cluster is flagged relevant and a human (or an LLM analyst) needs to actually read it.
4. Cache translations keyed by content hash; news articles are immutable, so you'll never translate the same article twice.

Machine translation quality now is good enough that entity-level monitoring — "did our company get mentioned?" — works fine on translated text. Tone and nuance analysis still benefits from doing it in the original language with a multilingual model, then mapping scores to a common scale.

Step 3: Cross-Language Story Clustering

The interesting intellectual problem in multilingual news monitoring is: is this German article about the same event as that Japanese article? They share few string tokens, quote different sources, and emphasize different angles. My approach is a two-stage clustering:

Stage 1 — entity spine. Extract named entities from each item (per-language NER, or a multilingual transformer NER model — spaCy's multi-language models or XLM-R based NER both work). Normalize entities across languages via Wikipedia/Wikidata ID: "Deutsche Bahn" and "ドイツ鉄道" are both Q131452. Entities are the language-independent spine of a story.

# Simplified: entity vector + time window clustering
from dataclasses import dataclass

@dataclass
class NewsItem:
    url: str
    lang: str
    title: str
    entities: set[str]      # Wikidata IDs
    published_utc: str
    embed: list[float]      # multilingual sentence embedding of translated title

def same_story(a: NewsItem, b: NewsItem) -> bool:
    time_close = abs(t(a) - t(b)) < 36 * 3600
    entity_overlap = len(a.entities & b.entities) >= 2
    cosine = dot(a.embed, b.embed) / (norm(a.embed) * norm(b.embed))
    return time_close and entity_overlap and cosine > 0.75

Stage 2 — multilingual embeddings. Embed the (translated) titles with a multilingual sentence-transformer — paraphrase-multilingual-MiniLM or similar maps different languages into one vector space well enough that same-story articles from different countries cluster together. Two shared entities, a 36-hour window, and cosine above ~0.75 is my same-story threshold; below that, they're related coverage, which is also worth recording as an edge.

Once stories are clusters rather than items, the analyses that clients actually want become straightforward: coverage volume per market per story (did the German press cover this more than the US press?), angle divergence (embed German-language coverage of a cluster separately from English-language coverage and look at the vector distance — a big distance means the markets are telling different stories), and lead-lag (which market's outlets picked the story up first).

Step 4: Freshness and Dedup

News volume is spiky: quiet mornings, then a story breaks and a hundred outlets publish within two hours. Design for the spike. Poll high-priority feeds every few minutes and long-tail outlets hourly; on aggregator pages, respect rate limits and keep per-session request counts low. Dedup is aggressive in news: the same wire story (Reuters/AP copy) appears verbatim across dozens of outlets. Near-duplicate detection on the first paragraph with shingle hashing catches most of it — and unlike review dedup, here you usually do drop the copy and keep one representative with an outlet count.

A Word on Rights

News text is copyrighted, full stop. Monitoring, clustering, counting, and summarizing for internal analysis is well-trodden ground. Republishing article text — including in automated client-facing reports — is a different activity with a different legal weight. Keep your outputs as counts, clusters, links, and short extracts, and check the licensing posture of any outlet whose text you move around.

Wrapping Up

Multilingual news monitoring done right is less a scraping problem than an alignment problem: aligning geographic editions to markets (via geo-matched collection IPs), aligning languages to a common entity space (via Wikidata-grounded NER), and aligning stories across languages (via multilingual embeddings on translated titles). Get the spine right and a six-market dashboard turns out to be one pipeline with per-market glasses, not six pipelines.

Disclosure: I use Thordata's residential proxies for the geo-matched news collection described in this post. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*