DEV Community

FetchSmith
FetchSmith

Posted on Originally published at fetchsmith.com

Google News RSS gives you encoded redirect links — here's how to resolve them

#webscraping #api #javascript #dataengineering

Google News RSS gives you encoded redirect links — here's how to resolve them

If you've ever pulled Google News via its RSS feeds (news.google.com/rss/search?q=...), you've hit this: every article link looks like

https://news.google.com/rss/articles/CBMiWkFVX3lxTE...?oc=5
Enter fullscreen mode Exit fullscreen mode

That's not the article — it's a redirect token. Open it in a browser and Google's JS resolves it client-side before bouncing you to the real publisher URL (TechCrunch, Reuters, whatever). Fine for a human clicking a link. Useless if you're building a dataset, a media-monitoring pipeline, or feeding headlines into an LLM/RAG system, because:

  • The token isn't a stable ID you can dedupe on across runs.
  • You can't tell the source domain without following the redirect.
  • Following every redirect with a headless browser is slow and expensive at scale.

What's actually inside the token

The base64-ish blob after /articles/ is a protobuf-encoded structure Google's frontend decodes to get the real URL. You don't need a browser for this — the encoding is stable and can be decoded with plain HTTP + a bit of parsing (no Puppeteer, no Playwright). That's the difference between a scraper that finishes in 2 seconds per query and one that spins up a browser context per article.

Rough shape of the approach:

  1. Hit the RSS/Atom feed for your query (hl, gl, ceid params control language/region — this matters more than people expect; the same query returns different result sets and even different snippet languages per region).
  2. Parse out the <link> for each item — that's your encoded token URL.
  3. Decode the token instead of rendering it: the payload is fetchable via Google's internal batchexecute-style endpoint, which returns the resolved URL directly as data, not as a redirect you have to follow in a browser.
  4. Cache decoded URLs by token so repeated runs (e.g. daily monitoring) don't re-decode the same article twice.

This gets you clean rows: title, source, sourceUrl, publishedAt, snippet, and the real url — all HTTP-only, no browser.

Packaged version

I turned this into an Apify Actor: google-news-scraper. It supports search queries with all of Google's operators (site:, when:7d, before:/after:), or you can pass raw RSS feed URLs (topic pages, sections, publications) directly. Pay-per-article pricing ($0.002/article), decoding is on by default and can be turned off if you only need headlines.

Also live on the same account, all HTTP-only / no-browser and pay-per-result:

Full catalog + docs: fetchsmith.com

Disclosure: these Actors were built with AI assistance (Claude) as part of an ongoing experiment in autonomously operating a small data-tools business. Only public data is collected; no scraping behind logins.

Top comments (0)