Google's News API has been gone since 2016, and people still search for it

#python #api #webdev #datascience

Search for "google news api" and a striking share of the results are people
looking for something that no longer exists. Google deprecated the News Search
API in May 2011 as part of a wider AJAX Search API cull, kept it limping until
the final shutdown in February 2016, and never shipped a replacement. A decade
later the searches are still there.

What almost nobody mentions is that Google News still exposes a perfectly good
public endpoint. It is just an RSS feed rather than a JSON API, it needs no key,
and it has quietly outlived the API that was supposed to be the real product.

The endpoint

https://news.google.com/rss/search?q=<query>&hl=en-US&gl=US&ceid=US:en

That is the whole thing. Swap hl, gl and ceid for a different language and
country and you get that edition instead. There is no key, no quota page, and no
sign-up.

The catches are real but small:

It is RSS, so you get XML. Title, link, publication date, source name and a short description. No article body, and no way to ask for one.
The links are Google redirects, not publisher URLs. You have to follow them if you want the destination.
The publisher name lives in a nested <source> element, not where a normal feed parser looks for it.
Roughly 100 items per query. Deeper history is not available through it.

None of that is a reason to go and pay for a news API. It is a reason to write
about forty lines once.

Two sources are better than one

Bing News publishes its own RSS in the same shape at a different URL. Running
both and merging costs one extra request and meaningfully changes what you see,
because the two indexes disagree about what is worth surfacing.

Here is a real run over both:

16 articles
sources: {'google:semiconductor export controls': 8,
          'bing:semiconductor export controls': 8}

2026-07-09  Just Security          It Takes More Than Two to Tango: Creating...
2026-07-20  Reuters                China considers tighter export controls on...
2026-03-24  CSIS                   China's Localization Drive in Semiconductors...
2026-07-28  EU Today               China's Domestic DUV Push Puts ASML's Export...
2026-06-24  Purdue University      Intellectual Property Protection and Export...
2025-11-20  Cfr.org                Protecting the Foundation: Strengthening...
2026-07-19  The Motley Fool        Semiconductor Trade Statistics: U.S. Imports...

Every row carries title, url, publisher, publishedAt, summary,
source, image and guid.

One thing worth knowing if you parse Bing's feed yourself. It carries the
publisher in <News:Source>, and declares that prefix as

xmlns:News="https://www.bing.com/news/search?q=<your query>&format=RSS"

The namespace URI is the request URL, so it changes on every query and no fixed
namespace constant will ever match it. Match on the local element name instead.
I found this the way you would expect: publisher was null on exactly the Bing
half of a two-source run while the value was sitting in the feed the whole time.

The code

import json
import os
import urllib.request

ACTOR = "glitchbound~news-scraper"
API = "https://api.apify.com/v2"


def headlines(query, per_source=10):
    payload = {
        "searchQueries": [query],
        "maxArticlesPerSource": per_source,
        "language": "en",
        "includeBingNews": True,
        "dedupeHeadlines": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A feed that fails comes back as a row with an `error` field rather than
    # failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for a in headlines("semiconductor export controls", per_source=5):
    # Fields can be absent on any given row, so read them defensively rather
    # than subscripting. Feeds are not schemas.
    when = (a.get("publishedAt") or "")[:10] or "no date"
    who = (a.get("publisher") or "unknown")[:28]
    print(f'{when:10}  {who:28} {(a.get("title") or "")[:60]}')

That runs News Scraper on
Apify, which is the same forty lines plus the parts that are
tedious rather than hard: both feeds, the nested publisher element, the date
normalisation, and deduplication.

If you would rather own the code, the endpoint above is public and this post has
told you everything you need.

Deduplication is most of the value

One wire story appears on dozens of syndicating domains, and Google and Bing will
both hand you several copies. A hundred raw results is frequently thirty stories.

Dedupe on the URL first, then on a normalised headline, because the same Reuters
piece appears under slightly different titles across outlets. Doing it on
headline alone merges genuinely different stories that share a phrasing; doing it
on URL alone leaves most of the duplication in place.

Where this does and does not fit

Fits: brand and competitor monitoring, tracking a topic across outlets,
building a corpus of headlines, seeing which publishers cover a beat, feeding a
summarisation pipeline.

Does not fit: anything needing article text, anything needing more than a few
months of history, or academic archival work. For those you want a licensed
provider, and the honest answer is that no free endpoint replaces one.

What it costs

If you write it yourself, nothing beyond your own hosting.

Through the Actor it is $1.50 per 1,000 articles on the free tier, so a daily run
pulling 50 articles is about seven cents a month, plus a $0.02 Actor-start event
per run. Duplicates that get dropped are not charged, and neither are feeds that
fail.

The wider point

An enormous amount of "you need an API for this" is habit. Google News, Apple's
app charts, the SEC, the World Bank, the ECB and Open Food Facts all publish
usable public endpoints that need no key and get very little attention, largely
because there is no vendor with an incentive to tell you about them.

Worth checking before you pay for access to public data.