Google News RSS has no cursor, so the query IS the cursor

Quick answer

Google News RSS has no cursor and no page parameter. GET https://news.google.com/rss/search?q=openai&hl=en-US&gl=US&ceid=US:en returns 200, application/xml, 123,974 bytes — byte-for-byte identical whether the request goes out as curl-cffi chrome131 or firefox133 — and hard-caps at roughly 100 <item> nodes no matter how long the underlying story actually runs. There's no page 2 to ask for. The only way past the cap is to make the query itself walk a date range, and even then the feed has a second trap: a window that comes back with exactly 100 items isn't a complete page, it's a truncated one, and there's no error to tell you so.

Why does the feed cap at 100 items? 📰

We measured it directly: openai over any sufficiently wide window returns 98 <item> nodes and stops, feed metadata be damned. There's no page, no offset, no next link anywhere in the XML. The mechanism Google actually exposes for "give me more" is the search operator after:YYYY-MM-DD before:YYYY-MM-DD appended to q — so pagination on this target isn't a request parameter, it's a second query with a narrower window. We confirmed that two adjacent 5-day windows on openai each returned 100 items with zero overlapping links between them — the windowing genuinely partitions the result set, it doesn't just resample the same top-100.

The rule that makes this honest: a window returning exactly 100 items is truncated, not finished. A scraper that treats a full page as a done page will silently under-report every time a 5-day span happens to have more than 100 stories in it, and Google gives you no signal that it happened — the response is a clean 200 either way. Our date-window walker treats "hit the cap" as "halve the window and re-fetch," recursively, until each sub-window returns comfortably under 100.

Why is <item><link> not the article URL? 🔗

The second trap is the one that costs money if you don't catch it. Every <item><link> in the feed is not the publisher's URL — it's an encoded Google redirect token: https://news.google.com/rss/articles/CBMi…. Resolving that token to the real article URL means an HTTP request per row, which quietly turns a 100-item page into 101 requests and doubles your run's request budget for a field most use cases don't need.

We made resolution opt-in (resolveArticleUrl, default false) rather than automatic, because the feed already gives you the publisher domain for free: the <source url="https://techcrunch.com"> attribute sits right there on every item, no extra request required. If you actually need the resolved deep link — say, for a downstream dedupe against your own CMS — flip the flag and we'll follow the redirect per row on your behalf; otherwise you get the domain at zero marginal cost.

What about monitoring a beat instead of a query? 🎯

Topic feeds are the second axis, and they need no search query at all — useful for "monitor this beat continuously" rather than "search this term once." https://news.google.com/rss/headlines/section/topic/TECHNOLOGY?hl=en-US&gl=US&ceid=US:en returned 47 items in our measurement; BUSINESS returned 70. Same feed shape, same date-window rules apply once you cross the cap, just no query string to walk.

What we handle for you

• 🛡️ We rotate browser fingerprints — curl-cffi impersonating Chrome and Firefox TLS handshakes, verified byte-identical output across both, so the target sees a browser either way.
• 🔁 We retry with exponential backoff on 408 / 429 / 5xx, honouring Retry-After, up to 5 attempts per request.
• 🧱 We walk the date-window recursively whenever a window hits the ~100-item cap, so you get the true count for wide date ranges instead of a silently truncated 100.
• 🧊 We keep the dataset clean — Pydantic-validated rows, ISO-8601 timestamps, guid as a stable dedupe key.
• 💰 You pay only for results that land. No data → no charge beyond the small actor-start fee.

How much does it cost?

$0.20 to start a run, plus $1.00 per 1,000 results — call it $1.20 per 1,000 articles landed, no subscription.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("DevilScrapes/google-news-scraper").call(
    run_input={
        "queries": ["openai", "anthropic"],
        "dateFrom": "2026-09-06",
        "dateTo": "2026-09-13",
        "resolveArticleUrl": False,
        "maxItemsPerQuery": 200,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["pub_date"], "|", item["source_name"], "|", item["title"])

One honest scope note: we also ship a generic rss-feed-scraper for any RSS/Atom feed as-is. This Actor exists because a generic parser doesn't attempt the date-window truncation walk or the redirect-token unwrapping — those are Google News-specific, and getting them wrong either under-reports silently or bills you for requests you didn't need.

→ Google News Scraper on Apify

---

Built by Devil Scrapes. We walk the date windows Google's own cap makes necessary, unwrap the redirect token only when you ask for it, and never let a truncated 100 masquerade as a complete page.