DEV Community: Richard Nemeth

How to get Google News results as structured JSON

Richard Nemeth — Sun, 27 Sep 2026 16:20:07 +0000

The Google News API takes a query and returns the articles Google News shows for it, in order, as data: headline, publisher, link, snippet, date and thumbnail, for a chosen country and language. Below: the request and response, date normalization, storage and deduplication, and three patterns (brand alerts, coverage share, adverse media screening).

If you are deciding between RSS feeds and structured results, read Google News RSS feeds: what they give you and where they stop first.

Definitions

News result: one article on the Google News results page for a query, returned as an item of result.newsResults[].
Position: the 1-indexed order of the article in the news results, returned with the page it appeared on.
Source: the publisher name as Google displays it, such as a newspaper or a trade site.
Coverage share: the fraction of news results for a query set that belong to a given publisher or mention a given brand.

The request

curl -X POST https://api.answerline.dev/v1/monitor/google/news \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "acme corp recall",
    "country": "DE",
    "hl": "de",
    "pages": 2
  }'

Request fields:

Field	Type	Notes
`query`	string, 1-10,000 chars	Required. The News search.
`country`	string	ISO 3166-1 alpha-2. Send `country` or `gl`.
`gl`	string	Google's name for the result geography; same codes, either case. Different values in `country` and `gl` are a 400.
`hl`	string	Interface language (`en`, `de`, `pt-BR`). Overrides the language derived from the country.
`pages`	integer 1-10	Default 1.
`include.html`	boolean	Not available on Google News: a request that sets it is refused with a 400.

News targeting is by country and language. Supported countries are listed by GET /v1/countries?model=google. Don't route News through the Google Search endpoint's url shape: a Google URL carrying tbm=nws is rejected there, because News has its own endpoint and pricing.

The response

{
  "success": true,
  "result": {
    "newsResults": [
      {
        "position": 1,
        "title": "Acme Corp expands recall to second product line",
        "link": "https://news.example.de/acme-recall",
        "snippet": "The company said on Tuesday that...",
        "source": "Example Nachrichten",
        "date": "3 hours ago",
        "page": 1,
        "thumbnail": "https://news.example.de/images/acme.jpg"
      }
    ]
  }
}

Field by field:

position: 1-indexed rank in the news results. With pages above 1, page tells you which page the article came from; store both.
title, snippet: the headline and the text excerpt shown.
link: the article URL. If Google served it as a redirect, redirectLink carries Google's redirect next to it.
source: the publisher's display name. It is a name, not a domain; derive the domain from link when you need a stable key.
date: the date text as displayed, for example "2 hours ago".
thumbnail: the image URL when one is shown.
html[]: not returned for Google News; include.html is refused on this endpoint.

Normalizing dates

date is display text, such as "2 hours ago", so do not assume one format. Store three things: the raw string, the capture time, and a parsed estimate.

import re
from datetime import datetime, timedelta, timezone

UNITS = {"minute": "minutes", "hour": "hours", "day": "days", "week": "weeks"}

def published_estimate(raw: str, captured_at: datetime) -> datetime | None:
    """Parse English relative dates like '3 hours ago'. Returns None for formats it doesn't know."""
    m = re.match(r"(\d+)\s+(minute|hour|day|week)s?\s+ago", raw.strip().lower())
    if not m:
        return None
    return captured_at - timedelta(**{UNITS[m.group(2)]: int(m.group(1))})

captured = datetime.now(timezone.utc)
print(published_estimate("3 hours ago", captured))

Display text can depend on the interface language you request with hl, so a parser written for English may not match other languages. Rather than parse every format, keep None for the unknown ones and fall back to the first time you saw the article: for alerting, "first seen" is usually the timestamp that matters anyway.

Storing and deduplicating articles

The same article comes back on every run while it stays in the results. Split the data into what an article is and where it appeared:

create table news_articles (
  url_key      text primary key,          -- normalized link
  title        text not null,
  source       text,
  first_seen   timestamptz not null,
  published_at timestamptz                -- estimate, nullable
);

create table news_observations (
  run_id     bigint not null,
  query      text   not null,
  country    text   not null,
  url_key    text   not null references news_articles(url_key),
  position   int    not null,
  page       int    not null,
  date_raw   text,
  captured_at timestamptz not null,
  primary key (run_id, query, country, url_key)
);

Normalize link into url_key by lowercasing the host, dropping www., removing the fragment and common tracking parameters (utm_*), and trimming a trailing slash. Do not strip all query parameters: some publishers identify articles by them.

With this split, "new article" is an insert into news_articles, and "article moved up" is a comparison of position between two observations of the same url_key.

Writing queries that return what you mean

The query is the largest source of noise in news monitoring. Quotes for exact matches and - for exclusions are both documented in Google's Refine web searches help page (checked 2026-09-17). A few rules keep result sets reviewable:

Quote multi-word names. "acme corp" asks for the exact phrase; without quotes the two words are searched as separate terms.
Exclude known collisions. If your brand shares a name with a place, a film or a sports team, add - terms for the collision ("acme" -cartoon) and review what the exclusion removes on the first few runs.
One intent per query. Keep "brand news", "brand plus product", and "brand plus risk terms" as separate queries so each result set answers one question and can have its own schedule.
Match the market's language. Send local-language names and terms with the matching hl for each market; a translated English query can miss how local outlets write about you.
Version the query list. Store a version with every observation, so a change in results can be separated from a change in the query.

Pattern 1: brand and competitor alerts

Write one query per entity and variant: the brand name in quotes, the brand plus key product names, and the executives' names if they are public figures.
Run each query per market as an async task on a schedule.
On each webhook, insert new articles; alert on inserts whose title or snippet matches the entity.
Suppress repeats by url_key, and group alerts by source so one story syndicated across many outlets arrives as one message with a list.

Quoted phrases and operators such as - go inside query exactly as you would type them into Google. Google search operators lists the ones worth using.

Pattern 2: coverage share

Coverage share answers "who owns the news results for our topics?". For a fixed query set, count results per publisher domain or per brand mention and divide by all results:

from collections import Counter
from urllib.parse import urlparse

def publisher_share(results: list[dict], top_n: int = 10) -> list[tuple[str, float]]:
    """results: newsResults items from one run of a query set."""
    domains = Counter((urlparse(r["link"]).hostname or "").removeprefix("www.")
                      for r in results if r.get("page") == 1 and r.get("position", 99) <= top_n)
    total = sum(domains.values()) or 1
    return [(d, n / total) for d, n in domains.most_common()]

Limit to the first page or the top 10 positions so a query with more results does not dominate. Track the share weekly: a PR team sees whether coverage is concentrated in a few outlets, and a publisher sees which competitors win the topics it covers.

Pattern 3: adverse media screening

Compliance and risk teams screen names against negative news. The shape:

For each entity, combine the name with risk terms in separate queries ("Jane Example" fraud, "Jane Example" lawsuit), because a long OR chain returns a mixed set that is harder to review.
Run in each country where the entity operates, with the local hl, since local-language coverage often appears there first.
Store every observation, not only matches: an auditor may ask what the screen returned on a given date.
Route matches to a reviewer; do not auto-decide on a headline.

The adverse media screening use case describes the workflow end to end.

Scheduling

News moves quickly, but not every query needs the same frequency. A practical split:

Query type	Suggested interval	Why
Active incident or launch	15-60 minutes	Pickup speed matters
Brand and executive names	Every few hours	Catch stories the same day
Topic and competitor coverage	Daily	Share changes slowly
Screening of third parties	Daily or weekly	Driven by policy

Send each run as one batch of async tasks (up to 500 per request) with a webhook, so nothing polls. Use an idempotencyKey built from the run slot, query id and market, so a retried scheduler run creates no duplicates. See sync, async and webhooks and async tasks.

curl -X POST https://api.answerline.dev/v1/async/task/batch \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    { "taskType": "GOOGLE_NEWS", "idempotencyKey": "news-2026-09-15T10-acme-de",
      "payload": { "query": "\"acme corp\"", "country": "DE", "hl": "de" },
      "webhook": { "url": "https://your-app.example/hooks/news" } },
    { "taskType": "GOOGLE_NEWS", "idempotencyKey": "news-2026-09-15T10-acme-us",
      "payload": { "query": "\"acme corp\"", "country": "US" },
      "webhook": { "url": "https://your-app.example/hooks/news" } }
  ]'

Cost in credits

From the current credit table (see /pricing for credit prices):

Request	Credits
Google News task, 1 page	2
Google News task, 3 pages	6
Google News synchronous call, 1 page	4

A budget is queries × markets × runs per month × credits per run. 50 queries in 2 markets, hourly (about 720 runs a month), one page as async tasks: 50 × 2 × 720 × 2 = 144,000 credits. The same set every 6 hours is 24,000. Frequency is the largest lever, so match it to the table above rather than running everything at the fastest interval.

Pitfalls

Treating source as a key. Publisher display names change and collide. Key on the normalized link domain.
Parsing dates in one language. The date text follows hl. Keep the raw string and a first-seen timestamp.
Alerting on every run. Alert on new url_key inserts, not on every observation.
Using long OR chains. Split risk terms into separate queries so each result set is reviewable.

Field reference: Google News engine page. More patterns: news monitoring.

What in-house AI visibility tracking costs to run

Richard Nemeth — Sun, 27 Sep 2026 16:16:05 +0000

The cost of collecting AI answers in-house is rarely the API bill you avoid. It is the engineering time to reach production quality on each engine, plus the maintenance that follows for as long as you run it. Building is the right call when the collection layer is itself your product or you need data no provider returns; the build vs buy comparison covers that decision, and the framework below fills in the numbers behind it.

Definitions

Collection layer: everything between "we have a prompt" and "we have a structured record of what the engine answered": execution, parsing, retries and storage.
Analysis layer: what you build on top: share of voice, citation tracking, competitor reports, alerts. This is where customers compare products.
Run: one prompt, on one engine, in one market, collected once.

What building involves

Consumer AI products such as ChatGPT, Perplexity, Gemini, Copilot, Grok and Google AI Mode show answers that include web search, citations, product cards and local results. Those surfaces live in the consumer applications, not in the model providers' developer APIs. Each of the components below exists once per engine.

1. Collection runs

You run automated collection against each product and keep throughput inside what each product allows. That is infrastructure to provision, scale, patch and monitor, sized for your peak run.

2. Parsing that keeps up with interface changes

Each engine renders answers differently: inline citations, grouped sources, shopping cards, maps, follow-up queries. Each needs its own parser, a fixture set, and tests. When an interface changes, the failure can be silent: the run succeeds and a field such as sources comes back empty. Catching that takes data-quality monitoring per field, per engine.

3. Market coverage

Answers differ by country, and for local intent by region. Reporting per market requires collecting from each market and validating that the stored answers reflect it.

4. Queueing, retries and storage

At volume, partial failure is the normal case. You need a queue with priorities, retries that cannot create duplicates, per-run deadlines, result storage and a way to push results to downstream systems. That is a distributed-systems project in its own right.

5. Terms, compliance and review

Each consumer product has its own terms of service. Collecting from them at volume is a decision for your legal and compliance review, per engine, before engineering starts, and again when terms change.

6. On-call

Interfaces change without notice and on no schedule. Somebody owns the pager for each engine.

A cost framework you can fill in

Team cost, engine count and quality bar vary too much for universal numbers. Fill in this table with your own.

Variable	Meaning	Your number
E	Engines you need
B	Engineer-months to reach production quality on one engine
M	Engineer-months of maintenance per engine, per month
C	Loaded cost of one engineer-month
I	Monthly infrastructure: compute, storage, monitoring
R	One-off review and compliance cost
T	Months you plan to run it

Then:

Build, cost over T months = E × B × C + E × M × C × T + I × T + R
Buy, cost over T months = the monthly price of the plan that covers your credits (below) × T

Adjust for three effects the formulas miss:

Build is paid before value. E × B months pass before the first reliable dataset; for a product that competes on time to market, that delay has its own cost.
Maintenance scales with engines, not volume. A fifth engine adds a fifth parser and a fifth pager rotation, even if you run few prompts on it.
Opportunity cost. Every engineer-month in the collection layer is one not spent on the analysis layer.

Computing the buy side

Buying is priced in credits per request. As async tasks, the base price per request is:

Engine	Credits per request
ChatGPT	5
Gemini	4
Copilot	5
Google AI Mode	4
Google Search	3
Google News	2

Perplexity and Grok are paused for now and coming back soon; requests to them are refused and not billed. Synchronous calls add 2 credits each. Some options add to the price: on ChatGPT, +2 once if include.rawResponse or shopping is on (searchQueries and ads are free); on AI Mode, +1 per expanded product cluster, up to 6; on Google Search, +2 per extra page and +2 once for the AI Overview; on Google News, +2 per extra page. Credits are reserved at the maximum and charged at the actual cost on success; failed requests are charged nothing, and queued tasks you clear are never charged. The details are in credits.

Worked example 1: a large program

20,000 prompts on ChatGPT and Gemini in the US, plus 2,000 Google News queries in 10 markets:

Runs per sweep: 20,000 × 2 + 2,000 × 10 = 60,000.
Credits for the assistants: 20,000 × (5 + 4) = 180,000.
Credits for Google News: 2,000 × 10 × 2 = 40,000.
Credits per full sweep, async, no add-ons: 180,000 + 40,000 = 220,000.

A weekly sweep (4 a month) is 880,000 credits, inside the Growth plan's 1,350,000. A daily sweep is 6,600,000 a month and needs an Enterprise plan. Plan prices and a volume estimator are on the pricing page.

Worked example 2: a focused program

200 prompts on ChatGPT and Gemini in the US, plus 50 Google News queries in 3 markets, weekly (4 runs a month):

Credits per prompt: 5 + 4 = 9.
Per sweep: 200 × 9 + 50 × 3 × 2 = 1,800 + 300 = 2,100 credits.
Per month: 2,100 × 4 = 8,400 credits, inside the Lite plan's 37,500.

Unused monthly credits do not roll over, so size the plan to your steady cadence, not your peak. The cost planning guide covers sampling and cadence choices that change these numbers, and sampling vs census covers when you can run fewer prompts without losing the signal.

What the buy side returns

Compare the output as well as the price. Each engine returns structured JSON under result, with fields that differ by engine:

All AI engines: text, sources[], and markdown with include.markdown.
ChatGPT: citationPills[], entities[], shoppingCards[] (with include.shopping), map[]. The searchQueries[] (free with include.searchQueries), inlineProducts[] (with include.shopping) and ads[] (with include.ads) fields exist but come back empty on the answers served today.
Gemini: citationPills[], places[].

For volume, tasks go through POST /v1/async/task/batch (up to 500 per request) with idempotency keys and webhooks. Retries, deadlines and queueing are part of the API contract rather than code you maintain; see async tasks.

What you still build when you buy

Buying the collection layer moves engineering to the parts that differ between products:

Prompt set and schedule. Which prompts, engines and markets, and how often. Prompt set design and monitoring cadence cover the choices.
Submission and retries. Batching, idempotency keys derived from each task's meaning, and a retry policy that never duplicates work; see idempotency keys.
A webhook receiver. Signature verification over the raw body, deduplication by task.id, and storage; see verifying webhook signatures.
Normalization. Field names differ by engine (shopping_cards on Perplexity, shoppingCards on ChatGPT; citation ids 0-based on ChatGPT and 1-based elsewhere), so map them into your own schema once.
The analysis layer. Mentions, citations, share of voice and reports, which is what customers pay for.

SDKs and workflow integrations are listed under integrations; for AI agents, the MCP server exposes each engine as a tool.

Decision criteria

Situation	Lean
Answers feed an analytics, SEO or visibility product	Buy
You need several engines and several markets	Buy
Time to first reliable dataset matters	Buy
Collection itself is the product you sell	Build
You need a field no provider returns	Build that part, scoped to one engine
One engine, low volume, and spare engineering capacity	Either; compare with the framework

A hybrid also works: buy the engines and markets a provider covers, and build narrowly where you need something specific.

How to evaluate in a week

Pick your ten hardest prompts: the ones with shopping, local or citation-heavy answers that break parsers.
Run them on every engine you need. The free tier (500 credits, once, no card) works on every live engine, so you can evaluate the assistants before paying: 10 prompts as synchronous calls on ChatGPT and Gemini cost 10 × (9 + 2 × 2) = 130 credits. As async tasks, 10 × 9 = 90.
Compare field by field against what your pipeline or manual checks show: citations present, ordering, product cards, places.
Check the failure path: send a malformed request and a burst of requests, and confirm your code handles 422, 429 and batch item errors as described in the errors post.
Price your volume with the credit math above and your cadence.
Fill in the build table with your own team's estimates and compare.

Pitfalls in the comparison

Comparing one engine built to several engines bought. Price the scope you need.
Ignoring maintenance. First-version cost is the smaller term over any horizon longer than a few months.
Treating developer model APIs as equivalent. A completion from a model API does not include the consumer app's web search, citations or product cards; see provider APIs.
Pricing at peak cadence. Credits do not roll over; plan for the steady state.
Skipping the terms review. It belongs at the start of a build plan, not after launch.

The table version of this comparison is at build vs buy; to run your first prompts, start with the quickstart.