DEV Community: AnswerLine

Ask ChatGPT from inside Cursor with one MCP line

Richard Nemeth — Tue, 29 Sep 2026 15:24:29 +0000

You can ask ChatGPT and Gemini, and search Google News, from inside Cursor or Claude Code by adding one remote MCP server: https://mcp.answerline.dev/mcp, authenticated with your API key as a bearer token. Each engine then appears as a tool the editor's agent can call mid-session, and every call returns the same structured JSON as the REST API (answer text, sources[] and citation pills).

What the MCP server is

The Model Context Protocol (MCP) server is a hosted endpoint that exposes this API's monitor routes as tools an MCP client can list and call. It speaks Streamable HTTP over POST, and supports protocol revisions 2024-11-05 through 2025-11-25 as well as 2026-07-28 (MCP guide).

There are two endpoint forms:

Endpoint	Auth	Use when
`https://mcp.answerline.dev/mcp`	`Authorization: Bearer sk_...` header	The client can send headers. Cursor and Claude Code both can.
`https://mcp.answerline.dev/<API_KEY>/mcp`	Key in the path	The client only accepts a URL. The URL itself is then a secret.

Prefer the header form whenever the client supports it. A URL with a key in it ends up in logs, screenshots and shell history more easily than a header value.

Claude Code setup

Create an API key in the dashboard and export it in the shell you start Claude Code from: export API_KEY=sk_....
Add the server:

claude mcp add --transport http answerline \
  https://mcp.answerline.dev/mcp \
  --header "Authorization: Bearer $API_KEY"

Start a session and ask for something that needs a tool, for example "check my AnswerLine credits". The agent should call credits.

Two details from the Claude Code MCP documentation (code.claude.com/docs/en/mcp, checked 2026-09-17):

Scope decides where the key is stored. The default local scope and the user scope write to ~/.claude.json, which is private to you. The projectscope writes .mcp.json in the project root, which is meant to be shared through version control. Because your shell expands $API_KEY before claude mcp add runs, the literal key is what gets stored. Never use --scope project with a literal key.
Shared configs can reference variables. .mcp.json supports ${VAR} expansion in url and headers, so a committed project file can hold "Authorization": "Bearer ${ANSWERLINE_API_KEY}" while each developer sets the variable locally. The same page documents MCP_TOOL_TIMEOUT for the tool execution timeout, which matters for long engine calls.

A project-scoped file that is safe to commit:

{
  "mcpServers": {
    "answerline": {
      "type": "http",
      "url": "https://mcp.answerline.dev/mcp",
      "headers": { "Authorization": "Bearer ${ANSWERLINE_API_KEY}" }
    }
  }
}

Cursor setup

Cursor reads MCP servers from .cursor/mcp.json in the project root or ~/.cursor/mcp.json for all projects, supports stdio, SSE and Streamable HTTP transports, and takes remote servers as a url with optional headers (cursor.com/docs/context/mcp, checked 2026-09-17).

Pick the file. Use ~/.cursor/mcp.json if you want the tools everywhere; use .cursor/mcp.json if the project needs them and your teammates should get the entry too.
Add the server, referencing an environment variable for the key. Cursor resolves ${env:NAME} in url and headers (same source):

{
  "mcpServers": {
    "answerline": {
      "url": "https://mcp.answerline.dev/mcp",
      "headers": { "Authorization": "Bearer ${env:ANSWERLINE_API_KEY}" }
    }
  }
}

Set ANSWERLINE_API_KEY in the environment Cursor is started from; a variable exported in a terminal opened later is not visible to an editor that is already running.
Confirm the server is enabled in Cursor's MCP settings and that the tools are listed. If the server shows an error, Cursor's Output panel has an "MCP Logs" channel with connection and authentication errors (same source).
Expect an approval prompt. Cursor asks for approval before running MCP tools by default (same source). Leave that on while you learn what the calls cost; each approved engine call spends credits.

Other editors and agents configured with an mcpServers JSON file that supports remote servers take the sameurlandheaders` shape; check your client's own documentation for variable interpolation before relying on it.

Testing the key and endpoint

If the editor shows no tools, test the endpoint directly. First the REST API, which isolates key problems:

bash curl -s https://api.answerline.dev/v1/credits \ -H "Authorization: Bearer $API_KEY"

A valid key returns remaining, perCycle and cycleResetsAt. A 401 with MISSING_API_KEY, INVALID_API_KEY_FORMAT or INVALID_OR_EXPIRED_API_KEY means the key, not the MCP setup, is the problem (authentication).

Then ask the MCP server for its tool list with a plain JSON-RPC request:

bash curl -s https://mcp.answerline.dev/mcp \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -H "Accept: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

The response lists the tools with their input schemas. GET and DELETE on the endpoint answer 405, so a browser visit to the URL tells you nothing; use POST.

The tools and what each call costs

Tool names follow the route: /v1/monitor/google/news becomes google_news. Each engine tool's input schema is that endpoint's request body, so the agent passes prompt and country to chatgpt, or query and country to google_news, plus any options the endpoint accepts.

Every engine tool call is a synchronous API call. It costs the engine's base price plus the 2-credit synchronous surcharge:

Tool	Route	Base	Cost per tool call	Common add-ons
`chatgpt`	`/v1/monitor/chatgpt`	5	5 + 2 = 7	+2 once for `include.rawResponse` or `shopping`
`gemini`	`/v1/monitor/gemini`	4	4 + 2 = 6	none
`copilot`	`/v1/monitor/copilot`	5	5 + 2 = 7	none
`aimode`	`/v1/monitor/aimode`	4	4 + 2 = 6	+1 per expanded product cluster, up to 6
`google`	`/v1/monitor/google`	3	3 + 2 = 5	+2 per extra page; +2 once for the AI Overview
`google_news`	`/v1/monitor/google/news`	2	2 + 2 = 4	+2 per extra page
`credits`	`GET /v1/credits`	none	0	none

The perplexity and grok tools are paused for now and coming back soon, and are not listed until then.

Worked examples:

ChatGPT with include.shopping: true: 5 + 2 (feature group) + 2 (sync) = 9 credits.
Google News, pages: 2: 2 + 2 (one extra page) + 2 (sync) = 6 credits.
The Free plan's 500 one-time credits work on every engine tool: 71 plain chatgpt or copilot calls at 7, 83 gemini or aimode calls at 6, 100 google calls at 5, or 125 google_news calls at 4.

Failed calls are charged nothing, and the charged amount of each call is in the result's _meta as x-credits-charged, next to x-credits-remaining. See credits for the full rules.

Slots and timeouts

Two limits shape how an editor agent should use these tools.

Concurrency slots

A synchronous call holds one of your plan's concurrency slots for its whole run. The Free plan has 1 slot, Lite 10, Hobby 20, Starter 50, Growth 75 and Business 100 (rate limits). An agent that fires three tool calls in parallel on the Free plan will see the extra calls come back with CONCURRENT_LIMIT_EXCEEDED. Tell the agent to run engine calls one at a time unless you know your plan has room.

Duration

A synchronous call can take up to five minutes. The server streams keep-alive comments to clients that accept text/event-stream, but the client's own tool timeout still applies. In Claude Code, MCP_TOOL_TIMEOUT controls it; in other clients, look for a tool or request timeout setting and set it above five minutes where you can.

Keep the key out of the repository

The key spends credits on your account, so treat every place it can land as sensitive:

Never commit a config file with a literal key. Reference an environment variable (${env:NAME} in Cursor, ${VAR} in Claude Code's .mcp.json) and commit only that.
Treat the key-in-path URL as a password. https://mcp.answerline.dev/<API_KEY>/mcp should never appear in a README, an issue, a chat message or a screen recording.
Use a separate key for editor work. If a laptop config leaks, you can revoke that key alone; a revoked key stops working within a minute.
Git-ignore any project config that holds a literal key, such as a .cursor/mcp.json you have not yet switched to a variable.
Keep approvals on. Cursor's default approval prompt is also a spending check.

Errors the agent should handle

API errors do not break the MCP session. The server returns a normal tool result with isError: true and the API's error body as text, and puts the request id in _meta under x-request-id. The model can read the code and act:

Code	Meaning	What the agent should do
`CONCURRENT_LIMIT_EXCEEDED`	All slots were busy	Wait for its other call to finish, then retry once
`INSUFFICIENT_CREDITS`	Balance too low for the call	Stop and tell you; do not retry
`RATE_LIMIT_EXCEEDED`	Over the per-second request limit	Retry after a short pause
`400` "Request validation failed" with `details[]`	A field was wrong, for example a `prompt` over 10,000 characters	Read `details[].field` and fix the arguments
`EXTERNAL_SERVICE_ERROR` (502)	The engine request failed	Report it; the call was not charged

A standing instruction for the agent: "Call engine tools sequentially. If a result has isError, read the code; stop on INSUFFICIENT_CREDITS and report the x-request-id for anything else you cannot fix."

Example session

"Before we ship this pricing page, ask ChatGPT and Gemini 'best invoicing tool for freelancers' in the US, list every domain each one cites, and tell me which of our competitors appear."

The agent makes two calls, chatgpt (7 credits) and gemini (6 credits), 13 in total. It reads sources[].url from both results, reduces them to domains, and compares the two sets. It can also match brand names in each answer's text to see which brands were named even where no source is cited. Don't ask it for ChatGPT's underlying searches: include.searchQueries is free, but searchQueries[] comes back empty on the ChatGPT answers served today (search queries).

Because the JSON lands in the editor's context, the agent can then write a monitoring script against the field names it has just seen.

When to leave the editor

Editor tools are for questions you ask now. Recurring work, such as the same 200 prompts every Monday, belongs in async tasks and batches: they skip the 2-credit surcharge (200 ChatGPT prompts cost 200 × 5 = 1,000 credits as tasks versus 200 × 7 = 1,400 as tool calls), queue instead of failing when slots are busy, and can deliver results by webhook (sync, async and webhooks). For agent architecture beyond the editor, see giving an AI agent live answer data and the LangChain tools; for tracking competitors over time, see competitor AI answer monitoring.

Full tool reference: /docs/mcp.

The queries ChatGPT runs before it answers yours

Richard Nemeth — Mon, 28 Sep 2026 13:35:17 +0000

When ChatGPT answers with web search, it does not have to search your prompt word for word. It can run several searches, narrower, rephrased or with added context, and its cited sources come from what those searches return. That set of searches is the query fan-out. ChatGPT does not show it to users, and on the ChatGPT answers this API serves today the searchQueries field comes back empty. What you can see is the result of the fan-out: the pages cited, in order, and the answer text. This post covers how to reason about the fan-out from those.

Google uses the same term for AI Mode: its announcement describes AI Mode breaking a question into subtopics and running many searches at once with a "query fan-out technique" (Google, May 20, 2025, checked 2026-09-17).

Definitions

Query fan-out: the searches an AI assistant runs to answer one prompt.
Fan-out query: one search in that set. The API field for them is result.searchQueries, empty on the ChatGPT answers served today.
Source frequency: the number of answers in a prompt set that cite a given domain.
Inferred search: a search you work out from the cited pages and the answer text. It is a hypothesis, not returned data.

One prompt, several searches, one cited answer
A single request carrying one prompt fans out into several web searches, which ChatGPT runs but does not return today. The pages those searches return feed one answer, which comes back as result.text together with sources and citationPills, each entry carrying a url and a label.

POST /v1/monitor/chatgpt
one prompt
one request
search 1
search 2
search 3
search 4
result.text
sources[]
url, label
citationPills[]
url, domain
searches ChatGPT ran: not returned today

The searches in the middle column are what the answer was actually built from, so they, not the prompt's own wording, are the queries a page has to win. For ChatGPT you can't see them today; the right-hand column is what you can see.

Why the fan-out matters

A prompt such as "iPhone 17 or Galaxy S26?" is one question to the user and several retrieval problems to the assistant: specs, camera comparisons, battery tests, prices, maybe reviews from a specific year. The pages that win those narrower searches are the ones the answer can cite. Three consequences for anyone working on AI visibility:

Your target keyword is not the retrieval query. A page optimized for the prompt's wording can miss the searches actually run.
Fan-out queries can repeat across prompts. When different prompts in one category trigger overlapping searches, one page that serves that search is relevant to many answers.
The fan-out leaves traces. ChatGPT does not return its searches, but the cited sources are what those searches found. Which domains get cited, and in what order, is the measurable side of the fan-out.

What the API returns

{
  "prompt": "iPhone 17 or Galaxy S26?",
  "country": "US"
}

Send it to POST /v1/monitor/chatgpt or as the payload of a CHATGPT task. The result carries text, sources[] in the order ChatGPT shows them, and citationPills[]. The contract also has result.searchQueries for the fan-out, requested with include.searchQueries: true at no extra cost, but it comes back empty on the ChatGPT answers served today.

Two request details decide whether you get useful data:

Web search is forced by default. Leave disableWebSearch unset. With true, you get the answer ChatGPT gives on its own, which searches only when it decides to, so sources and citationPills are often empty. That mode is useful for a different question (what does the model say without retrieval?), not for studying retrieval.
Market matters. country is required. ChatGPT answers from 213 countries; US-state targeting is coming soon. Run each market as its own series and don't pool markets before you have compared them.

curl -X POST https://api.answerline.dev/v1/monitor/chatgpt \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "best CRM for a 10-person B2B sales team",
    "country": "US"
  }'

Aggregate cited sources across a prompt set

One answer's sources say little; across 50 or 200 prompts they give a distribution. Count each cited domain once per answer that cited it:

from collections import Counter
from urllib.parse import urlparse

def domain(url: str) -> str:
    return (urlparse(url).hostname or "").removeprefix("www.")

def source_report(results: list[dict]) -> list[tuple[str, int]]:
    """Cited domains with the number of answers that cited them, most frequent first."""
    counts = Counter()
    for result in results:
        counts.update({domain(s["url"]) for s in result.get("sources", []) if s.get("url")})
    return counts.most_common()

results are result objects: from synchronous responses, or response.result of completed tasks.

Counting per answer, not per occurrence, keeps one answer that cites a domain several times from inflating it. Also report coverage: the share of answers that cited any sources at all. If only 40% did, your counts describe that 40%, and you should say so next to the numbers.

Reason about the searches you can't see

Without the searches, work backwards from what was cited and what was said:

Read the cited pages. Each source's label and description say what the page is about, and so what the search that found it must have matched. A page about camera tests for two phones was found by a camera search, not by the prompt as typed.
Read the answer text for subtopics. An answer that covers specs, price, battery and reviews in turn most likely searched for each. Each subtopic is a candidate query for your content plan.
Group prompts by shared sources. Prompts whose answers cite the same domains probably rely on overlapping searches, and one page can serve all of them.

from collections import defaultdict

def prompts_by_domain(rows: list[tuple[str, dict]]) -> dict[str, set[str]]:
    """rows are (prompt_id, result) pairs. Maps each cited domain to the prompts whose answers cite it."""
    m: dict[str, set[str]] = defaultdict(set)
    for prompt_id, result in rows:
        for s in result.get("sources", []):
            if s.get("url"):
                m[domain(s["url"])].add(prompt_id)
    return m

This is inference, not returned data: a cited page tells you a search found it, not which search. Label inferred searches as such in reports. How AI engines choose citations covers the source side in more depth.

Turn it into work

Rank topics by how many answers cite pages on them across the prompt set.
Map each topic to a page. Note the page on your site that should answer it, or that none does. A missing page for a frequently cited topic is the clearest content gap you will find.
Check the organic side. Inferred searches are still searches, so run the top ones through a Google Search request and see who ranks. A domain that ranks organically and is cited in answers holds both positions; one that ranks but is never cited suggests the page doesn't answer the narrower question well. Structured SERP data shows the request.
Split by market. Run the set with each country you sell in; fan-out can differ by market, and so do the cited domains.
Keep the set fixed. Re-run the same prompts on a schedule so changes in the cited sources are visible, and version the prompt set when you change it. Prompt set design covers versioning.
Feed new prompts back. Subtopics that recur across answers are good candidates for new monitoring prompts, because they reflect how the engine decomposes your category.

A report that people act on

A report content teams can act on has four columns per topic:

Topic (inferred from cited pages)	Answers citing it	Top cited domains	Our page
best crm small business	34 of 150	review site A, vendor B, publisher C	/crm-for-small-teams (not cited)
crm pricing comparison	21 of 150	vendor B, publisher D	none
crm with email automation	12 of 150	vendor E, our domain	/features/email (cited 5 times)

How to read it:

High frequency, no page: a content gap. Write the page that answers the narrower question directly.
High frequency, page exists, not cited: the page is not what the engine retrieves or uses for that topic. Compare it with the cited pages: scope, freshness, structure, and whether it answers the exact question the topic implies.
Page cited: protect it. Keep it current and watch the topic in the next runs.

Add a fifth column with a change indicator against the previous run of the same prompt set version, and the report doubles as a trend view. Keep the denominator ("of 150") visible, and state that topics are inferred from cited pages, not from returned searches.

Fan-out on other engines

Some engines return their searches, with different field names:

Engine	Field	How to get it	Base credits
ChatGPT	`result.searchQueries`	`include.searchQueries: true` (free); empty on the answers served today	5
Copilot	`result.searchQueries`	Coming; not returned yet	5
Grok	`result.searchQueries`	Returned in the response	4
Perplexity	`result.search_model_queries`	Returned in the response	4

Grok and Perplexity are paused for now and coming back soon. Perplexity also returns related_queries, its suggested follow-up searches. Those are suggestions for the user, not searches it ran, so keep them in a separate table. The contract also defines product.generatedProductQuery on ChatGPT inline products, a product-level cousin of the fan-out, but inlineProducts comes back empty on the answers served today; see ChatGPT shopping cards for the product data you get.

Where an engine returns searches, count each normalized search once per answer that ran it, and group near-duplicates ("best crm small business 2026", "best CRM for small businesses") before counting. Normalize every engine's queries the same way so you can see which query groups every engine runs and which are engine-specific.

Run it at scale

For a prompt set, send async tasks in batches (up to 500 per request) with a webhook:

[
  { "taskType": "CHATGPT", "idempotencyKey": "fanout-v3-2026-w38-crm-012-US",
    "payload": { "prompt": "best CRM for a 10-person B2B sales team", "country": "US" },
    "webhook": { "url": "https://your-app.example/hooks/answerline" } }
]

Put the prompt set version in the idempotency key, as above, so a re-run under a new version never collides with an old one. Sync, async and webhooks explains the flow.

Cost, from the current credit table (/pricing has credit prices): 150 prompts Ă— 1 market Ă— weekly (about 4.3 runs a month) Ă— 5 credits for a ChatGPT task â‰ˆ 3,200 credits a month. include.searchQueries is free, so turning it on does not change that.

Pitfalls

Reading an empty searchQueries as "no search". On ChatGPT today it means the searches were not exposed, not that none ran; check sources.
Counting occurrences instead of answers. One answer citing a domain several times should count once.
Treating inferred searches as data. A search worked out from cited pages is a hypothesis; label it that way.
Disabling web search by accident. disableWebSearch: true changes the question you are measuring.
Pooling markets early. Compare markets before merging them.
Judging one run. Answers and their sources vary between runs; trend over repeated runs. AI answer volatility covers sampling.
Comparing across prompt set versions. A reworded prompt produces a different fan-out. Compare runs only within one version, and when you promote a new version, run old and new side by side once so the change in counts can be attributed to the wording rather than to the engine.

See the full ChatGPT field list on the ChatGPT engine page, or start with the quickstart.

If you want ChatGPT's answer and cited sources as JSON instead of reading browser traffic: POST /v1/monitor/chatgpt returns them in one call. Free tier, no card: https://answerline.dev

Is there a Google Search API? What exists in 2026

Richard Nemeth — Sun, 27 Sep 2026 20:55:18 +0000

No. As of September 2026 Google does not offer new customers an API that returns its web search results. The Custom Search JSON API, the closest thing Google had, is closed to new customers, and existing customers must move off it by January 1, 2027. What Google does offer are search products for your own sites and data, and a way to let Gemini models search the web and answer. For Google's actual results page as data, teams use a third-party SERP API or build a scraper.

Definitions

Google Search API (as people search for it): a programmatic way to send a query and get Google's web results back as data.
SERP: the search engine results page, including organic results and modules such as People Also Ask, ads, local packs and AI Overviews. See what is a SERP.
SERP API: a service that runs searches on Google and returns the SERP as structured JSON.
Grounding: giving a language model search results as context so its answer cites current sources.

What Google offers

Everything in this section was checked on Google's own pages on 2026-09-17.

Custom Search JSON API (closing)

The overview page states that the Custom Search JSON API is closed to new customers and that existing customers have until January 1, 2027 to transition. It returns JSON results for a query against a Programmable Search Engine you configure. For existing customers it lists 100 free queries per day, then $5 per 1,000 queries, up to 10,000 queries per day. The cse.list reference limits each request to 10 results and a query to 100 results in total.

Even for customers who have it, it was never Google.com as data. Google's help page says an engine configured to search the entire web is limited to a subset of the Google Web Search corpus, emphasizes results from your own sites, and lacks features such as Oneboxes, real-time results and universal search. Migration options are covered in Google Custom Search JSON API alternatives.

Programmable Search Engine

Programmable Search Engine lets you create a search engine for your website, blog or a collection of websites. Since January 20, 2026, per the help page, new engines must use "Sites to search", covering up to 50 designated domains; existing engines set to "Search the entire web" can keep that setting until January 1, 2027. It is a site search product, not a web search API.

The related Custom Search Site Restricted JSON API ceased to serve traffic on January 8, 2025.

Agent Search (formerly Vertex AI Search)

Google's Custom Search overview names Vertex AI Search as an alternative for searching up to 50 domains, and asks customers needing full web search to contact Google. Google Cloud's documentation now calls the product Agent Search (formerly Vertex AI Search): enterprise search and recommendations over your public websites and data stores with your own data. It is the right tool for searching content you own. It is not described as a way to query Google's public web results.

Grounding with Google Search (Gemini API)

Grounding with Google Search connects a Gemini model to real-time web content. The model decides what to search, and the response contains the synthesized answer with citations and the queries the model ran; the documentation also refers to usage requirements for displaying search suggestions. The pricing page lists, for Gemini 3.x models, 5,000 free search requests per month shared across those models, then $14 per 1,000 requests, with each search query the model executes billed; Gemini 2.5 models are billed per grounded prompt.

This is useful when your end goal is an answer for a user. It does not give you positions, a list of ten results, or the modules on the page, so it cannot answer "where do we rank" or "what does the AI Overview say".

SERP APIs

A SERP API runs the query on Google for a chosen market, language, location and device, parses the page, and returns JSON. Pricing models differ (monthly plans, pay-as-you-go per 1,000, credits), and so do the fields each one extracts. Examples with prices from their own pages, checked 2026-09-17:

Provider	Published price	Source
SerpApi	Free: 250 searches/month; Starter $25/month for 1,000; Developer $75/month for 5,000	serpapi.com/pricing
DataForSEO (Google Organic)	Standard queue $0.6 per 1K SERPs; Priority $1.2 per 1K; Live $2 per 1K	dataforseo.com pricing
Serper	2,500 free queries, no credit card	serper.dev

Broader comparisons: best SERP APIs, cheapest SERP API, best Google scraper, and SerpApi compared.

What this API returns

A request to this API's POST /v1/monitor/google:

curl -X POST https://api.answerline.dev/v1/monitor/google \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "query": "crm for small business", "country": "US", "pages": 2, "include": { "aioverview": { "markdown": true } } }'

returns result.organicResults[] (position, title, link, displayedLink, snippet, date, page, sitelinks), peopleAlsoAsk[], relatedSearches[], ads[], and, when Google shows them, localResults[], shoppingCards[], knowledgeGraph, peopleAreSaying[] and the aioverview with sources[] and citationPills[]. Targeting fields are country or gl, hl, location or uule, device and pages (1 to 10); alternatively, pass a full Google search URL in url. See Google search parameters.

Python, with requests:

import os

import requests

r = requests.post(
    "https://api.answerline.dev/v1/monitor/google",
    headers={"Authorization": f"Bearer {os.environ['ANSWERLINE_API_KEY']}"},
    json={"query": "crm for small business", "country": "US", "pages": 2},
    timeout=360,
)
r.raise_for_status()
for o in r.json()["result"]["organicResults"]:
    print(o["page"], o["position"], o["link"])

TypeScript, with fetch:

const res = await fetch("https://api.answerline.dev/v1/monitor/google", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.ANSWERLINE_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ query: "crm for small business", country: "US", pages: 2 }),
});
if (!res.ok) throw new Error(`${res.status}: ${await res.text()}`);
const { result } = await res.json();
for (const o of result.organicResults) console.log(o.page, o.position, o.link);

Cost: a Google Search async task is 3 credits, each page after the first 2 more, include.aioverview or include.paaAioverview 2 more once, and a synchronous call like the ones above adds 2. The curl request above (2 pages with AI Overview, synchronous) is 3 + 2 + 2 + 2 = 9 credits.

Building your own scraper

Building in-house means you own every part of the pipeline: request infrastructure, market and location targeting, parsers for each SERP module, change detection, retries and storage. The ongoing costs:

Layouts change. Modules are added, renamed and restructured; every change is a silent parser break until you detect it.
Parameters change. In September 2025, Search Engine Roundtable reported that the &num=100 parameter stopped working and that this broke most Google rank trackers. A scraper built on 100 results per request has to switch to paging when that happens.
Terms and legal exposure. Review Google's terms of service and your jurisdiction's rules with counsel before collecting data at scale.
Operational load. Queues, rate control, monitoring and on-call for a system that is not your product.

Getting around bot detection is out of scope here and is not a sound basis for a business. For a structured build-versus-buy analysis, see build vs buy.

Comparison

	Custom Search JSON API	Programmable Search / Agent Search	Grounding with Google Search	SERP API	Own scraper
Available to new customers	No	Yes	Yes	Yes	n/a
Data source	Subset of Google's corpus via a configured engine	Your sites, up to 50 domains, your data	Google Search, used by the model	Google's results page	Google's results page
Output	Up to 10 links per request	Search results over your content	Generated answer with citations	Structured SERP JSON	Whatever you parse
Positions and SERP modules	No modules	n/a	No	Yes	If you build them
AI Overview content	No	No	No	Depends on provider	If you build it
Maintenance on your side	Low, until January 1, 2027	Low	Low	Low	High

Evaluating a SERP API

Provider pages look alike. Run the same test on each candidate before you commit:

Build a query set that reflects your use. 50 to 200 queries from your own workload across your markets, languages and devices, including local-intent queries if you track locations and a few queries where you know an AI Overview appears.
Check field coverage, not just organic results. For each response, record whether People Also Ask, ads (with their block position), local results, the knowledge graph and the AI Overview with its sources are present when Google shows them. A missing module is data you cannot report on.
Check targeting. Confirm the provider can set country, interface language, city-level location and device independently. City targeting by canonical location name or uule matters for local work.
Check depth. Since results come in pages of about ten, ask how the provider handles depth (pages per request, pricing per page) and whether results carry page and position numbers.
Check failure semantics. Are failed requests charged? Is there an async mode with webhooks, batch submission and idempotency keys, so retries do not create duplicate charges? See idempotency keys.
Price the whole workload. Multiply queries by markets, devices, pages and frequency, add optional features such as the AI Overview, and compare the monthly total rather than the headline price per 1,000.
Read the contract. An OpenAPI document lets you generate clients and validate responses; undocumented fields tend to change.

For this API, the fields, limits and error shapes are in the API reference.

When to use which

Search box for your own site or docs: Programmable Search Engine or Agent Search.
Chatbot or agent that needs current facts with citations: Grounding with Google Search, or a SERP API if you want to choose which results reach the model. See search APIs for AI agents.
Rank tracking, SERP feature monitoring, AI Overview tracking: a SERP API. See Google rank tracking API, Google AI Overview API and SERP feature change tracking.
Keyword research from live SERPs: a SERP API that returns People Also Ask and related searches; see People Also Ask keyword research.
Search data is your core product and you have a team for it: consider building, knowing the maintenance and legal work involved.

Pitfalls

Treating Custom Search results as Google rankings. Google itself says the corpus and features differ.
Using grounding to measure visibility. You see the model's answer, not the results page.
Planning around undocumented URL parameters. num=100 stopped working without notice; anything Google does not document can change.
Comparing providers on price alone. Check which fields each returns for your queries: an API that omits the AI Overview or local pack is cheaper because it gives you less.

Try a request from the quickstart.

How to get Google News results as structured JSON

Richard Nemeth — Sun, 27 Sep 2026 16:20:07 +0000

The Google News API takes a query and returns the articles Google News shows for it, in order, as data: headline, publisher, link, snippet, date and thumbnail, for a chosen country and language. Below: the request and response, date normalization, storage and deduplication, and three patterns (brand alerts, coverage share, adverse media screening).

If you are deciding between RSS feeds and structured results, read Google News RSS feeds: what they give you and where they stop first.

Definitions

News result: one article on the Google News results page for a query, returned as an item of result.newsResults[].
Position: the 1-indexed order of the article in the news results, returned with the page it appeared on.
Source: the publisher name as Google displays it, such as a newspaper or a trade site.
Coverage share: the fraction of news results for a query set that belong to a given publisher or mention a given brand.

The request

curl -X POST https://api.answerline.dev/v1/monitor/google/news \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "acme corp recall",
    "country": "DE",
    "hl": "de",
    "pages": 2
  }'

Request fields:

Field	Type	Notes
`query`	string, 1-10,000 chars	Required. The News search.
`country`	string	ISO 3166-1 alpha-2. Send `country` or `gl`.
`gl`	string	Google's name for the result geography; same codes, either case. Different values in `country` and `gl` are a 400.
`hl`	string	Interface language (`en`, `de`, `pt-BR`). Overrides the language derived from the country.
`pages`	integer 1-10	Default 1.
`include.html`	boolean	Not available on Google News: a request that sets it is refused with a 400.

News targeting is by country and language. Supported countries are listed by GET /v1/countries?model=google. Don't route News through the Google Search endpoint's url shape: a Google URL carrying tbm=nws is rejected there, because News has its own endpoint and pricing.

The response

{
  "success": true,
  "result": {
    "newsResults": [
      {
        "position": 1,
        "title": "Acme Corp expands recall to second product line",
        "link": "https://news.example.de/acme-recall",
        "snippet": "The company said on Tuesday that...",
        "source": "Example Nachrichten",
        "date": "3 hours ago",
        "page": 1,
        "thumbnail": "https://news.example.de/images/acme.jpg"
      }
    ]
  }
}

Field by field:

position: 1-indexed rank in the news results. With pages above 1, page tells you which page the article came from; store both.
title, snippet: the headline and the text excerpt shown.
link: the article URL. If Google served it as a redirect, redirectLink carries Google's redirect next to it.
source: the publisher's display name. It is a name, not a domain; derive the domain from link when you need a stable key.
date: the date text as displayed, for example "2 hours ago".
thumbnail: the image URL when one is shown.
html[]: not returned for Google News; include.html is refused on this endpoint.

Normalizing dates

date is display text, such as "2 hours ago", so do not assume one format. Store three things: the raw string, the capture time, and a parsed estimate.

import re
from datetime import datetime, timedelta, timezone

UNITS = {"minute": "minutes", "hour": "hours", "day": "days", "week": "weeks"}

def published_estimate(raw: str, captured_at: datetime) -> datetime | None:
    """Parse English relative dates like '3 hours ago'. Returns None for formats it doesn't know."""
    m = re.match(r"(\d+)\s+(minute|hour|day|week)s?\s+ago", raw.strip().lower())
    if not m:
        return None
    return captured_at - timedelta(**{UNITS[m.group(2)]: int(m.group(1))})

captured = datetime.now(timezone.utc)
print(published_estimate("3 hours ago", captured))

Display text can depend on the interface language you request with hl, so a parser written for English may not match other languages. Rather than parse every format, keep None for the unknown ones and fall back to the first time you saw the article: for alerting, "first seen" is usually the timestamp that matters anyway.

Storing and deduplicating articles

The same article comes back on every run while it stays in the results. Split the data into what an article is and where it appeared:

create table news_articles (
  url_key      text primary key,          -- normalized link
  title        text not null,
  source       text,
  first_seen   timestamptz not null,
  published_at timestamptz                -- estimate, nullable
);

create table news_observations (
  run_id     bigint not null,
  query      text   not null,
  country    text   not null,
  url_key    text   not null references news_articles(url_key),
  position   int    not null,
  page       int    not null,
  date_raw   text,
  captured_at timestamptz not null,
  primary key (run_id, query, country, url_key)
);

Normalize link into url_key by lowercasing the host, dropping www., removing the fragment and common tracking parameters (utm_*), and trimming a trailing slash. Do not strip all query parameters: some publishers identify articles by them.

With this split, "new article" is an insert into news_articles, and "article moved up" is a comparison of position between two observations of the same url_key.

Writing queries that return what you mean

The query is the largest source of noise in news monitoring. Quotes for exact matches and - for exclusions are both documented in Google's Refine web searches help page (checked 2026-09-17). A few rules keep result sets reviewable:

Quote multi-word names. "acme corp" asks for the exact phrase; without quotes the two words are searched as separate terms.
Exclude known collisions. If your brand shares a name with a place, a film or a sports team, add - terms for the collision ("acme" -cartoon) and review what the exclusion removes on the first few runs.
One intent per query. Keep "brand news", "brand plus product", and "brand plus risk terms" as separate queries so each result set answers one question and can have its own schedule.
Match the market's language. Send local-language names and terms with the matching hl for each market; a translated English query can miss how local outlets write about you.
Version the query list. Store a version with every observation, so a change in results can be separated from a change in the query.

Pattern 1: brand and competitor alerts

Write one query per entity and variant: the brand name in quotes, the brand plus key product names, and the executives' names if they are public figures.
Run each query per market as an async task on a schedule.
On each webhook, insert new articles; alert on inserts whose title or snippet matches the entity.
Suppress repeats by url_key, and group alerts by source so one story syndicated across many outlets arrives as one message with a list.

Quoted phrases and operators such as - go inside query exactly as you would type them into Google. Google search operators lists the ones worth using.

Pattern 2: coverage share

Coverage share answers "who owns the news results for our topics?". For a fixed query set, count results per publisher domain or per brand mention and divide by all results:

from collections import Counter
from urllib.parse import urlparse

def publisher_share(results: list[dict], top_n: int = 10) -> list[tuple[str, float]]:
    """results: newsResults items from one run of a query set."""
    domains = Counter((urlparse(r["link"]).hostname or "").removeprefix("www.")
                      for r in results if r.get("page") == 1 and r.get("position", 99) <= top_n)
    total = sum(domains.values()) or 1
    return [(d, n / total) for d, n in domains.most_common()]

Limit to the first page or the top 10 positions so a query with more results does not dominate. Track the share weekly: a PR team sees whether coverage is concentrated in a few outlets, and a publisher sees which competitors win the topics it covers.

Pattern 3: adverse media screening

Compliance and risk teams screen names against negative news. The shape:

For each entity, combine the name with risk terms in separate queries ("Jane Example" fraud, "Jane Example" lawsuit), because a long OR chain returns a mixed set that is harder to review.
Run in each country where the entity operates, with the local hl, since local-language coverage often appears there first.
Store every observation, not only matches: an auditor may ask what the screen returned on a given date.
Route matches to a reviewer; do not auto-decide on a headline.

The adverse media screening use case describes the workflow end to end.

Scheduling

News moves quickly, but not every query needs the same frequency. A practical split:

Query type	Suggested interval	Why
Active incident or launch	15-60 minutes	Pickup speed matters
Brand and executive names	Every few hours	Catch stories the same day
Topic and competitor coverage	Daily	Share changes slowly
Screening of third parties	Daily or weekly	Driven by policy

Send each run as one batch of async tasks (up to 500 per request) with a webhook, so nothing polls. Use an idempotencyKey built from the run slot, query id and market, so a retried scheduler run creates no duplicates. See sync, async and webhooks and async tasks.

curl -X POST https://api.answerline.dev/v1/async/task/batch \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    { "taskType": "GOOGLE_NEWS", "idempotencyKey": "news-2026-09-15T10-acme-de",
      "payload": { "query": "\"acme corp\"", "country": "DE", "hl": "de" },
      "webhook": { "url": "https://your-app.example/hooks/news" } },
    { "taskType": "GOOGLE_NEWS", "idempotencyKey": "news-2026-09-15T10-acme-us",
      "payload": { "query": "\"acme corp\"", "country": "US" },
      "webhook": { "url": "https://your-app.example/hooks/news" } }
  ]'

Cost in credits

From the current credit table (see /pricing for credit prices):

Request	Credits
Google News task, 1 page	2
Google News task, 3 pages	6
Google News synchronous call, 1 page	4

A budget is queries × markets × runs per month × credits per run. 50 queries in 2 markets, hourly (about 720 runs a month), one page as async tasks: 50 × 2 × 720 × 2 = 144,000 credits. The same set every 6 hours is 24,000. Frequency is the largest lever, so match it to the table above rather than running everything at the fastest interval.

Pitfalls

Treating source as a key. Publisher display names change and collide. Key on the normalized link domain.
Parsing dates in one language. The date text follows hl. Keep the raw string and a first-seen timestamp.
Alerting on every run. Alert on new url_key inserts, not on every observation.
Using long OR chains. Split risk terms into separate queries so each result set is reviewable.

Field reference: Google News engine page. More patterns: news monitoring.

What in-house AI visibility tracking costs to run

Richard Nemeth — Sun, 27 Sep 2026 16:16:05 +0000

The cost of collecting AI answers in-house is rarely the API bill you avoid. It is the engineering time to reach production quality on each engine, plus the maintenance that follows for as long as you run it. Building is the right call when the collection layer is itself your product or you need data no provider returns; the build vs buy comparison covers that decision, and the framework below fills in the numbers behind it.

Definitions

Collection layer: everything between "we have a prompt" and "we have a structured record of what the engine answered": execution, parsing, retries and storage.
Analysis layer: what you build on top: share of voice, citation tracking, competitor reports, alerts. This is where customers compare products.
Run: one prompt, on one engine, in one market, collected once.

What building involves

Consumer AI products such as ChatGPT, Perplexity, Gemini, Copilot, Grok and Google AI Mode show answers that include web search, citations, product cards and local results. Those surfaces live in the consumer applications, not in the model providers' developer APIs. Each of the components below exists once per engine.

1. Collection runs

You run automated collection against each product and keep throughput inside what each product allows. That is infrastructure to provision, scale, patch and monitor, sized for your peak run.

2. Parsing that keeps up with interface changes

Each engine renders answers differently: inline citations, grouped sources, shopping cards, maps, follow-up queries. Each needs its own parser, a fixture set, and tests. When an interface changes, the failure can be silent: the run succeeds and a field such as sources comes back empty. Catching that takes data-quality monitoring per field, per engine.

3. Market coverage

Answers differ by country, and for local intent by region. Reporting per market requires collecting from each market and validating that the stored answers reflect it.

4. Queueing, retries and storage

At volume, partial failure is the normal case. You need a queue with priorities, retries that cannot create duplicates, per-run deadlines, result storage and a way to push results to downstream systems. That is a distributed-systems project in its own right.

5. Terms, compliance and review

Each consumer product has its own terms of service. Collecting from them at volume is a decision for your legal and compliance review, per engine, before engineering starts, and again when terms change.

6. On-call

Interfaces change without notice and on no schedule. Somebody owns the pager for each engine.

A cost framework you can fill in

Team cost, engine count and quality bar vary too much for universal numbers. Fill in this table with your own.

Variable	Meaning	Your number
E	Engines you need
B	Engineer-months to reach production quality on one engine
M	Engineer-months of maintenance per engine, per month
C	Loaded cost of one engineer-month
I	Monthly infrastructure: compute, storage, monitoring
R	One-off review and compliance cost
T	Months you plan to run it

Then:

Build, cost over T months = E × B × C + E × M × C × T + I × T + R
Buy, cost over T months = the monthly price of the plan that covers your credits (below) × T

Adjust for three effects the formulas miss:

Build is paid before value. E × B months pass before the first reliable dataset; for a product that competes on time to market, that delay has its own cost.
Maintenance scales with engines, not volume. A fifth engine adds a fifth parser and a fifth pager rotation, even if you run few prompts on it.
Opportunity cost. Every engineer-month in the collection layer is one not spent on the analysis layer.

Computing the buy side

Buying is priced in credits per request. As async tasks, the base price per request is:

Engine	Credits per request
ChatGPT	5
Gemini	4
Copilot	5
Google AI Mode	4
Google Search	3
Google News	2

Perplexity and Grok are paused for now and coming back soon; requests to them are refused and not billed. Synchronous calls add 2 credits each. Some options add to the price: on ChatGPT, +2 once if include.rawResponse or shopping is on (searchQueries and ads are free); on AI Mode, +1 per expanded product cluster, up to 6; on Google Search, +2 per extra page and +2 once for the AI Overview; on Google News, +2 per extra page. Credits are reserved at the maximum and charged at the actual cost on success; failed requests are charged nothing, and queued tasks you clear are never charged. The details are in credits.

Worked example 1: a large program

20,000 prompts on ChatGPT and Gemini in the US, plus 2,000 Google News queries in 10 markets:

Runs per sweep: 20,000 × 2 + 2,000 × 10 = 60,000.
Credits for the assistants: 20,000 × (5 + 4) = 180,000.
Credits for Google News: 2,000 × 10 × 2 = 40,000.
Credits per full sweep, async, no add-ons: 180,000 + 40,000 = 220,000.

A weekly sweep (4 a month) is 880,000 credits, inside the Growth plan's 1,350,000. A daily sweep is 6,600,000 a month and needs an Enterprise plan. Plan prices and a volume estimator are on the pricing page.

Worked example 2: a focused program

200 prompts on ChatGPT and Gemini in the US, plus 50 Google News queries in 3 markets, weekly (4 runs a month):

Credits per prompt: 5 + 4 = 9.
Per sweep: 200 × 9 + 50 × 3 × 2 = 1,800 + 300 = 2,100 credits.
Per month: 2,100 × 4 = 8,400 credits, inside the Lite plan's 37,500.

Unused monthly credits do not roll over, so size the plan to your steady cadence, not your peak. The cost planning guide covers sampling and cadence choices that change these numbers, and sampling vs census covers when you can run fewer prompts without losing the signal.

What the buy side returns

Compare the output as well as the price. Each engine returns structured JSON under result, with fields that differ by engine:

All AI engines: text, sources[], and markdown with include.markdown.
ChatGPT: citationPills[], entities[], shoppingCards[] (with include.shopping), map[]. The searchQueries[] (free with include.searchQueries), inlineProducts[] (with include.shopping) and ads[] (with include.ads) fields exist but come back empty on the answers served today.
Gemini: citationPills[], places[].

For volume, tasks go through POST /v1/async/task/batch (up to 500 per request) with idempotency keys and webhooks. Retries, deadlines and queueing are part of the API contract rather than code you maintain; see async tasks.

What you still build when you buy

Buying the collection layer moves engineering to the parts that differ between products:

Prompt set and schedule. Which prompts, engines and markets, and how often. Prompt set design and monitoring cadence cover the choices.
Submission and retries. Batching, idempotency keys derived from each task's meaning, and a retry policy that never duplicates work; see idempotency keys.
A webhook receiver. Signature verification over the raw body, deduplication by task.id, and storage; see verifying webhook signatures.
Normalization. Field names differ by engine (shopping_cards on Perplexity, shoppingCards on ChatGPT; citation ids 0-based on ChatGPT and 1-based elsewhere), so map them into your own schema once.
The analysis layer. Mentions, citations, share of voice and reports, which is what customers pay for.

SDKs and workflow integrations are listed under integrations; for AI agents, the MCP server exposes each engine as a tool.

Decision criteria

Situation	Lean
Answers feed an analytics, SEO or visibility product	Buy
You need several engines and several markets	Buy
Time to first reliable dataset matters	Buy
Collection itself is the product you sell	Build
You need a field no provider returns	Build that part, scoped to one engine
One engine, low volume, and spare engineering capacity	Either; compare with the framework

A hybrid also works: buy the engines and markets a provider covers, and build narrowly where you need something specific.

How to evaluate in a week

Pick your ten hardest prompts: the ones with shopping, local or citation-heavy answers that break parsers.
Run them on every engine you need. The free tier (500 credits, once, no card) works on every live engine, so you can evaluate the assistants before paying: 10 prompts as synchronous calls on ChatGPT and Gemini cost 10 × (9 + 2 × 2) = 130 credits. As async tasks, 10 × 9 = 90.
Compare field by field against what your pipeline or manual checks show: citations present, ordering, product cards, places.
Check the failure path: send a malformed request and a burst of requests, and confirm your code handles 422, 429 and batch item errors as described in the errors post.
Price your volume with the credit math above and your cadence.
Fill in the build table with your own team's estimates and compare.

Pitfalls in the comparison

Comparing one engine built to several engines bought. Price the scope you need.
Ignoring maintenance. First-version cost is the smaller term over any horizon longer than a few months.
Treating developer model APIs as equivalent. A completion from a model API does not include the consumer app's web search, citations or product cards; see provider APIs.
Pricing at peak cadence. Credits do not roll over; plan for the steady state.
Skipping the terms review. It belongs at the start of a build plan, not after launch.

The table version of this comparison is at build vs buy; to run your first prompts, start with the quickstart.