DEV Community: Ricardo Batista

What does it take to run a headless browser farm for ChatGPT at scale? The engineering build

Ricardo Batista — Fri, 09 Oct 2026 20:10:00 +0000

A headless browser farm for ChatGPT at scale takes five components in every browser and three systems around the browsers. Each browser needs a stealth-patched Chromium, a sticky residential proxy session, a network interceptor on the conversation endpoint, a completion detector and a citation expander. Around them, the farm needs a pool of warm sessions, routing per region and monitoring per field.

The build is the smaller cost. One published estimate from cloro puts a self-run ChatGPT scraper at roughly $4,800 to $14,100 a month at 500,000 to 1 million requests, with engineer time as the largest line. The alternatives are rented browser infrastructure such as Bright Data or Browserbase, where the parser stays yours, and a managed endpoint such as cloro, which bills 5 credits for each ChatGPT web-search answer.

The response is a Server-Sent Events stream, so the scraper reads the network and not the DOM.
The failures that cost the most return a success status with part of the data missing.
Playwright wins when answer data is your product. A paid API wins when the data is an input.

What are the components of a headless browser farm for ChatGPT?

Each browser in the farm runs the same five parts. Each part answers one specific failure.

Stealth-patched Playwright Chromium. A WebDriver-controlled browser exposes navigator.webdriver as true, and detection scripts read that flag on the first navigation. Apply playwright-stealth or equivalent canvas, WebGL and audioContext patches.
A residential proxy with a sticky session. Datacenter IPs get a CAPTCHA on the first prompt. Cloudflare sets the cf_clearance cookie after a visitor solves a challenge and skips further challenges while the cookie is valid. The session must keep that cookie across the round trip.
A response interceptor scoped to backend-api/f/conversation. ChatGPT makes other API calls during the page lifecycle. Only this URL carries the stream.
A completion detector that looks for [DONE]. DOM checks for "is the response complete" return false positives during streaming. The [DONE] sentinel in the stream body is the reliable signal.
A citation expander. Citations are not in the stream. The scraper clicks button.group/footnote after the response completes and parses the flyout at [data-testid="screen-threadFlyOut"].

The interceptor is a Playwright response handler:

async def setup_page_interceptor(self, page: Page):
    """Set up network request interception."""

    async def handle_response(response):
        # Capture conversation API responses
        if 'backend-api/f/conversation' in response.url:
            response_body = await response.text()
            self.captured_responses.append(response_body)

    page.on('response', handle_response)

The captured body is raw SSE text. The parser splits it on data: lines, drops the sentinel and keeps the valid JSON objects:

import json
from typing import List, Dict, Any

def extract_raw_response(input_string: str) -> List[Dict[str, Any]]:
    """Parse ChatGPT's Server-Sent Events stream."""
    json_objects = []

    # Split by lines that start with "data: "
    lines = input_string.split("\n")

    for line in lines:
        # Skip empty lines and non-data lines
        if not line.strip() or not line.startswith("data: "):
            continue

        # Remove "data: " prefix
        json_str = line[6:].strip()

        # Skip special markers like [DONE]
        if json_str == "[DONE]":
            continue

        # Try to parse as JSON
        try:
            json_obj = json.loads(json_str)

            # Only include if it's a dictionary (object), not string or other types
            if isinstance(json_obj, dict):
                json_objects.append(json_obj)
        except json.JSONDecodeError:
            # Skip invalid JSON
            continue

    return json_objects

The full scraper class that these two excerpts come from launches Chromium with headless=False. That choice is deliberate. Headless defaults are the first signal that bot detection reads, so a production farm runs full browser environments with consistent canvas, font and TLS fingerprints.

What actually breaks when you build an LLM scraper in-house?

Five failure modes hit a ChatGPT scraper in production. We run about 1,000 engine answers a day across six engines for cloro's own tracking corpus, and these are the failures that operation meets.

Failure	What you see	Why a retry does not fix it
Response format shift	The answer arrives as HTML fragments. The model name and search queries are absent, and the source list is shorter.	OpenAI serves a share of logged-out traffic in a mobile-web format and selects the format per request.
Model rotation	Fields that depend on the model, such as fan-out queries under `gpt-5-3`, appear on some answers only.	ChatGPT selects the serving model per request. No parameter controls it.
Regional login wall	Anonymous sessions fail for one geography.	The provider changed its access rules for that region.
Anti-bot escalation	Rate limits, session churn and fingerprint checks start at volume.	A browser that works at 10 answers a day meets them at 10,000.
Silent quality drift	The scrape succeeds and the source list is missing half its entries.	A DOM class changed. No request failed.

The first and last rows cost the most. A parser built against one format reports the other as a partial success, and a partial success sends no alert.

The targets also move. The backend-api/f/conversation URL changed twice in 2025. The shopping-card event moved from a flat products array to a nested offers block. OpenAI's build pipeline generates CSS class names that change between deploys, so a scraper that relies on class selectors breaks roughly weekly. ARIA labels, role attributes and text content last longer than class names, and both still break.

How do you keep the farm from getting blocked at scale?

The farm must look and behave like the logged-out users that ChatGPT already serves. Do these four things.

Keep a pool of established sessions. Engines challenge new sessions harder than warm ones, and one cleared session serves many requests before rotation.
Match the exit region to the request. An answer requested for Germany must exit in Germany. Traffic concentrated in one region meets volume ceilings.
Space the retries and add jitter. Immediate identical retries are a bot signature.
Treat a degraded format as a block. It costs the engine less to serve a reduced response to suspected automation than to refuse it.

The IP type sets the ceiling. In testing published by proxies.sx, real mobile IPs lasted 50 to 100 or more queries, and datacenter IPs were blocked within the first few requests. That figure is the vendor's and was not reproduced for this article.

How do you know the farm still returns good data?

Status codes do not show the two failures that cost the most, so monitor the fields. Track these rates per day:

answers with an empty model name
answers with a short source list
answers with no fan-out queries where you expect them

A change in any of the three is a format change or a model rotation in your sample.

One invariant helps with ChatGPT. We measured grounding on roughly 2,500 prompts per engine, and ChatGPT grounded 98.4% of its answers in live sources at 14.1 sources each. A re-measurement on 2026-08-19 gave 78.9% and 12.7 sources. An empty citation list from a ChatGPT scrape is more often a parse failure than an ungrounded answer, so assert a source list that is not empty in your tests.

Playwright or a paid API for scraping AI chat answers?

Playwright is the correct choice when answer data is your product, your volume pays for a dedicated team and you need control that a vendor cannot give. It is free and open source, every layer is yours to tune and there is no vendor lock-in. A team with no budget and strong engineers can also run it at low volume. The roundup linked below budgets 8 to 15 engineer hours a month for a Playwright stack at 1,000 queries a day.

A paid API is the correct choice when answer data is an input to another product. Under roughly 1,000 answers a day, a provider costs less as soon as you price your own time. Two kinds of paid API exist, and they are not the same purchase:

Browser and proxy infrastructure (Bright Data, Oxylabs, Browserbase, Browserless). The access layer is solved. The SSE parser, the citation expander and the selectors stay with you. Bright Data prices its hosted Browser API from $5 per GB of traffic.
A managed answer endpoint (cloro is one). The provider owns all five components and returns parsed JSON.

The managed call looks like this:

import requests
import json

# Your prompt
prompt = "Compare the top 3 programming languages for web development in 2026"

# API request to cloro
response = requests.post(
    'https://api.cloro.dev/v1/monitor/chatgpt',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    json={
        'prompt': prompt,
        'country': 'US',
        'include': {
            'markdown': True,
            'rawResponse': True,
            'searchQueries': True
        }
    }
)

result = response.json()
print(json.dumps(result, indent=2))

A ChatGPT web-search answer costs 5 credits there as an async request. A synchronous request, as in the sample above, adds 2 credits. 5,000 prompts a day is about 750,000 credits a month, which fits in the Growth plan at $500 a month for 1,350,000 credits. That is $1.85 per 1,000 answers. Failed requests are not billed. The trade is control: you cannot tune a session pool that you do not run.

Neither option fits if you want the model's output and not the answer that ChatGPT shows its users. That is the OpenAI API's job, priced in tokens.

How these figures were collected

The failure modes come from cloro's own tracking operation, about 1,000 engine answers a day across six engines. The components and the code come from a working Playwright scraper for the logged-out ChatGPT surface. The $4,800 to $14,100 figure is an estimate and not a measured bill. It adds list prices ($999 for 332 GB of residential proxy bandwidth on Bright Data, $496 a month for each c6i.4xlarge browser instance) to a fifth to a half of one engineer at the US median software developer salary of $131,450, loaded.

This article is published by cloro, which is one of the managed options in it. Figures for other vendors are those vendors' published figures. The cloro prices are self-reported and were checked against its price list on 2026-10-07.

The source pages

Failure modes, SLAs and cost per answer: LLM scraping API reliability at scale
The full Playwright scraper, with the citation, shopping-card and entity parsers: How to scrape ChatGPT
Eight ChatGPT scrapers compared, with Playwright as the DIY option: Best ChatGPT scraper

How to get the UI answers of ChatGPT through an API (if you want what users see, not the OpenAI API output)

Ricardo Batista — Thu, 08 Oct 2026 20:11:00 +0000

To get the UI answers of ChatGPT through an API, call a service that runs your prompt in the ChatGPT web interface and returns the rendered answer as JSON. The OpenAI API does not do this. It returns model output, without the web search, source citations, shopping cards and entity panels that the web interface adds. Three routes return the UI answer: a managed ChatGPT endpoint such as cloro, a scraping platform such as Apify or Bright Data, and a Playwright stack that you run yourself.

With a managed endpoint the job is one POST request. cloro's endpoint is POST https://api.cloro.dev/v1/monitor/chatgpt. You send a prompt and a country. The response carries text, sources, citationPills and entities, plus markdown, searchQueries and shoppingCards when you ask for them. If you need model output for your own application, the OpenAI API is the correct tool and you do not need a scraper.

Key takeaways

The OpenAI API and the ChatGPT web UI are different surfaces. Citations, shopping cards, brand entities and the search queries ChatGPT issued exist only in the web UI.
At 1,000 queries a day, the source comparison estimates $100 to $300 a month for a managed API and $980 to $2,140 a month for DIY Playwright.
DIY is the better choice with zero budget, a strong engineering team and 8 to 15 hours a month for maintenance.

Steps with a managed endpoint

Get an API key from the provider.
Send a POST request to https://api.cloro.dev/v1/monitor/chatgpt with prompt and country.
Set the include flags for the optional fields that you need.
Set the client timeout to 300 seconds or more.
Read the parsed fields from result in the JSON response.

Why does the OpenAI API return a different answer than the ChatGPT UI?

The chat.completions API and the chatgpt.com web interface use different stacks. The API answers from a base model. The web interface does more work before the answer reaches the screen:

It breaks the prompt into sub-queries (query fan-out).
It runs a web search against OpenAI's search index.
It adds citations from the search results.
For shopping prompts, it adds a product card with merchant offers.

Two measurements show the size of the difference. Across roughly 2,500 prompts, we found that ChatGPT grounded 98.4% of its answers in live sources, at 14.1 sources each. A second measurement on 2026-08-19 gave 78.9% and 12.7 sources. And Surfer's analysis reports only about 20% overlap between API responses and the answers that users see in the ChatGPT UI. That figure is Surfer's and was not reproduced here.

Is there an API that returns what ChatGPT actually shows real users?

Yes. A ChatGPT scraper API runs the prompt in the consumer product and returns the answer that a user sees. OpenAI does not offer one. Its developer API (platform.openai.com) returns model completions, with no source pills, shopping cards or ads.

Four data types exist only in the web UI: sources and citations, shopping cards with merchant offers, brand entities, and the sub-queries that ChatGPT issued before it answered.

How do I get ChatGPT responses programmatically without using the OpenAI API?

There are four routes. The cost column is an estimate at 1,000 queries a day from cloro's comparison of 8 tools, with proxies, CAPTCHA solving and engineer time included.

Route	Examples	What you still build	Estimated cost per month
Managed ChatGPT endpoint	cloro	Nothing. Fields arrive parsed.	$100 to $300
Scraping platform	Apify actors, Bright Data Scraping Browser	Cookie refresh (Apify) or the parsing layer (Bright Data)	$280 to $900
Browser infrastructure or anti-bot API	Browserbase, Browserless, ScrapingBee, ZenRows	Stream assembly, citation parsing, selectors	$449 to $1,040
DIY framework	Playwright with stealth plugins	All of it	$980 to $2,140

The DIY route needs five components:

A stealth-patched Playwright Chromium. Vanilla Playwright is detected on the first navigation.
A residential proxy with a sticky session. Datacenter IPs get a CAPTCHA on the first prompt.
A network interceptor for backend-api/f/conversation. Only that URL carries the answer stream.
A completion check that waits for the [DONE] line in the stream.
A step that clicks the sources button and reads the flyout after the answer completes. Citations are not in the stream.

ChatGPT sends the answer as Server-Sent Events. The first function a DIY scraper needs is the parser for that stream:

import json
from typing import List, Dict, Any

def extract_raw_response(input_string: str) -> List[Dict[str, Any]]:
    """Parse ChatGPT's Server-Sent Events stream."""
    json_objects = []

    # Split by lines that start with "data: "
    lines = input_string.split("\n")

    for line in lines:
        # Skip empty lines and non-data lines
        if not line.strip() or not line.startswith("data: "):
            continue

        # Remove "data: " prefix
        json_str = line[6:].strip()

        # Skip special markers like [DONE]
        if json_str == "[DONE]":
            continue

        # Try to parse as JSON
        try:
            json_obj = json.loads(json_str)

            # Only include if it's a dictionary (object), not string or other types
            if isinstance(json_obj, dict):
                json_objects.append(json_obj)
        except json.JSONDecodeError:
            # Skip invalid JSON
            continue

    return json_objects

The complete Playwright class, the citation extractor and the shopping-card parser are in the full DIY guide.

How do you call an API for the ChatGPT UI answer?

This request comes from the public chatgpt-scraper README:

import requests

payload = {
    'prompt': 'best project management software for remote teams',
    'country': 'US',
    'include': {'markdown': True, 'searchQueries': True},
}

response = requests.post(
    'https://api.cloro.dev/v1/monitor/chatgpt',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    json=payload,
)

print(response.json())

The same call with cURL, this time with shopping cards:

curl -X POST https://api.cloro.dev/v1/monitor/chatgpt \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "best running shoes for flat feet", "country": "US", "include": {"shopping": true}}'

Parameter	Description	Default
`prompt` (required)	The query or question, 1 to 10,000 characters	none
`country` (required)	Country code for localized results (`US`, `GB`, `DE`)	none
`state`	US state code	none
`include.markdown`	Return the answer as Markdown	`false`
`include.html`	Return a URL to the full HTML, which expires after 24 hours	`false`
`include.rawResponse`	Return the unparsed upstream payload	`false`
`include.searchQueries`	Return the query fan-out terms	`false`
`include.shopping`	Return shopping cards and inline products	`false`
`include.ads`	Return sponsored blocks	`false`
`disableWebSearch`	Return the answer ChatGPT gives without a forced web search	`false`

The response:

{
  "success": true,
  "result": {
    "text": "For remote teams, the strongest options are...",
    "citationPills": [{ "citationPillId": 0, "label": "Asana", "url": "https://asana.com", "domain": "asana.com" }],
    "searchQueries": ["best project management software 2026", "asana vs monday remote teams"],
    "markdown": "For remote teams, the strongest options are **Asana**..."
  }
}

Field in `result`	Content
`text`, `markdown`	The answer as plain text and as Markdown
`sources`	Each cited URL with `position`, `url`, `label` and `description`
`citationPills`	The inline citation chips. Entries share a `citationPillId` when one pill cites several sources.
`entities`	The products, brands and concepts that the answer names, each with a type
`searchQueries`	The search terms that ChatGPT ran
`shoppingCards`	Product carousels. Each one groups `products[]` with price and merchant offers.
`inlineProducts`	Products in the answer text, separate from the carousels
`ads`	Sponsored blocks
`map`	Business and location entries with rating, reviews and address
`rawResponse`	The unparsed upstream payload

Three details affect a production client:

Timeout. The README recommends 300 seconds or more, because failed attempts are retried on the server before the response returns. A typical response takes 30 to 45 seconds.
Search queries. The README states that searchQueries comes back empty on ChatGPT's default mobile-web interface. The legacy: true option asks for the interface that returns it, on a best-effort basis.
Batches. For thousands of prompts a day, use POST /v1/async/task with taskType: "CHATGPT" and receive the results by webhook.

The price list has two ChatGPT rows. "ChatGPT (web search)" is the base request at 5 credits. "ChatGPT (full response)" is 7 credits: the base request plus one flat charge of 2 credits for any of include.rawResponse, include.searchQueries, include.ads and include.shopping. These are async prices. A synchronous request to /v1/monitor/chatgpt adds 2 credits, so the two sample requests above cost 9 credits each. The Hobby plan is $100 a month for 250,000 credits. A panel of 1,000 prompts a day uses about 210,000 credits a month at 7 credits, or about 150,000 at 5 credits. The Free plan is 500 credits a month.

What should I use instead of building on unofficial ChatGPT API wrappers?

Use a route that someone maintains against ChatGPT's changes, or budget for that maintenance yourself. Code that depends on ChatGPT's private web endpoint breaks when the endpoint changes. The backend-api/f/conversation path changed twice in 2025. The source guide also records two changes to the stream chunk shape in one year, and the shopping-card event moved from a flat products array to a nested offers block. OpenAI's CSS class names also change between deploys, so class selectors break roughly weekly.

API routers such as LiteLLM and OpenRouter are a different category. They wrap the official API, so they return the API answer and not the UI answer.

The contract matters too. In a 2023 thread on OpenAI's developer forum, users report that the Terms of Use bar data extraction by scraping other than through the API. That risk is highest for an account that accepted the terms. Read the current OpenAI Terms of Use before you start. This is not legal advice.

When is DIY or the OpenAI API the better choice?

The OpenAI API is the better choice when your application needs generated text. A scraper endpoint is built for monitoring. A typical answer takes 30 to 45 seconds and each query is billed, which is expensive for casual use.
DIY Playwright is the better choice with zero budget and a strong engineering team. It is free, open source and has no vendor lock-in. Plan for 8 to 15 engineer hours a month.
A managed endpoint fits recurring monitoring: brand mentions, citation tracking and the same prompt from several countries. At 500,000 to 1 million requests a month, the source guide estimates $4,800 to $14,100 a month for an in-house stack, and engineer time is the largest part.

How these figures were collected

We measured grounding on roughly 2,500 prompts for each of six AI engines. For the tool comparison, we ran 30 queries through chatgpt.com with each of 8 tools and scored six criteria: auth handling, stream assembly, citation extraction, anti-bot survival, selector stability and cost per query.

The monthly costs are estimates. They assume engineer time at $100 an hour, a CAPTCHA solver at $2 per 1,000 challenges on 5% of requests, and mobile residential proxies at $10 per GB. The request, the parameters and the response fields come from the public README and were not changed.

This tutorial is published by cloro, which is one of the options in it. The cloro cost row and field list are self-reported. The 20% overlap figure belongs to Surfer.

Sources and full guides

The DIY build, with the complete Playwright scraper: How to scrape ChatGPT
The 8-tool comparison and the cost table: Best ChatGPT scraper 2026
The endpoint page: ChatGPT scraper API

What should I migrate to now that Google's Custom Search JSON API is retiring?

Ricardo Batista — Wed, 07 Oct 2026 20:11:29 +0000

Migrate to a SERP API if you used Google's Custom Search JSON API to search the open web. DataForSEO, cloro and SerpApi each return the live Google results page as JSON, and Brave Search API returns results from an independent index. If you searched your own site or a fixed list of up to 50 domains, migrate to Vertex AI Search or Agent Search, which are the two products Google recommends.

The deadline is January 1, 2027. Google's documentation states that the API is closed to new customers and that existing customers have until that date to move. Both of Google's replacements are site-restricted, so neither one searches the open web.

Site search: follow Google's migration guide to Agent Search.
Open-web search: move to a SERP API. The request changes in four places: the endpoint, the authentication, the query parameter and the results array.
Price: Custom Search charges $5 per 1,000 queries. At ten results, DataForSEO Live lists $2.00 per 1,000 and cloro's Hobby plan works out to $1.20 per 1,000.

What is the deadline for the Custom Search JSON API?

The Custom Search JSON API stops being available to existing customers on January 1, 2027. It is already closed to new customers. Until the deadline, the published terms apply: 100 free queries a day, $5 per 1,000 queries above that, and a limit of 10,000 queries a day.

The change also covers Programmable Search Engine. According to Brave's summary of the announcement, each engine created since January 20, 2026 must use the "Sites to search" setting, which has a limit of 50 domains. An existing engine set to "Search the entire web" keeps that setting until January 1, 2027.

A new API key fails today. A developer reported on r/googlecloud in January 2026 that a new key returned 403 Forbidden, although the engine ID worked in the console.

What does Google recommend as a replacement?

Google names two products, and both are site-restricted.

The API overview recommends Vertex AI Search for search across up to 50 domains. For anything larger it refers you to a full web search solution, and you must contact Google to get it.
The migration guide recommends Agent Search, which it describes as Google-quality site-restricted search. You deploy it as a widget or through an API.

The migration guide describes these steps:

Enable Agent Search in the Google Cloud Console.
Create an app configured for site search with AI mode.
Turn on Enterprise edition features. The guide makes them mandatory for website search.
Create a data store and populate it with website content.
Specify the URL patterns to index.

Authentication changes too. Custom Search used an API key. Agent Search primarily expects OAuth 2.0, and its searchLite method still accepts an API key. The guide also notes that summaries and domain verification can carry additional indexing costs.

Google plans a paid open-web service for large customers. An email from Google's Programmable Search Engine team, posted on Hacker News, describes an enterprise Web Search Service at $15 per 1,000 requests with a minimum monthly fee of $30,000. The email says more details are due later in 2026. That minimum equals 2,000,000 requests a month.

Which replacement fits your workload?

What you used Custom Search for	Migrate to	Reason
Search on your own site, or on a fixed list of up to 50 domains	Vertex AI Search or Agent Search	Google built both for site-restricted search
Google results from the open web, billed pay as you go	DataForSEO (Live)	$2.00 per 1,000 queries with no monthly minimum
Google results with the AI Overview in the same call	cloro	The AI Overview and its cited sources are in the response
Google plus other engines	SerpApi	It covers more than 80 engines, with Bing, Baidu and Yandex among them
Bing results	SerpApi or DataForSEO	Both return Bing results directly
A non-Google index	Brave Search API	It runs an independent index
Context for an LLM agent, where rank position is not needed	A retrieval API such as Tavily or Exa	These return relevance-ranked content and not the Google results page

Two limits apply to cloro. Its plans are monthly, so a workload with low or irregular volume can cost less on a pay-as-you-go provider. It does not return Bing results.

A SERP API has no cx parameter and no site restriction. If the site restriction was the reason you used Custom Search, the first row is your case.

What does each open-web replacement cost per 1,000 queries?

Each rate is for 1,000 queries at ten results, which is the shape Custom Search returned. Each rate is the vendor's published real-time list price.

Option	Rate per 1,000 queries	Index	AI Overview
Custom Search JSON API (until 2027)	$5.00, with a limit of 10,000 a day	Programmable Search Engine	No
DataForSEO (Live)	$2.00, or $4.00 with AI Overview	Google	Yes, behind a flag
cloro (Hobby plan)	$1.20, or $2.00 with AI Overview	Google	Yes
SerpApi (Big Data plan)	$9.17	Google plus more than 80 engines	Yes
Brave Search API	$5.00	Independent index	No
ScraperAPI	About $12.00 through its structured Google endpoint	Google	No
Google Web Search Service (planned)	$15.00, with a $30,000 monthly minimum	Google	Not stated

The plan sets the rate on the two subscription vendors.

A cloro Google call costs 3 credits, or 5 credits with the AI Overview. These are the prices of an async request. A synchronous request to /v1/monitor/google adds 2 credits.
The Hobby plan is $100 a month for 250,000 credits. The two calls cost $1.20 and $2.00 per 1,000.
The Growth plan is $500 a month for 1,350,000 credits. The same calls cost $1.11 and $1.85 per 1,000.
The Lite plan is $30 a month for 37,500 credits. The same calls cost $2.40 and $4.00 per 1,000.
SerpApi's Big Data plan is $275 a month for 30,000 searches.

The daily limit also goes away. Custom Search stops at 10,000 queries a day at any price, and the per-call vendors have no equivalent limit.

The results page has changed since most Custom Search integrations were written. In cloro's SERP feature census, we scored 65,945 results and found an AI Overview on 87.2% of US results and People Also Ask on 90.6%. Custom Search returned neither block. A port that keeps only the ten organic links omits both.

How do Custom Search JSON fields map to a SERP API response?

The Custom Search request and response:

GET https://www.googleapis.com/customsearch/v1
  ?key=<key>&cx=<engine id>&q=serp+api
  Results: items[] → { title, link, snippet, displayLink }

The same query on a SERP API, with cloro's Google endpoint as the worked example:

POST https://api.cloro.dev/v1/monitor/google
  Authorization: Bearer <key>
  { "query": "serp api", "country": "US",
    "include": { "aioverview": { "markdown": true } } }
  Results: result.organicResults[] → { position, title, link, snippet, displayedLink }

Custom Search	SERP API (worked example)
`q`	`query`
`key` query parameter	`Authorization: Bearer` header
`cx` (search engine ID)	Not needed. There is no engine to configure.
`items[]`	`result.organicResults[]`
`title`	`title`
`link`	`link`
`snippet`	`snippet`
`displayLink`	`displayedLink`
No equivalent	`position`
No equivalent	`result.aioverview`, with cited sources

Three of the four result fields keep their names. Other vendors use different paths, so check the array name in the vendor's documentation before you write the parser.

How do you migrate from Custom Search to a SERP API?

Steps 3 to 8 use the field names from the worked example above.

Find out which half of Custom Search you used. If your engine was restricted to your own sites, follow Google's migration guide and stop here.
Select a vendor from the workload table.
Replace the GET https://www.googleapis.com/customsearch/v1 request with a POST to the vendor's endpoint.
Move the API key from the key query parameter to the Authorization: Bearer header.
Remove the cx parameter.
Rename q to query and send it in the JSON body.
Read the results from result.organicResults[] and not from items[].
Rename displayLink to displayedLink in your parser. Keep title, link and snippet as they are.
Decide whether to store position and result.aioverview. Custom Search returned neither.
Send a real query set to the new API on its free tier. The free tier in the worked example is 500 credits a month, and SerpApi's is 250 searches a month.
Compare the new results with your current Custom Search results for the same queries.
Run both APIs in parallel for one month before January 1, 2027.

Should you wait for a first-party replacement from Google?

No like-for-like replacement arrived the last time a first-party search API was retired. Microsoft retired the Bing Search API v7 on August 11, 2025, and the endpoints now return HTTP 410 Gone. The official replacement was Grounding with Bing Search in Azure AI Agents, which is an agent product and not a SERP API. Teams that waited for a like-for-like successor then migrated under time pressure.

Google has given more notice. The first-party path again leads to a product with a different shape: site search for most customers, and a $30,000 monthly minimum for the open web.

How these figures were collected

The dates, limits and migration steps come from Google's Custom Search JSON API overview and Google's migration guide, as read on September 2, 2026. The Programmable Search Engine change is from Brave's summary. The Web Search Service price is from an email posted on Hacker News and Google has not published it on a pricing page. Each vendor rate is that vendor's published list price for its real-time mode, and prices in this category change often.

This guide is published by cloro, which is one of the providers in it. Its own figures are self-reported. The census figures come from 65,945 results that we scored.

The full guide

The full shutdown guide with the cost chart is here: Google Custom Search API shutdown: what to use instead. The comparison of 9 SERP APIs at depth 10 and depth 100 is here: Best SERP APIs 2026. The Bing retirement is covered here: Bing Search API key in 2026.

How to benchmark a SERP API: six axes and a cost model that survives n=100

Ricardo Batista — Fri, 21 Aug 2026 06:04:00 +0000

Key takeaways

Score six axes separately and never average them. A provider can be fast and shallow, or excellent on head terms and blind on local packs.
Compute cost at your own depth. DataForSEO is half cloro's price at n=10 and level with it at n=100 with AI Overview, and SerpApi's gap widens tenfold.
Score a missing block and an absent block differently, or your trend lines encode the parser's blind spots as real changes in Google.

Most SERP API comparisons publish one number: a price per 1,000 calls, or a median latency off a handful of queries. Both are true and both describe the easy case.

A SERP API is a parser sitting on scraping infrastructure. It can be fast and shallow. It can be cheap at ten results and ruinous at a hundred. It can be excellent on commercial head terms and blind on local packs. Average those into one score and you have hidden every failure mode that will actually bite you.

Run it in this order:

Build the query set. Fifty queries, evenly split across the four classes. Freeze the list, because you will re-run it against every provider and again in three months.
Fix the request shape. Same country, language, depth and AIO flag everywhere. Any provider-specific parameter is a variable you can no longer attribute.
Capture ground truth by hand for ten of them, so a parser miss is distinguishable from an absent block.
Score the six axes as six columns. Do not average.
Compute cost at your depth using each vendor's own unit, at both n=10 and the depth you actually need.
Measure latency under concurrent load, reporting p50 and p95. A sequential median tells you nothing about queue behaviour.
Ask the two questions the harness cannot answer: how schema changes are versioned, and where the data comes from.

Six axes, scored separately

Never collapse them into a single number. What you weight depends on your workload: a rank tracker running nightly batches cares about cost at depth, an agent doing grounding cares about latency, a GEO platform cares almost entirely about AI Overview field fidelity.

Axis	What you are actually testing
AI Overview fidelity	Not "does it return AIO" but in what shape. Cited sources with URLs and positions, a text-only summary, or raw HTML you parse yourself. Only the first survives a Google DOM change without work on your side
Multi-engine coverage	Whether each surface is a real endpoint or a wrapper around the same Google call. Coverage claims are cheap
Envelope completeness	Ads, People Also Ask, related searches, knowledge panel, shopping. Ten of the twelve we tested return the full envelope, so this rarely separates the field. It does catch the ones quietly returning organic only
Geolocation precision	Country, city, or UULE-precise. This is where a local SEO tracker gets burned late, because a national average and a metro result diverge on exactly the queries a client checks by hand
True cost per 1,000	At your depth and feature set, not the advertised base rate. Its own section below
Latency p50 and p95	The p95 is the one that matters. Dispatch faster than the queue clears and the slowest 5% balloon past the median. A 2-second median hides a 15-second tail

Fifty queries, four classes

Same request shape through every provider. Commercial head terms, which stress the envelope. AI-Overview-eligible informational queries, which separate real parsing from a text summary. Local and maps-heavy, which stress geolocation. Long tail, where thin SERPs show up as ambiguous nulls rather than errors.

The classes matter more than the count. A provider that looks identical to its competitors on head terms can be missing a third of AI Overviews on informational ones, and an aggregate success rate will never show you that.

One thing worth stealing: score a missing block and an absent block differently. When a provider returns nothing for an AI Overview, you cannot tell whether the SERP had no Overview or the parser missed it. Capture ground truth by hand for ten queries so you can tell the two apart. Skip that and your trend lines encode the parser's blind spots as real changes in Google.

The cost model, where most evaluations go wrong

Three things decide your bill, and the headline rate is none of them.

Depth. Google removed &num=100 on September 11, 2025, so everyone paginates at 10 results a page. Locomotive Agency put the infrastructure cost increase at roughly 10x at top-100 depth. If you need 100 results you are buying ten pages, and how a vendor bills those pages is now the dominant term.

AI Overview enrichment. Usually a surcharge, not an included field. DataForSEO gates it behind load_async_ai_overview at $0.0006 per keyword, roughly doubling the first-page price. Others include it. Credit-pool vendors charge extra credits.

Billing unit. Per-call, per-successful-result, and per-credit are not comparable until you do the arithmetic at your own depth. A credit pool turns a fixed cost into a variable one. Per-successful-result billing turns your failure rate into a price.

Provider	Unit	n=10	n=100 + AIO
cloro (Hobby, $0.40/1k credits)	3 credits at n=10, 23 at n=100 with AIO	$1.20/1k	$9.20/1k
DataForSEO (Standard queue)	per call, +$2/1k for AIO on page one	$0.60/1k	$9.20/1k
SerpApi (Big Data tier)	per search, one search = one 10-result page (pricing)	$9.17/1k	$91.70/1k

The ranking inverts between those two columns. DataForSEO is half cloro's price at n=10 and level with it at n=100 with AIO. SerpApi's gap widens by an order of magnitude, because each batch of ten bills as its own search. Run this before you shortlist. The provider that wins your evaluation at n=10 can be the most expensive thing you run in production, and the whole category is priced this way here.

Two things the harness cannot tell you

Whether the schema will hold. A benchmark captures one moment. It says nothing about whether the vendor renames a field next quarter and breaks your parsers. You cannot measure this from outside, so ask: how are schema changes versioned, and what is the deprecation window? No answer means a maintenance cost you have not priced.

Where the data comes from. Google sued SerpApi on December 19, 2025, alleging circumvention of anti-bot protections under DMCA 1201 (announcement, Search Engine Land has the detail). SerpApi moved to dismiss in February 2026, arguing it returns only what a signed-out user sees. Unresolved as of August 2026. No court has ruled that scraping public results is unlawful and the suit targets circumvention rather than scraping, but provenance is still a procurement question: official feed, scraped results pages, or the vendor's own index. Where the legal line actually sits is worth reading before you pick on this axis.

Both official routes are closing anyway. Bing's search APIs retired in August 2025, and Google's Custom Search JSON API is closed to new customers with a January 1, 2027 shutdown per Google's own docs. What is actually being retired and what is not is worth checking, because most write-ups get it wrong.

If you want the filled-in version, 12 providers scored on all six axes with the cost table at four depths, it is in Best SERP APIs 2026. Check vendor pricing pages yourself before committing. The category has repriced twice since num=100 and will again.

LLM visibility tracking tools: 15 compared, and when to build instead

Ricardo Batista — Wed, 19 Aug 2026 06:02:00 +0000

Key takeaways

LLM observability and LLM visibility are unrelated products sold under overlapping search terms. This is about the second one.
Citation overlap between engines on the same prompt runs under 30%, so a pooled six-engine average can hide being invisible on the engine your category actually uses.
Build-versus-buy flips around 50,000 queries a month or 10 tracked clients. Below 5,000 a dashboard wins on time-to-value alone.

First, a disambiguation that wastes a lot of people's time. LLM observability is telemetry for models your application calls: tokens, latency, traces, cost. Datadog, LangSmith, Helicone. LLM visibility is what models say about your brand: whether ChatGPT names you, which sources it cites, how you move against competitors. Same search terms, unrelated products. This is about the second one.

Search Console will not show you any of it, because the surface that decides whether ChatGPT names your brand is the answer generated before anyone clicks.

What do LLM visibility tracking tools actually measure?

Mention rate is the share of sampled answers naming your brand at all. Everything else is a cut of this number. A platform that shows a composite "visibility score" without the mention count underneath is asking you to trust a black box, and that score is not comparable to anyone else's.

Share of voice is your mention rate against the competitors named on the same prompts. It only means something if you control the competitor list, so check whether you define it or the platform infers it from your category.

Citation rate and position is whether the answer links your domain and where you sit in the source list. Mention and citation move independently. Your brand can be named in an answer that cites someone else entirely, and a tool that collapses the two hides the more actionable half.

Sentiment and accuracy is whether the answer describes your product correctly. Newest of the four, least standardised across vendors, so it deserves the most scrutiny in a demo.

Per-engine, not pooled. In our own cross-engine monitoring, citation overlap between engines on the same prompt runs under 30%. A single average across six engines can hide being invisible on the one that matters to your category.

Two numbers for scale before the tools. Ahrefs measured position-1 organic CTR falling roughly 58% on queries that trigger an AI Overview, and per a Globe Newswire industry report only 14% of marketers track AI citations while 89% of brands already appear in AI answers.

The 15

Tool	From	Best for	Notable
Gauge	$599/mo	B2B SaaS doing GEO	Prompt intelligence, Action Center recommendations
Profound	$5,000/mo	Enterprise compliance	10+ engines, sentiment, ISO-certified
Peec AI	$2,000/mo	Competitor depth	Real-time citations, clickstream correlation
Brandlight	$2,000/mo	Accuracy alerting	Brand accuracy monitoring, health scoring
DemandSphere	Custom	Layering AI onto an existing search programme	AI citations on the same footing as SERP share of voice
Nightwatch	Contact	Agencies already rank tracking	Citation Intelligence, unlimited seats, white-label on all plans
SE Ranking	$119/mo	SMBs on a budget	AIO and AI Mode trackers, competitor citation gap
Semrush AI Toolkit	$99/mo add-on	Existing Semrush users	25 prompts, up to 9 competitors
AthenaHQ	$199/mo	Growing SaaS and agencies	Clean dashboard, share of voice, weekly trends
OtterlyAI	$29/mo	Monitoring plus optimisation	ChatGPT Ads and Shopping tracking, GEO audit
Scrunch	Custom	AI-native reporting	Answer-first reporting, competitor benchmarking
Evertune	Custom	Prompt-level analytics	Share of voice broken down per prompt
aiclicks.io	$39/mo promo	Tracking plus content in one place	Built-in writer, GSC integration, prompt database
Authoritas Visibility Explorer	$99/mo	Multi-client agencies	Daily difference reports, 30+ markets
Ahrefs Brand Radar	$129/mo	Existing Ahrefs users	243M+ monthly prompts, joined to Ahrefs SEO data

Metric sets cluster by product focus rather than price. Evertune and AthenaHQ are built around share of voice. Peec AI and Nightwatch lead on citation-level detail. Profound and Brandlight are the two treating sentiment and brand accuracy as first-class rather than as add-ons. DemandSphere is the one that puts AI citations on the same foundation as the search-visibility number you already report to the business.

The gap that shows up in testing is engine coverage versus claimed engine coverage. Most cover ChatGPT, Perplexity, Gemini and AI Overview. Copilot and AI Mode are spottier, and depth varies even where the logo appears on the pricing page. Ask whether each engine is queried through an official API or browser automation, because API-only coverage structurally misses AI Overview and Perplexity's web-search surface, which are the citation-heavy ones.

Worth knowing before you read anyone's citation-gap report: per Otterly's analysis of over a million citations, 73% of sites carry technical barriers (robots.txt blocks, CDN rules, JS-only content) that stop AI crawlers reaching the page at all. Some "citation gap" findings are crawler-access findings wearing a different hat.

How we tested

One B2B SaaS brand and one consumer-product brand, 25 commercial-investigation queries each, four weeks, spanning comparison, definitional, recommendation and how-to intents. We captured ground truth by hand across ChatGPT with web search, Perplexity, the Gemini app, Copilot, AI Overviews and AI Mode, then compared what each tool reported for the same queries on the same days. Scored on engine coverage, citation fidelity, update frequency, reporting depth, pricing fairness, and methodology transparency.

This roundup is published by cloro, a data provider rather than a dashboard, so none of the 15 tools above is a competitor. cloro appears once below, in the infrastructure section, and is not scored against them.

A roundup is editorial judgement on one date. The AI visibility platforms leaderboard is the standing measurement beside it, re-scored weekly and free to cite.

The API question, which most comparisons get wrong

Vendors use "API" for three different things:

Scheduled export. CSV drops, Sheets, sometimes a Looker Studio connector. Fine for reporting, useless for a product feature. This is the most common thing sold as integration.
Read API over their stored data. Query the vendor's results for your tracked prompts. Good for pulling their numbers into your BI stack, still bounded by the prompts, engines and cadence your plan allows.
Raw data API. Submit any prompt to any surface, get the parsed response. Different product category, not a dashboard feature.

Two things a comparison table cannot hold. API availability moves between pricing tiers often enough that any published list is stale within a quarter. And an API on a dashboard product is normally rate-limited against the prompt allowance you already bought, so "has an API" does not mean "can back a product feature."

When building your own starts to make sense

Three bands, from watching teams do this:

Under ~5,000 queries a month. Buy a dashboard. Anything you build will take longer to stand up than the value it returns.
Over ~50,000 queries a month, or 10+ tracked clients. Per-seat dashboard pricing crosses per-call API pricing, and an in-house build pays back in two to four months.
In between. It depends entirely on whether you already have BI and ops infrastructure to host the output. If the answer is a CSV landing in a warehouse, you are paying for a UI layer and throwing it away.

The builds themselves are small. Three patterns cover most of it: nightly scheduler into Postgres into Metabase with Slack alerts; per-client query lists with per-tenant keys and white-label reports; or a daily batch into BigQuery joined against GA4, Search Console and CRM data. Each is one to two weeks of engineering on top of a managed data layer.

What is not small is the parsing layer. Scraping it yourself runs $500-2,000 a month in proxies and browser infrastructure plus two to three months of build, and the hard part is not the scraping. It is that a citation parser working cleanly in January can miss 30% of them by April after a UI change, silently. Official model APIs avoid that and cannot help either, because they do not expose citation data the way a live search surface does.

cloro is one such layer: a single endpoint across ChatGPT, Perplexity, Gemini, Copilot, AI Overview, AI Mode, Google Search and Google News, returning parsed source URLs with position and label, plus query fan-out terms, as JSON rather than screenshots. Credits are deducted only on successful extraction. Hobby is $100/month for 250,000 credits ($0.40 per 1,000), scaling to $0.34 at the top tier, with 500 free credits to test and docs rather than an onboarding call. Async is the right call for nightly batches; sync costs two extra credits and returns inline.

It is not a dashboard, and if what you want is a dashboard then one of the 15 above is the better purchase.

What to pressure-test in a demo

Engine coverage method. Official API or browser automation, per engine. Vagueness here is the answer.
Citation format. Structured queryable URLs, or screenshots. This decides whether you can compute anything yourself.
Query control. Can you import prompts from Search Console and define your own, or does it generate them from your domain?
Cost at your real cadence. Affordable at 50 queries a week is a different product at 500. Most teams find they need weekly rather than daily, which changes the maths a lot.

If you want the ChatGPT surface specifically rather than all six, monitoring ChatGPT mentions is covered end to end separately.

Best ChatGPT scraper in 2026: what breaks, and what 8 tools cost

Ricardo Batista — Mon, 17 Aug 2026 05:57:00 +0000

The best ChatGPT scraper for brand monitoring is a managed API that returns parsed citations, and cloro is the only one of the eight below that ships that parsing rather than handing back HTML. For general-purpose scraping at volume, Bright Data's Scraping Browser is the stronger pick. For zero budget with engineering capacity, it is Playwright and a proxy bill. Which of those three you are depends on the table further down, not on the verdict.

Key takeaways

The official OpenAI API cannot answer brand-monitoring questions, because citations, shopping cards and the search-versus-memory decision only exist in the web UI.
At 1,000 queries a day, DIY Playwright runs $980-2,140 a month once proxies, CAPTCHA and 8-15 engineer hours are counted, against $100-300 for a managed API.
Selector maintenance is the hidden line item. OpenAI's dynamic CSS class names change roughly weekly, and the failure is silent rather than loud.

There are two ChatGPTs. The API developers build on, and the web interface 900 million weekly users actually see. Only the second one browses the web, cites sources, renders shopping cards, and decides per-query whether to search or answer from memory.

If you want to know whether ChatGPT recommends your product or names your competitor instead, the API structurally cannot tell you. You have to read the rendered UI, which means scraping one of the better-defended properties on the public web.

What makes chatgpt.com hard to scrape?

1. Cloudflare fingerprints your TLS handshake

Per Scrapfly's analysis, the 2026 stack starts with JA4 TLS fingerprinting. Default requests or urllib fail on the handshake, before any of your logic runs. Then behavioral profiling of timing and movement, JavaScript challenges a static client cannot execute, and Turnstile CAPTCHAs on anything that smells automated.

Datacenter IPs are the fastest way to get nowhere. proxies.sx testing has real mobile IPs surviving 50-100+ queries while datacenter ranges get blocked in the first few requests.

2. The response streams

ChatGPT does not return HTML. It streams tokens over Server-Sent Events, so you keep the connection alive, parse event: message frames as they land, assemble the answer, and detect the end signal. Static HTTP clients see nothing. Headless browsers handle it transparently and charge you in compute and latency.

3. Class names change between deploys

OpenAI's React build emits dynamic classes like ._a4b3f. Anything selecting on them breaks roughly weekly. Semantic selectors (ARIA labels, roles, text content) survive longer, and you still want fallback chains. This is the line item every DIY estimate underprices.

4. Login, 2FA, and session persistence

Most useful workflows need an account, most accounts have 2FA, and the login flow sits behind the same Cloudflare checks. Three realistic paths: export cookies after a manual login and refresh on a schedule, pay someone to hold the session, or run headful Playwright with stealth plugins and accept the breakage.

5. Proxy economics

Databay's 2026 breakdown puts residential proxies at $3-15/GB, with the mobile IPs ChatGPT tolerates at the top of that range. At 1,000 queries a day you burn roughly 3-8 GB a month, plus CAPTCHA credits at $1-3 per 1,000 challenges. That line item alone often exceeds a managed subscription. There is a fuller treatment of proxy selection for SERP work elsewhere.

The eight options

Tool	What it solves	What you still write	Starting price
cloro	Access, SSE assembly, citation parsing, fan-out, auth	Nothing	$100/mo, 500 free credits
Apify	Access and compute, via marketplace actors	Cookie refresh; parsing varies by actor	$49/mo + compute
Bright Data	Access at any scale, 72M+ IPs, CAPTCHA	SSE assembly and all parsing	~$1.50/1k at volume
Browserbase	Persistent sessions, stealth, agent ergonomics	SSE assembly, parsing, selectors	$50/mo, free tier
Browserless	Headless Chrome, self-hostable	SSE assembly, parsing, selectors, some evasion	$50/mo, free Docker image
ScrapingBee	Cloudflare bypass with a clean API	SSE assembly, parsing, selectors	$49/mo, 100k credits
ZenRows	Cloudflare bypass, cheaper tier	SSE assembly, parsing, selectors	$69/mo, 250k credits
Playwright	Nothing. It is the toolkit	Everything	Free, plus your time

This roundup is published by cloro, which is one of the eight tools in it. The prices and capabilities attributed to the other seven come from their own documentation, linked where they are not obvious.

The split that matters is the third column. Every tool except the managed APIs leaves you owning SSE assembly and selector maintenance, which is the part that fails weekly rather than the part that fails loudly.

A few notes the table cannot hold. Apify's actors are community-maintained more often than not, so reliability tracks whoever still cares about that actor. Bright Data's Scraping Browser is genuinely hard for Cloudflare to block and overkill below serious volume. Browserbase is built for agents and its session persistence solves the auth problem cleanly, which is a real advantage here. ScrapingBee's team published their own ChatGPT scraper roundup, which is a decent signal they understand the target.

Out of scope on purpose: the official OpenAI API (different surface), wrappers like LiteLLM and OpenRouter (they wrap the API, not the UI), Cloudflare-bypass libraries like Camoufox and FlareSolverr (they solve the WAF and leave you the rest), and answer APIs like Tavily and Exa (synthesized answers, not the rendered UI).

What it costs at 1,000 queries a day

Tool	Subscription	Proxies/credits	CAPTCHA	Eng hrs/mo	Total $/mo
cloro	$30-300	included	included	0	$30-300
Apify (official actor)	$49	$30-80	included	2	$280-410
ScrapingBee	$49-249	included	included	4	$449-849
ZenRows	$69-249	included	included	4	$469-849
Browserless	$50-100	$100-300	$30-90	6	$580-890
Bright Data	pay-as-you-go	$200-500	included	4	$600-900
Browserbase	$50-200	$50-150	$30-90	6	$730-1,040
Playwright (DIY)	$0	$150-450	$30-90	8-15	$980-2,140

Assumptions, because a cost table without them is decoration: engineer time at $100/hour fully loaded, CAPTCHA solver at $2 per 1,000 challenges firing on 5% of requests, mobile residential proxies at $10/GB, requests averaging 4-8 KB.

Two things fall out. The DIY gap widens with volume, because proxy bandwidth scales linearly while per-call rates flatten or improve with tiers. And the browser-infrastructure tier sits in an awkward middle: cheaper than Bright Data, more expensive than a managed API once you price the parsing work.

Change the engineer-hours assumption and the ranking moves. If your team already runs Playwright infrastructure and selector patches cost you an hour a month rather than ten, DIY wins outright. That is a real case, not a strawman.

Picking

Already on Apify, ChatGPT is one target of several. Use the official actor rather than a community one.
Volume is the binding constraint. Bright Data, and budget for writing the parsing layer.
Building an agent that needs a live session. Browserbase.
You want Cloudflare handled and nothing else. ScrapingBee, or ZenRows if the budget is tight.
Zero budget, real engineering capacity. Playwright with stealth plugins and mobile proxies. Plan for 8-15 hours a month.
You want parsed citations and fan-out with no maintenance. cloro is built for that case and is a monitoring tool rather than a general scraper. If you need arbitrary browser automation, buy browser infrastructure instead.

One legal note, since it comes up in every thread: scraping your own session or publicly visible content is generally fine, while bypassing authentication or pulling other people's data is not. Where the line sits is covered properly elsewhere.

If you are doing this for brand monitoring rather than data collection, the tools built for that job are a different category and probably the better starting point.

How to scrape Google AI Mode: citation pills + place cards

Ricardo Batista — Fri, 14 Aug 2026 05:49:00 +0000

Key takeaways

AI Mode and AI Overview cite the same URLs only 13.7% of the time, so scraping one tells you nothing reliable about the other.
Source URLs live in HTML comments keyed by UUID, not in anchor tags, so a scraper reading <a href> returns an empty citation list rather than an error.
Strip #:~:text scroll-to-text fragments before counting, or one article reads as several distinct URLs and inflates your citation numbers.

Google AI Mode does not put its sources in anchor tags. It renders each one as a button, and hides the actual URL in an HTML comment keyed by UUID. A scraper that reads <a href> finds nothing. That one design choice is most of the work in this post.

The rest is that nothing is in the initial HTML at all. The shell loads, then the answer streams in over a separate call to /async/folwr. Parse on page load and you get an empty container.

Build the URL with udm=50 and aep=11 to force the interface.
Register a network interceptor on https://www.google.com/async/folwr before navigating.
Navigate, and treat a non-200 as a CAPTCHA rather than a dead page.
Poll until the intercepted response lands, with a hard timeout.
Read the answer text off data-session-thread-id, not off a snapshot.
Collect the citation buttons, then rejoin them to their HTML-comment metadata by UUID.
Rewrite the pills into anchors before converting anything to markdown.

Steps 6 and 7 are where the work is. The rest is plumbing.

Why bother with this surface specifically

AI Mode is not AI Overview with a different skin. Per ALM Corp's analysis of 1.3 million AI Mode citations, the two cite the same URLs only 13.7% of the time on semantically similar queries, and google.com self-cites in 17.42% of AI Mode answers, more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. Scraping one tells you nothing reliable about the other.

It is also where the comparison queries go. Seer Interactive's 2026 trigger data puts AI surfaces on 95.4% of comparison queries and 85.9% of question-format ones, and AI Mode passed 75 million daily active users by late 2025 per Alphabet's Q3 2025 earnings call. The query fan-out behind it, where Google splits your question into subtopics and runs many searches at once, is why one answer stitches together so many cited pages.

Surface	Trigger	Citation style	Difficulty
AI Mode	`udm=50`	Pills, metadata in HTML comments	High: async streaming, drifting selectors
AI Overview	Organic query, varies by class	Inline source list above the SERP	Medium: single fetch
Gemini	gemini.google.com	Grounded sources panel	High: separate auth domain
ChatGPT Search	chatgpt.com with search on	Numbered citations	Highest: SSE plus Cloudflare

Force the interface

udm=50 is the switch. aep=11 goes with it.

search_url = build_url_with_params(
    "https://www.google.com/search",
    {
        "udm": 50,        # AI Mode
        "aep": 11,
        "q": prompt,
        "hl": google_params["hl"],
        "gl": google_params["gl"],
    },
)

Wait for the stream, then parse

We drive Playwright and capture the async call, then poll until it lands. Sixty seconds, then fail loudly. A hard timeout beats a silent partial parse, because a half-streamed answer looks like a real answer with fewer citations.

page_interceptor = PlaywrightInterceptor(do_not_block_resources=True)
page_interceptor.add_capture_urls(["https://www.google.com/async/folwr"])
await page_interceptor.setup_page_interceptor(page)

for _ in range(120):                      # 500ms intervals, 60s cap
    if len(page_interceptor.captured_responses):
        break
    await sleep(500)
else:
    raise Exception("Never received AI Mode response after 60 seconds")

The answer text hangs off data-session-thread-id. Target that and read its parent, because AI Mode rewrites its own container several times while streaming and a locator survives that where a snapshot does not.

thread_element = page.locator("[data-session-thread-id]")
text = await thread_element.locator("..").text_content() or ""

The citation pills

Here is the shape you are working against. A UUID ties the visible button to a comment block carrying the real URLs.

<!--Sv6Kpe[["uuid-12345",["label","description"],["https://example.com","source2"]]]-->
<button data-icl-uuid="uuid-12345" data-amic="true">[1]</button>

So: find the buttons, read their UUIDs, regex the comments back out of the page HTML, and filter Google's own domains out of the URL list. One pill can carry several sources.

async def extract_aimode_citation_pills(page: Page) -> Dict[str, List[LinkData]]:
    citation_pills: Dict[str, List[LinkData]] = {}
    pill_locators = page.locator('button[data-icl-uuid][data-amic="true"]')
    page_html = html.unescape(await page.content())

    for i in range(await pill_locators.count()):
        pill_button = pill_locators.nth(i)
        if not await pill_button.is_visible():
            continue
        uuid = await pill_button.get_attribute("data-icl-uuid")
        if not uuid:
            continue

        pattern = rf'<!--Sv6Kpe\[\["{re.escape(uuid)}".*?]]-->'
        current_pill: List[LinkData] = []

        for content in re.findall(pattern, page_html, re.DOTALL):
            desc_match = re.search(
                rf'"{re.escape(uuid)}"\s*,\s*\[\s*"[^"]+"\s*,\s*"([^"]+)"', content
            )
            url = next(
                (u for u in re.findall(r'"(https://[^"]+)"', content)
                 if not any(skip in u for skip in
                            ["google.com", "gstatic.com", "encrypted-tbn"])),
                None,
            )
            if url:
                url = url.split("#:~:text")[0]
                url = url.replace("\\u003d", "=").replace("\\u0026", "&")
                current_pill.append(LinkData(
                    position=len(current_pill) + 1,
                    label=f"Source {len(current_pill) + 1}",
                    url=url,
                    description=desc_match.group(1) if desc_match else None,
                ))

        if current_pill:
            citation_pills[uuid] = current_pill

    return citation_pills

The #:~:text strip matters more than it looks. Google appends scroll-to-text fragments, so the same article comes back as several distinct URLs and your citation counts inflate quietly.

The sources panel needs its own selector per layout. Web-results pages expose it one way, the dialog another. Playwright's actionability checks handle the waiting, so log and continue on timeout rather than killing the run.

sources_selector = (
    '[data-container-id="rhs-col"] [role="dialog"] a'
    if not is_web_results_page
    else "a.ZbQNgf"
)

Converting to markdown without losing the sources

Hand the raw HTML to html2text and every pill becomes a dead button. Rewrite the pills into real anchors first, using the UUID map you already built, then convert.

soup = BeautifulSoup(html_content, "html.parser")
for button in soup.find_all("button", attrs={"data-icl-uuid": True, "data-amic": "true"}):
    uuid = button.get("data-icl-uuid")
    for link_data in reversed(citation_pills.get(uuid, [])):
        anchor = soup.new_tag("a", href=link_data.get("url"))
        anchor.string = link_data.get("label")
        button.insert_after(anchor)
    button.decompose()

h = html2text.HTML2Text()
h.ignore_links = False
h.body_width = 0
markdown = h.handle(str(soup)).strip()

What breaks at scale

A non-200 is usually a CAPTCHA, not a dead page, so solve before you give up. One blocked request should not tank a batch.

response = await page.goto(search_url, timeout=20_000)
if not is_http_success(response.status):
    if not await solve_captcha(page, page_interceptor):
        raise Exception(f"HTTP error: {response.status} (probably captcha)")

The real running cost is selector drift. AI Mode's DOM changes without notice, and a scraper that worked last month returns empty rather than throwing. Alert on success rate, not on exceptions. A sudden drop is a layout change, and you will find it weeks late if you are only watching for errors. You will also want a proxy pool and fingerprint rotation before this runs at any volume.

Or skip it

cloro maintains this pipeline as a managed endpoint, which is the disclosure to weigh this section against. One POST, parsed text and sources back, P50 under 8 seconds.

response = requests.post(
    "https://api.cloro.dev/v1/monitor/aimode",
    headers={"Authorization": "Bearer sk_live_your_api_key"},
    json={"prompt": "What do you know about Tesla's latest updates?",
          "country": "US",
          "include": {"markdown": True}},
)
result = response.json()["result"]
print(f"{len(result['sources'])} citations, fan-out: {result['searchQueries']}")

Built in-house, this runs $5,000 to $10,000 a month once you count engineering time, browser instances and proxies. Below a few thousand queries a month that maths does not favour us, and you should build it. Above that it does, and cloro's AI Mode endpoint absorbs the selector drift.

If you want the dashboard rather than the pipeline, the tools that track this surface are compared separately.

SERP API reliability at scale: uptime, latency, concurrency

Ricardo Batista — Wed, 12 Aug 2026 05:46:10 +0000

Key takeaways

Two of the four headline reliability numbers in this category are not uptime at all: DataForSEO publishes 99.95% with no credit terms, and Bright Data's 99.99% is a success rate from a benchmark with a disclosed vested interest.
At 10,000 requests a day the P95 decides your architecture, not the median. Bright Data measures 2.61s P50 against 4.92s P95, and a 2-second median can hide a 15-second tail once queuing starts.
Google's &num=100 removal plus DataForSEO's depth-based pricing took the Live tier from $2.00 to $17.50 per 1,000 at 100 results with AI Overview, a ninefold jump from a change you did not make.

Below a few hundred requests a day, every SERP API looks fine. The failure modes only show up somewhere past 10,000: queue latency, rate-limit rejections, concurrency ceilings. By then you have built on top of one.

Three things decide it, and only one of them appears on a pricing page.

This comparison is published by cloro, which is one of the four providers in it. Every figure attributed to another vendor is sourced below; the cloro rows are self-reported and should be read as such.

What does a SERP API uptime SLA actually commit to?

Provider	Uptime SLA	Median response	Rate limits
cloro	99.9% with service credits	Sub-second (async queue)	Parallel lanes, no hard cap
SerpApi	None published	1.96s	Hard monthly cap, request queuing
DataForSEO	99.95%, no credit terms stated	Live fastest; ~5-min Standard queue	High default RPS, async batching
Bright Data	None published	2.61s standard, 0.89s premium	Configurable 5-20+, no documented ceiling

Two of those four numbers are not what they look like.

DataForSEO's 99.95% is higher than ours and the documentation states no credit or penalty terms. That makes it a target rather than a remedy. A published SLA without credits tells you what a provider aims for, not what you recover when they miss.

Bright Data's widely quoted 99.99% is a success rate, not uptime, and it comes from a benchmark whose author disclosed a vested interest. Success rate measures request completion. Uptime measures whether the endpoint answers at all. They are different failures and only one of them wakes you up.

So when you compare this column, check three things: is it uptime or success rate, is it contractual or measured, and does anything happen when they miss it.

Latency: the median is the wrong number

Almost every published latency figure is measured sequentially, which is the easy case. What decides production behaviour is what happens when you dispatch faster than the provider's queue clears.

The best public numbers come from that same 30-day benchmark of nine providers: Bright Data's standard P50 at 2.61s against a P95 of 4.92s, in a field ranging from Serper at 2.23s to DataForSEO at 4.54s. Premium routing takes the P50 to 0.89s, and that is the vendor-adjacent figure, so discount it.

At 10,000 daily requests the P95 is the number that matters. A 2-second median hides a 15-second tail once queuing kicks in. An overnight rank tracker can absorb that tail. Brand monitoring cannot, because a five-minute delay on an incident query defeats the point.

DataForSEO's Standard queue is the clearest case: one tester measured async callbacks averaging 12 seconds for 100 keywords, stretching toward five minutes at production volume. That is fine for scheduled work and useless for anything real-time.

Concurrency is an architecture choice, not a setting

A 10k batch survives or fails on this, and each provider picked a different model.

SerpApi queues. Hit the rate limit and requests wait behind one another rather than running in parallel, so latency climbs with volume. Exhaust the monthly cap and requests fail outright until the next billing cycle. If your volume is variable you overprovision for the worst day and pay for headroom you rarely use.

DataForSEO batches asynchronously. Submit, wait, retrieve. It absorbs thousands of tasks without per-request rate errors, and you pay for it in freshness. No requests-per-second ceiling appears in the docs.

Bright Data hands you the dial. Its own rank-tracking demo defaults to 5 concurrent and recommends 5 to 10, documenting a 1 to 20+ range with no published ceiling. Past the recommended range you are tuning against undocumented limits and watching for rate errors yourself.

cloro runs parallel lanes behind an async job queue. You POST a batch, get a job ID back in milliseconds, and results arrive by webhook as workers finish. Request 10,000 does not wait behind the other 9,999, so adding volume widens the fan-out instead of lengthening the wait. Concurrency scales by tier rather than capping at a fixed lane count.

The honest summary of that difference: SerpApi and DataForSEO both trade real-time responsiveness for predictable batch cost, in opposite directions. If your workload is nightly and cost-sensitive, DataForSEO's queue is the right trade and cloro is overkill.

AI Overview coverage, where the gap is measurable

For GEO work the question is whether you get AI Overview data as consistent structured fields, not whether the provider returns it at all.

SerpApi detected AI Overviews in 68% of qualifying queries in a published 25-query benchmark. Roughly a third of pages that should show an Overview returned nothing detectable, and that is the ugly kind of gap: a missing Overview reads identically to an absent one, so your trend lines encode the parser's blind spots as changes in Google.

DataForSEO makes it opt-in with load_async_ai_overview at $0.0006 per keyword, which doubles the Live tier to about $4.00 per 1,000 at ten results, and $17.50 per 1,000 at 100 results with AIO. Structured, but you submit and poll rather than reading it inline.

cloro returns Overview text, cited sources and position in the same payload as organic results. The reason that matters is not convenience: Google changed the Overview DOM during its January 2025 JS-rendering update, and anyone parsing raw HTML rebuilt their extraction. Provider-managed fields absorb that server-side. It is the strongest argument for buying rather than building, covered at length in this build-versus-buy breakdown.

Cost at volume, which moved twice in 2025

Model this at your depth, not the advertised rate.

SerpApi is subscription with hard caps and no pay-as-you-go: roughly $2,500 at 100k monthly requests, $25,000 at 1M. DataForSEO is the cheapest at raw volume, about $0.60 per 1,000 on Standard, so 300k requests a month lands near $180. Bright Data prices per call near $0.005 with a minimum commitment cited around $499/month.

Then two changes reset the arithmetic. DataForSEO moved to depth-based pricing on September 19, 2025, and Google deprecated &num=100 on September 11, 2025, forcing pagination at 10 results a page. If you need 100 results plus AI Overview, that Live tier goes from $2.00 to $17.50 per 1,000. A ninefold jump, from a parameter change you did not make. A fuller pricing breakdown runs the same numbers at four depths.

Pick by workload

Nightly batches, cost-sensitive: DataForSEO. The queue latency is real and irrelevant to you, and $0.60 per 1,000 is hard to argue with. Watch the depth pricing.
City-level geo-targeting across many markets: Bright Data. 195 countries with city-level targeting is the widest coverage published, and you manage concurrency yourself.
Low volume, many engines, want it managed: SerpApi. Live inside a tier and the caps never bite.
Real-time monitoring past 10k/day: cloro is built for this shape, and its constraint is that it is API-first. If you want a dashboard rather than endpoints, buy a dashboard.

Whatever you pick, run your own 10k-request test against it under concurrent load before you commit. Every number above, cloro's included, was measured by someone with an interest in the result.

🚀 Launching FidForward! 🚀

Ricardo Batista — Mon, 08 Jul 2024 16:54:15 +0000

Today, Bernardo and I are excited to launch FidForward in private beta!

We're looking for companies with 10-50 employees willing to pilot the next generation of performance management.

With two pilot programs already running successfully (achieving a 30-50% improvement in eNPS), we’re eager to expand FidForward to more companies to validate these results further.

If you’re a founder seeking to implement performance management or a larger company aiming to elevate your current system, please reach out to me!

🔗 Learn more about FidForward at FidForward.com

We were tired of bureaucracy, so we built an opensource repo for the best guides

Ricardo Batista — Sun, 23 Jun 2024 21:09:11 +0000

Today we are launching Tramitit, a shared database on getting all those poor-UX local procedures done in a much simpler way!

How it works: The community provides detailed walkthroughs on every possible bureaucratic process you might go through.
For the community by the community: The content on this website is curated, verified, and rated by the community.
Welcoming providers that can help our users: Although our guides are as simple as possible, users may still prefer to delegate tasks to a curated provider.

We are in a bit a chicken & egg problem: for the traffic to pick up and have contributors, we need to start by adding some content - some of which we have already gone through and fixed it, other not yet there. We will get there over time, and work with providers that can also bring the quality up.

You can find our GitHub repo here, where you can contribute as well!
Our website is also open-source, so feel free to add anything.

Getting started with Open SaaS

Ricardo Batista — Sun, 21 Apr 2024 07:34:17 +0000

When building Article Generation and Meeting Reminders, I decided to get started as fast as possible. So I looked for a free boilerplate and stumbled upon Open SaaS, which used YC-backed Wasp. It is a full-stack React + NodeJS + Prisma that takes 8 hours to get started with.

Their documentation is brilliant, and great support over Discord. I decided to list here the full walkthrough on how to set up the development environment so that it is a simple copy-paste for you.

For frontend deployment, I used Netlify (for the generous free package) and the recommended fly.io for server + database (also cheap package).

Run development commands

Start development database:

cd app && wasp start db

View development database in Prisma:

wasp db studio

Start development app:

cd app && wasp start

Start development blog:

cd blog && npm install && npm run dev

Run frontend deployment commands

Locate the folder:

cd .wasp/build/web-app

Install dependencies:

npm install && REACT_APP_API_URL={your-server-URL} npm run build

Deploy preview:

netlify deploy

Deploy to production:

netlify deploy --prod

Every time there is a Wasp app rebuild (with wasp build), it will remove your .wasp/build/ directory. Until the GitHub issue is fixed, you need to update netlify.toml’s publish line to:

  publish = ".wasp/build/web-app/build"

Run backend & db deployment commands

Locate the folder:

cd .wasp/build/

Deploy preview:

flyctl deploy --remote-only --config ../../src/shared/fly-server.toml

Every time there is a Wasp app rebuild (with wasp build), it will remove your .wasp/build/ directory. That’s why there is a copy of the file in app/src/shared/.

Run blog deployment commands

Locate the folder:

cd blog

Build the app:

npm run build

Deploy preview:

netlify deploy

Deploy to production:

netlify deploy --prod

Good luck with building, hopefully it will be faster now!