DEV Community: cloro

Which web search API has an MCP server I can plug into Claude or Cursor?

Ricardo Batista — Sat, 14 Nov 2026 17:13:06 +0000

Seven web search APIs ship an official MCP server you can add to Claude, Claude Code or Cursor: Brave, Exa, Tavily, Firecrawl, Perplexity, Parallel and cloro. Pick by what the agent needs back. Exa, Tavily and Firecrawl return page text for a model to read. Brave returns results from its own index. Perplexity can return a written answer. cloro returns Google's own results page as JSON, with the AI Overview and People Also Ask. Parallel's server is free with no key.

Hosted or local: six of the seven run hosted, so you add one URL. Brave's runs on your machine through npx or Docker.
Key or no key: Parallel needs none. Exa and Firecrawl start without one. Brave and Perplexity need a key. The others take an OAuth sign-in or an API key.
Accuracy: in a benchmark of six web search APIs on multi-hop questions, the two Google SERP APIs answered most questions from one search (80.2% and 77.4%). Parallel answered 65.3%, Exa 64.0%, Brave 53.9% and Tavily 48.4%.

Which web search APIs ship an MCP server?

Server	Searches	Returns	Key needed	Where it runs
Brave Search MCP	Brave's own index	Results as JSON	Yes, `BRAVE_API_KEY`	Local, `npx` or Docker
Exa MCP	Exa's neural index	Results plus page text as markdown	No, to start	Hosted at `https://mcp.exa.ai/mcp`, or `npx`
Tavily MCP	Tavily's retrieval index	Results plus extracted page content	No, with the keyless header	Hosted at `https://mcp.tavily.com/mcp/`, or `npx`
Firecrawl MCP	Its own search, plus crawl and scrape	Markdown or structured extraction	No, capped per IP per day	Hosted at `https://mcp.firecrawl.dev/v2/mcp`, or self-hosted
Perplexity MCP	Perplexity's Search API and models	Ranked results, or an answer with citations	Yes	Hosted at `https://api.perplexity.ai/mcp`, or `npx`
Parallel Search MCP	Parallel's Search API	Results plus fetched pages	No	Hosted at `https://search.parallel.ai/mcp`
cloro	Google Search, plus Google AI Mode, ChatGPT, Gemini and Bing AI Mode	Google's results page as JSON: organic positions, AI Overview and its sources, People Also Ask, ads	OAuth in Claude, API key elsewhere	Hosted at `https://mcp.cloro.dev/mcp`

SerpApi, DataForSEO and Oxylabs also run official MCP servers over their Google APIs. Open-source servers such as web-search-mcp and duckduckgo-mcp-server scrape Bing, Brave or DuckDuckGo from your own machine with no key.

Google has no official MCP server that returns the Google.com results page. Its managed MCP servers search Workspace content and data you load into Agent Search, and its Agent Platform server can ground a Gemini answer in Google Search. None of them returns organic positions or the AI Overview for a public query.

Which one should you pick?

The servers differ most in what they hand back to the model, so pick by the job.

Your agent needs	Use
Clean page text to read and summarize, for RAG	Exa, Tavily or Firecrawl
An independent index that is not Google	Brave
A finished answer with citations	Perplexity
Search at no cost and with no key	Parallel, or an open-source server you run
What Google shows for a query in a country: rankings, the AI Overview, People Also Ask	A server on a Google SERP API
A correct answer from one search on a hard question	A Google SERP API, the leaders on multi-hop questions below

Three of these differences only show up once the agent runs.

Output shape. Exa, Tavily and Firecrawl put page text into the context, which is what a model needs to answer from, and it costs tokens on every call. Brave and the Google SERP servers return structured results, so the agent can answer "is this domain in the top 5?" by reading a field, not by parsing prose.

Index. Only a server on a Google SERP API sees Google's answer layer. When 500 queries ran through Exa and through Google's live results, about a third of Exa's top 10 URLs were also in Google's top 10. That is by design: Exa retrieves by meaning. It is the wrong source for a rank check and a good one for finding pages a keyword search misses.

Free tiers change. Brave has had no free plan since February 2026: every plan gets $5 of credit a month, enough for 1,000 searches, and sign-up needs a card. Exa gives $10 of credit a month, and Tavily and Firecrawl give 1,000 credits a month. None of the keyless servers publishes its limit, so expect throttling once an agent searches in a loop.

How do you add a web search MCP server to Claude Code?

A hosted server is one command. This adds Exa, which starts without a key. Replace the name and URL with the server you picked:

claude mcp add --transport http exa https://mcp.exa.ai/mcp

Then open a session and run /mcp. A server that uses OAuth asks you to sign in there. A server that needs a key takes it in a header or in the URL, as its docs describe.

A local server runs as a command. For Brave:

claude mcp add brave -e BRAVE_API_KEY=your_key -- npx -y @brave/brave-search-mcp-server

Each server you add puts its tool definitions into the context before you ask anything. Run /context to see what they cost, and keep the one or two you use.

How do you add one to Cursor?

Cursor reads MCP servers from its mcp.json. A hosted server is one entry with its URL:

{
  "mcpServers": {
    "exa": {
      "type": "http",
      "url": "https://mcp.exa.ai/mcp"
    }
  }
}

Restart Cursor and the tools appear. Some servers take the API key in the URL path, which works in clients that cannot send custom headers. Check each server's docs for the exact URL.

Which web search API is most accurate from one search?

Six web search APIs ran on the 824 multi-hop questions of the FRAMES set in November 2026. Each question needs facts from two or more pages. Each API got the question as its query, one search and 10 results. One model answered from those results alone, and a second model graded each answer against the dataset's answer key.

API	Multi-hop questions answered from one search
cloro	80.2%
SerpApi	77.4%
Parallel	65.3%
Exa	64.0%
Brave	53.9%
Tavily	48.4%

The lead of the top API over SerpApi holds in a paired test on the same questions (p = 0.022). The AI Overview does most of the work: with the same results and the AI Overview removed, accuracy on 100 SimpleQA Verified questions fell from 99% to 86%. On short factual questions the order changes: Parallel (97.8%), Exa (97.3%) and the top Google API (96.9%) are within the margin of error of each other. The full method, the per-question-type results and the raw data are in the web search API benchmark.

What does each search cost over MCP?

Server	Price per 1,000 searches	Free allowance
Brave	$5	$5 of credit a month
Perplexity Search API	$5	Not stated
Exa	$7	$10 of credit a month
Tavily	About $8 pay as you go ($0.008 a credit)	1,000 credits a month
Firecrawl	About $7.60 on Hobby, billed monthly	1,000 credits a month
Parallel	Free over MCP	No key needed
cloro	$2.00 on Hobby; $2.80 with the AI Overview	500 credits a month, about 100 searches

The Google SERP row assumes a synchronous MCP call: a Google search is 3 credits plus a 2-credit sync surcharge, so 5 credits, and Hobby is $100 a month for 250,000 credits. For thousands of scheduled searches, the async API skips the surcharge.

How this was checked

This comparison is published by cloro, which is one of the servers in it. The endpoints, tool names, free allowances and prices of the other servers come from each vendor's own docs, pricing page or GitHub repository, read on October 24, 2026. The Tavily price and the Brave package were checked again on November 14, 2026. The accuracy figures come from the benchmark linked above, which publishes its questions and scoring.

The full comparison, with the free open-source servers and a sample of what a Google results call returns, is in Best Web Search MCP Servers.

How to get the UI answers of ChatGPT through an API (if you want what users see, not the OpenAI API output)

Ricardo Batista — Thu, 08 Oct 2026 20:11:00 +0000

To get the UI answers of ChatGPT through an API, call a service that runs your prompt in the ChatGPT web interface and returns the rendered answer as JSON. The OpenAI API does not do this. It returns model output, without the web search, source citations, shopping cards and entity panels that the web interface adds. Three routes return the UI answer: a managed ChatGPT endpoint such as cloro, a scraping platform such as Apify or Bright Data, and a Playwright stack that you run yourself.

With a managed endpoint the job is one POST request. cloro's endpoint is POST https://api.cloro.dev/v1/monitor/chatgpt. You send a prompt and a country. The response carries text, sources, citationPills and entities, plus markdown, searchQueries and shoppingCards when you ask for them. If you need model output for your own application, the OpenAI API is the correct tool and you do not need a scraper.

Key takeaways

The OpenAI API and the ChatGPT web UI are different surfaces. Citations, shopping cards, brand entities and the search queries ChatGPT issued exist only in the web UI.
At 1,000 queries a day, the source comparison estimates $100 to $300 a month for a managed API and $980 to $2,140 a month for DIY Playwright.
DIY is the better choice with zero budget, a strong engineering team and 8 to 15 hours a month for maintenance.

Steps with a managed endpoint

Get an API key from the provider.
Send a POST request to https://api.cloro.dev/v1/monitor/chatgpt with prompt and country.
Set the include flags for the optional fields that you need.
Set the client timeout to 300 seconds or more.
Read the parsed fields from result in the JSON response.

Why does the OpenAI API return a different answer than the ChatGPT UI?

The chat.completions API and the chatgpt.com web interface use different stacks. The API answers from a base model. The web interface does more work before the answer reaches the screen:

It breaks the prompt into sub-queries (query fan-out).
It runs a web search against OpenAI's search index.
It adds citations from the search results.
For shopping prompts, it adds a product card with merchant offers.

Two measurements show the size of the difference. Across roughly 2,500 prompts, we found that ChatGPT grounded 98.4% of its answers in live sources, at 14.1 sources each. A second measurement on 2026-08-19 gave 78.9% and 12.7 sources. And Surfer's analysis reports only about 20% overlap between API responses and the answers that users see in the ChatGPT UI. That figure is Surfer's and was not reproduced here.

Is there an API that returns what ChatGPT actually shows real users?

Yes. A ChatGPT scraper API runs the prompt in the consumer product and returns the answer that a user sees. OpenAI does not offer one. Its developer API (platform.openai.com) returns model completions, with no source pills, shopping cards or ads.

Four data types exist only in the web UI: sources and citations, shopping cards with merchant offers, brand entities, and the sub-queries that ChatGPT issued before it answered.

How do I get ChatGPT responses programmatically without using the OpenAI API?

There are four routes. The cost column is an estimate at 1,000 queries a day from cloro's comparison of 8 tools, with proxies, CAPTCHA solving and engineer time included.

Route	Examples	What you still build	Estimated cost per month
Managed ChatGPT endpoint	cloro	Nothing. Fields arrive parsed.	$100 to $300
Scraping platform	Apify actors, Bright Data Scraping Browser	Cookie refresh (Apify) or the parsing layer (Bright Data)	$280 to $900
Browser infrastructure or anti-bot API	Browserbase, Browserless, ScrapingBee, ZenRows	Stream assembly, citation parsing, selectors	$449 to $1,040
DIY framework	Playwright with stealth plugins	All of it	$980 to $2,140

The DIY route needs five components:

A stealth-patched Playwright Chromium. Vanilla Playwright is detected on the first navigation.
A residential proxy with a sticky session. Datacenter IPs get a CAPTCHA on the first prompt.
A network interceptor for backend-api/f/conversation. Only that URL carries the answer stream.
A completion check that waits for the [DONE] line in the stream.
A step that clicks the sources button and reads the flyout after the answer completes. Citations are not in the stream.

ChatGPT sends the answer as Server-Sent Events. The first function a DIY scraper needs is the parser for that stream:

import json
from typing import List, Dict, Any

def extract_raw_response(input_string: str) -> List[Dict[str, Any]]:
    """Parse ChatGPT's Server-Sent Events stream."""
    json_objects = []

    # Split by lines that start with "data: "
    lines = input_string.split("\n")

    for line in lines:
        # Skip empty lines and non-data lines
        if not line.strip() or not line.startswith("data: "):
            continue

        # Remove "data: " prefix
        json_str = line[6:].strip()

        # Skip special markers like [DONE]
        if json_str == "[DONE]":
            continue

        # Try to parse as JSON
        try:
            json_obj = json.loads(json_str)

            # Only include if it's a dictionary (object), not string or other types
            if isinstance(json_obj, dict):
                json_objects.append(json_obj)
        except json.JSONDecodeError:
            # Skip invalid JSON
            continue

    return json_objects

The complete Playwright class, the citation extractor and the shopping-card parser are in the full DIY guide.

How do you call an API for the ChatGPT UI answer?

This request comes from the public chatgpt-scraper README:

import requests

payload = {
    'prompt': 'best project management software for remote teams',
    'country': 'US',
    'include': {'markdown': True, 'searchQueries': True},
}

response = requests.post(
    'https://api.cloro.dev/v1/monitor/chatgpt',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    json=payload,
)

print(response.json())

The same call with cURL, this time with shopping cards:

curl -X POST https://api.cloro.dev/v1/monitor/chatgpt \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "best running shoes for flat feet", "country": "US", "include": {"shopping": true}}'

Parameter	Description	Default
`prompt` (required)	The query or question, 1 to 10,000 characters	none
`country` (required)	Country code for localized results (`US`, `GB`, `DE`)	none
`state`	US state code	none
`include.markdown`	Return the answer as Markdown	`false`
`include.html`	Return a URL to the full HTML, which expires after 24 hours	`false`
`include.rawResponse`	Return the unparsed upstream payload	`false`
`include.searchQueries`	Return the query fan-out terms	`false`
`include.shopping`	Return shopping cards and inline products	`false`
`include.ads`	Return sponsored blocks	`false`
`disableWebSearch`	Return the answer ChatGPT gives without a forced web search	`false`

The response:

{
  "success": true,
  "result": {
    "text": "For remote teams, the strongest options are...",
    "citationPills": [{ "citationPillId": 0, "label": "Asana", "url": "https://asana.com", "domain": "asana.com" }],
    "searchQueries": ["best project management software 2026", "asana vs monday remote teams"],
    "markdown": "For remote teams, the strongest options are **Asana**..."
  }
}

Field in `result`	Content
`text`, `markdown`	The answer as plain text and as Markdown
`sources`	Each cited URL with `position`, `url`, `label` and `description`
`citationPills`	The inline citation chips. Entries share a `citationPillId` when one pill cites several sources.
`entities`	The products, brands and concepts that the answer names, each with a type
`searchQueries`	The search terms that ChatGPT ran
`shoppingCards`	Product carousels. Each one groups `products[]` with price and merchant offers.
`inlineProducts`	Products in the answer text, separate from the carousels
`ads`	Sponsored blocks
`map`	Business and location entries with rating, reviews and address
`rawResponse`	The unparsed upstream payload

Three details affect a production client:

Timeout. The README recommends 300 seconds or more, because failed attempts are retried on the server before the response returns. A typical response takes 30 to 45 seconds.
Search queries. The README states that searchQueries comes back empty on ChatGPT's default mobile-web interface. The legacy: true option asks for the interface that returns it, on a best-effort basis.
Batches. For thousands of prompts a day, use POST /v1/async/task with taskType: "CHATGPT" and receive the results by webhook.

The price list has two ChatGPT rows. "ChatGPT (web search)" is the base request at 5 credits. "ChatGPT (full response)" is 7 credits: the base request plus one flat charge of 2 credits for any of include.rawResponse, include.searchQueries, include.ads and include.shopping. These are async prices. A synchronous request to /v1/monitor/chatgpt adds 2 credits, so the two sample requests above cost 9 credits each. The Hobby plan is $100 a month for 250,000 credits. A panel of 1,000 prompts a day uses about 210,000 credits a month at 7 credits, or about 150,000 at 5 credits. The Free plan is 500 credits a month.

What should I use instead of building on unofficial ChatGPT API wrappers?

Use a route that someone maintains against ChatGPT's changes, or budget for that maintenance yourself. Code that depends on ChatGPT's private web endpoint breaks when the endpoint changes. The backend-api/f/conversation path changed twice in 2025. The source guide also records two changes to the stream chunk shape in one year, and the shopping-card event moved from a flat products array to a nested offers block. OpenAI's CSS class names also change between deploys, so class selectors break roughly weekly.

API routers such as LiteLLM and OpenRouter are a different category. They wrap the official API, so they return the API answer and not the UI answer.

The contract matters too. In a 2023 thread on OpenAI's developer forum, users report that the Terms of Use bar data extraction by scraping other than through the API. That risk is highest for an account that accepted the terms. Read the current OpenAI Terms of Use before you start. This is not legal advice.

When is DIY or the OpenAI API the better choice?

The OpenAI API is the better choice when your application needs generated text. A scraper endpoint is built for monitoring. A typical answer takes 30 to 45 seconds and each query is billed, which is expensive for casual use.
DIY Playwright is the better choice with zero budget and a strong engineering team. It is free, open source and has no vendor lock-in. Plan for 8 to 15 engineer hours a month.
A managed endpoint fits recurring monitoring: brand mentions, citation tracking and the same prompt from several countries. At 500,000 to 1 million requests a month, the source guide estimates $4,800 to $14,100 a month for an in-house stack, and engineer time is the largest part.

How these figures were collected

We measured grounding on roughly 2,500 prompts for each of six AI engines. For the tool comparison, we ran 30 queries through chatgpt.com with each of 8 tools and scored six criteria: auth handling, stream assembly, citation extraction, anti-bot survival, selector stability and cost per query.

The monthly costs are estimates. They assume engineer time at $100 an hour, a CAPTCHA solver at $2 per 1,000 challenges on 5% of requests, and mobile residential proxies at $10 per GB. The request, the parameters and the response fields come from the public README and were not changed.

This tutorial is published by cloro, which is one of the options in it. The cloro cost row and field list are self-reported. The 20% overlap figure belongs to Surfer.

How to benchmark a SERP API: six axes and a cost model that survives n=100

Ricardo Batista — Fri, 21 Aug 2026 06:04:00 +0000

Key takeaways

Score six axes separately and never average them. A provider can be fast and shallow, or excellent on head terms and blind on local packs.
Compute cost at your own depth. DataForSEO is half cloro's price at n=10 and level with it at n=100 with AI Overview, and SerpApi's gap widens tenfold.
Score a missing block and an absent block differently, or your trend lines encode the parser's blind spots as real changes in Google.

Most SERP API comparisons publish one number: a price per 1,000 calls, or a median latency off a handful of queries. Both are true and both describe the easy case.

A SERP API is a parser sitting on scraping infrastructure. It can be fast and shallow. It can be cheap at ten results and ruinous at a hundred. It can be excellent on commercial head terms and blind on local packs. Average those into one score and you have hidden every failure mode that will actually bite you.

Run it in this order:

Build the query set. Fifty queries, evenly split across the four classes. Freeze the list, because you will re-run it against every provider and again in three months.
Fix the request shape. Same country, language, depth and AIO flag everywhere. Any provider-specific parameter is a variable you can no longer attribute.
Capture ground truth by hand for ten of them, so a parser miss is distinguishable from an absent block.
Score the six axes as six columns. Do not average.
Compute cost at your depth using each vendor's own unit, at both n=10 and the depth you actually need.
Measure latency under concurrent load, reporting p50 and p95. A sequential median tells you nothing about queue behaviour.
Ask the two questions the harness cannot answer: how schema changes are versioned, and where the data comes from.

Six axes, scored separately

Never collapse them into a single number. What you weight depends on your workload: a rank tracker running nightly batches cares about cost at depth, an agent doing grounding cares about latency, a GEO platform cares almost entirely about AI Overview field fidelity.

Axis	What you are actually testing
AI Overview fidelity	Not "does it return AIO" but in what shape. Cited sources with URLs and positions, a text-only summary, or raw HTML you parse yourself. Only the first survives a Google DOM change without work on your side
Multi-engine coverage	Whether each surface is a real endpoint or a wrapper around the same Google call. Coverage claims are cheap
Envelope completeness	Ads, People Also Ask, related searches, knowledge panel, shopping. Ten of the twelve we tested return the full envelope, so this rarely separates the field. It does catch the ones quietly returning organic only
Geolocation precision	Country, city, or UULE-precise. This is where a local SEO tracker gets burned late, because a national average and a metro result diverge on exactly the queries a client checks by hand
True cost per 1,000	At your depth and feature set, not the advertised base rate. Its own section below
Latency p50 and p95	The p95 is the one that matters. Dispatch faster than the queue clears and the slowest 5% balloon past the median. A 2-second median hides a 15-second tail

Fifty queries, four classes

Same request shape through every provider. Commercial head terms, which stress the envelope. AI-Overview-eligible informational queries, which separate real parsing from a text summary. Local and maps-heavy, which stress geolocation. Long tail, where thin SERPs show up as ambiguous nulls rather than errors.

The classes matter more than the count. A provider that looks identical to its competitors on head terms can be missing a third of AI Overviews on informational ones, and an aggregate success rate will never show you that.

One thing worth stealing: score a missing block and an absent block differently. When a provider returns nothing for an AI Overview, you cannot tell whether the SERP had no Overview or the parser missed it. Capture ground truth by hand for ten queries so you can tell the two apart. Skip that and your trend lines encode the parser's blind spots as real changes in Google.

The cost model, where most evaluations go wrong

Three things decide your bill, and the headline rate is none of them.

Depth. Google removed &num=100 on September 11, 2025, so everyone paginates at 10 results a page. Locomotive Agency put the infrastructure cost increase at roughly 10x at top-100 depth. If you need 100 results you are buying ten pages, and how a vendor bills those pages is now the dominant term.

AI Overview enrichment. Usually a surcharge, not an included field. DataForSEO gates it behind load_async_ai_overview at $0.0006 per keyword, roughly doubling the first-page price. Others include it. Credit-pool vendors charge extra credits.

Billing unit. Per-call, per-successful-result, and per-credit are not comparable until you do the arithmetic at your own depth. A credit pool turns a fixed cost into a variable one. Per-successful-result billing turns your failure rate into a price.

Provider	Unit	n=10	n=100 + AIO
cloro (Hobby, $0.40/1k credits)	3 credits at n=10, 23 at n=100 with AIO	$1.20/1k	$9.20/1k
DataForSEO (Standard queue)	per call, +$2/1k for AIO on page one	$0.60/1k	$9.20/1k
SerpApi (Big Data tier)	per search, one search = one 10-result page (pricing)	$9.17/1k	$91.70/1k

The ranking inverts between those two columns. DataForSEO is half cloro's price at n=10 and level with it at n=100 with AIO. SerpApi's gap widens by an order of magnitude, because each batch of ten bills as its own search. Run this before you shortlist. The provider that wins your evaluation at n=10 can be the most expensive thing you run in production, and the whole category is priced this way here.

Two things the harness cannot tell you

Whether the schema will hold. A benchmark captures one moment. It says nothing about whether the vendor renames a field next quarter and breaks your parsers. You cannot measure this from outside, so ask: how are schema changes versioned, and what is the deprecation window? No answer means a maintenance cost you have not priced.

Where the data comes from. Google sued SerpApi on December 19, 2025, alleging circumvention of anti-bot protections under DMCA 1201 (announcement, Search Engine Land has the detail). SerpApi moved to dismiss in February 2026, arguing it returns only what a signed-out user sees. Unresolved as of August 2026. No court has ruled that scraping public results is unlawful and the suit targets circumvention rather than scraping, but provenance is still a procurement question: official feed, scraped results pages, or the vendor's own index. Where the legal line actually sits is worth reading before you pick on this axis.

Both official routes are closing anyway. Bing's search APIs retired in August 2025, and Google's Custom Search JSON API is closed to new customers with a January 1, 2027 shutdown per Google's own docs. What is actually being retired and what is not is worth checking, because most write-ups get it wrong.

If you want the filled-in version, 12 providers scored on all six axes with the cost table at four depths, it is in Best SERP APIs 2026. Check vendor pricing pages yourself before committing. The category has repriced twice since num=100 and will again.

Best ChatGPT scraper in 2026: what breaks, and what 8 tools cost

Ricardo Batista — Mon, 17 Aug 2026 05:57:00 +0000

The best ChatGPT scraper depends on the job. For brand monitoring, use a managed API that returns parsed citations; of the eight tools below, one returns them parsed and the rest return HTML. For general-purpose scraping at volume, Bright Data's Scraping Browser is the stronger pick. For zero budget with engineering capacity, it is Playwright and a proxy bill. Which of those three you are depends on the table further down, not on the verdict.

Key takeaways

The official OpenAI API cannot answer brand-monitoring questions, because citations, shopping cards and the search-versus-memory decision only exist in the web UI.
At 1,000 queries a day, DIY Playwright runs $980-2,140 a month once proxies, CAPTCHA and 8-15 engineer hours are counted, against $100-300 for a managed API.
Selector maintenance is the hidden line item. OpenAI's dynamic CSS class names change roughly weekly, and the failure is silent rather than loud.

There are two ChatGPTs. The API developers build on, and the web interface 900 million weekly users actually see. Only the second one browses the web, cites sources, renders shopping cards, and decides per-query whether to search or answer from memory.

If you want to know whether ChatGPT recommends your product or names your competitor instead, the API structurally cannot tell you. You have to read the rendered UI, which means scraping one of the better-defended properties on the public web.

What makes chatgpt.com hard to scrape?

1. Cloudflare fingerprints your TLS handshake

Per Scrapfly's analysis, the 2026 stack starts with JA4 TLS fingerprinting. Default requests or urllib fail on the handshake, before any of your logic runs. Then behavioral profiling of timing and movement, JavaScript challenges a static client cannot execute, and Turnstile CAPTCHAs on anything that smells automated.

Datacenter IPs are the fastest way to get nowhere. proxies.sx testing has real mobile IPs surviving 50-100+ queries while datacenter ranges get blocked in the first few requests.

2. The response streams

ChatGPT does not return HTML. It streams tokens over Server-Sent Events, so you keep the connection alive, parse event: message frames as they land, assemble the answer, and detect the end signal. Static HTTP clients see nothing. Headless browsers handle it transparently and charge you in compute and latency.

3. Class names change between deploys

OpenAI's React build emits dynamic classes like ._a4b3f. Anything selecting on them breaks roughly weekly. Semantic selectors (ARIA labels, roles, text content) survive longer, and you still want fallback chains. This is the line item every DIY estimate underprices.

4. Login, 2FA, and session persistence

Most useful workflows need an account, most accounts have 2FA, and the login flow sits behind the same Cloudflare checks. Three realistic paths: export cookies after a manual login and refresh on a schedule, pay someone to hold the session, or run headful Playwright with stealth plugins and accept the breakage.

5. Proxy economics

Databay's 2026 breakdown puts residential proxies at $3-15/GB, with the mobile IPs ChatGPT tolerates at the top of that range. At 1,000 queries a day you burn roughly 3-8 GB a month, plus CAPTCHA credits at $1-3 per 1,000 challenges. That line item alone often exceeds a managed subscription. There is a fuller treatment of proxy selection for SERP work elsewhere.

The eight options

Tool	What it solves	What you still write	Starting price
cloro	Access, SSE assembly, citation parsing, fan-out, auth	Nothing	$100/mo, 500 free credits
Apify	Access and compute, via marketplace actors	Cookie refresh; parsing varies by actor	$49/mo + compute
Bright Data	Access at any scale, 72M+ IPs, CAPTCHA	SSE assembly and all parsing	~$1.50/1k at volume
Browserbase	Persistent sessions, stealth, agent ergonomics	SSE assembly, parsing, selectors	$50/mo, free tier
Browserless	Headless Chrome, self-hostable	SSE assembly, parsing, selectors, some evasion	$50/mo, free Docker image
ScrapingBee	Cloudflare bypass with a clean API	SSE assembly, parsing, selectors	$49/mo, 100k credits
ZenRows	Cloudflare bypass, cheaper tier	SSE assembly, parsing, selectors	$69/mo, 250k credits
Playwright	Nothing. It is the toolkit	Everything	Free, plus your time

This roundup is published by cloro, which is one of the eight tools in it. The prices and capabilities attributed to the other seven come from their own documentation, linked where they are not obvious.

The split that matters is the third column. Every tool except the managed APIs leaves you owning SSE assembly and selector maintenance, which is the part that fails weekly rather than the part that fails loudly.

A few notes the table cannot hold. Apify's actors are community-maintained more often than not, so reliability tracks whoever still cares about that actor. Bright Data's Scraping Browser is genuinely hard for Cloudflare to block and overkill below serious volume. Browserbase is built for agents and its session persistence solves the auth problem cleanly, which is a real advantage here. ScrapingBee's team published their own ChatGPT scraper roundup, which is a decent signal they understand the target.

Out of scope on purpose: the official OpenAI API (different surface), wrappers like LiteLLM and OpenRouter (they wrap the API, not the UI), Cloudflare-bypass libraries like Camoufox and FlareSolverr (they solve the WAF and leave you the rest), and answer APIs like Tavily and Exa (synthesized answers, not the rendered UI).

What it costs at 1,000 queries a day

Tool	Subscription	Proxies/credits	CAPTCHA	Eng hrs/mo	Total $/mo
cloro	$30-300	included	included	0	$30-300
Apify (official actor)	$49	$30-80	included	2	$280-410
ScrapingBee	$49-249	included	included	4	$449-849
ZenRows	$69-249	included	included	4	$469-849
Browserless	$50-100	$100-300	$30-90	6	$580-890
Bright Data	pay-as-you-go	$200-500	included	4	$600-900
Browserbase	$50-200	$50-150	$30-90	6	$730-1,040
Playwright (DIY)	$0	$150-450	$30-90	8-15	$980-2,140

Assumptions, because a cost table without them is decoration: engineer time at $100/hour fully loaded, CAPTCHA solver at $2 per 1,000 challenges firing on 5% of requests, mobile residential proxies at $10/GB, requests averaging 4-8 KB.

Two things fall out. The DIY gap widens with volume, because proxy bandwidth scales linearly while per-call rates flatten or improve with tiers. And the browser-infrastructure tier sits in an awkward middle: cheaper than Bright Data, more expensive than a managed API once you price the parsing work.

Change the engineer-hours assumption and the ranking moves. If your team already runs Playwright infrastructure and selector patches cost you an hour a month rather than ten, DIY wins outright. That is a real case, not a strawman.

Picking

Already on Apify, ChatGPT is one target of several. Use the official actor rather than a community one.
Volume is the binding constraint. Bright Data, and budget for writing the parsing layer.
Building an agent that needs a live session. Browserbase.
You want Cloudflare handled and nothing else. ScrapingBee, or ZenRows if the budget is tight.
Zero budget, real engineering capacity. Playwright with stealth plugins and mobile proxies. Plan for 8-15 hours a month.
You want parsed citations and fan-out with no maintenance. cloro is built for that case and is a monitoring tool rather than a general scraper. If you need arbitrary browser automation, buy browser infrastructure instead.

One legal note, since it comes up in every thread: scraping your own session or publicly visible content is generally fine, while bypassing authentication or pulling other people's data is not. Where the line sits is covered properly elsewhere.

If you are doing this for brand monitoring rather than data collection, the tools built for that job are a different category and probably the better starting point.

How to scrape Google AI Mode: citation pills + place cards

Ricardo Batista — Fri, 14 Aug 2026 05:49:00 +0000

Key takeaways

AI Mode and AI Overview cite the same URLs only 13.7% of the time, so scraping one tells you nothing reliable about the other.
Source URLs live in HTML comments keyed by UUID, not in anchor tags, so a scraper reading <a href> returns an empty citation list rather than an error.
Strip #:~:text scroll-to-text fragments before counting, or one article reads as several distinct URLs and inflates your citation numbers.

Google AI Mode does not put its sources in anchor tags. It renders each one as a button, and hides the actual URL in an HTML comment keyed by UUID. A scraper that reads <a href> finds nothing. That one design choice is most of the work in this post.

The rest is that nothing is in the initial HTML at all. The shell loads, then the answer streams in over a separate call to /async/folwr. Parse on page load and you get an empty container.

Build the URL with udm=50 and aep=11 to force the interface.
Register a network interceptor on https://www.google.com/async/folwr before navigating.
Navigate, and treat a non-200 as a CAPTCHA rather than a dead page.
Poll until the intercepted response lands, with a hard timeout.
Read the answer text off data-session-thread-id, not off a snapshot.
Collect the citation buttons, then rejoin them to their HTML-comment metadata by UUID.
Rewrite the pills into anchors before converting anything to markdown.

Steps 6 and 7 are where the work is. The rest is plumbing.

Why bother with this surface specifically

AI Mode is not AI Overview with a different skin. Per ALM Corp's analysis of 1.3 million AI Mode citations, the two cite the same URLs only 13.7% of the time on semantically similar queries, and google.com self-cites in 17.42% of AI Mode answers, more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. Scraping one tells you nothing reliable about the other.

It is also where the comparison queries go. Seer Interactive's 2026 trigger data puts AI surfaces on 95.4% of comparison queries and 85.9% of question-format ones, and AI Mode passed 75 million daily active users by late 2025 per Alphabet's Q3 2025 earnings call. The query fan-out behind it, where Google splits your question into subtopics and runs many searches at once, is why one answer stitches together so many cited pages.

Surface	Trigger	Citation style	Difficulty
AI Mode	`udm=50`	Pills, metadata in HTML comments	High: async streaming, drifting selectors
AI Overview	Organic query, varies by class	Inline source list above the SERP	Medium: single fetch
Gemini	gemini.google.com	Grounded sources panel	High: separate auth domain
ChatGPT Search	chatgpt.com with search on	Numbered citations	Highest: SSE plus Cloudflare

Force the interface

udm=50 is the switch. aep=11 goes with it.

search_url = build_url_with_params(
    "https://www.google.com/search",
    {
        "udm": 50,        # AI Mode
        "aep": 11,
        "q": prompt,
        "hl": google_params["hl"],
        "gl": google_params["gl"],
    },
)

Wait for the stream, then parse

We drive Playwright and capture the async call, then poll until it lands. Sixty seconds, then fail loudly. A hard timeout beats a silent partial parse, because a half-streamed answer looks like a real answer with fewer citations.

page_interceptor = PlaywrightInterceptor(do_not_block_resources=True)
page_interceptor.add_capture_urls(["https://www.google.com/async/folwr"])
await page_interceptor.setup_page_interceptor(page)

for _ in range(120):                      # 500ms intervals, 60s cap
    if len(page_interceptor.captured_responses):
        break
    await sleep(500)
else:
    raise Exception("Never received AI Mode response after 60 seconds")

The answer text hangs off data-session-thread-id. Target that and read its parent, because AI Mode rewrites its own container several times while streaming and a locator survives that where a snapshot does not.

thread_element = page.locator("[data-session-thread-id]")
text = await thread_element.locator("..").text_content() or ""

The citation pills

Here is the shape you are working against. A UUID ties the visible button to a comment block carrying the real URLs.

<!--Sv6Kpe[["uuid-12345",["label","description"],["https://example.com","source2"]]]-->
<button data-icl-uuid="uuid-12345" data-amic="true">[1]</button>

So: find the buttons, read their UUIDs, regex the comments back out of the page HTML, and filter Google's own domains out of the URL list. One pill can carry several sources.

async def extract_aimode_citation_pills(page: Page) -> Dict[str, List[LinkData]]:
    citation_pills: Dict[str, List[LinkData]] = {}
    pill_locators = page.locator('button[data-icl-uuid][data-amic="true"]')
    page_html = html.unescape(await page.content())

    for i in range(await pill_locators.count()):
        pill_button = pill_locators.nth(i)
        if not await pill_button.is_visible():
            continue
        uuid = await pill_button.get_attribute("data-icl-uuid")
        if not uuid:
            continue

        pattern = rf'<!--Sv6Kpe\[\["{re.escape(uuid)}".*?]]-->'
        current_pill: List[LinkData] = []

        for content in re.findall(pattern, page_html, re.DOTALL):
            desc_match = re.search(
                rf'"{re.escape(uuid)}"\s*,\s*\[\s*"[^"]+"\s*,\s*"([^"]+)"', content
            )
            url = next(
                (u for u in re.findall(r'"(https://[^"]+)"', content)
                 if not any(skip in u for skip in
                            ["google.com", "gstatic.com", "encrypted-tbn"])),
                None,
            )
            if url:
                url = url.split("#:~:text")[0]
                url = url.replace("\\u003d", "=").replace("\\u0026", "&")
                current_pill.append(LinkData(
                    position=len(current_pill) + 1,
                    label=f"Source {len(current_pill) + 1}",
                    url=url,
                    description=desc_match.group(1) if desc_match else None,
                ))

        if current_pill:
            citation_pills[uuid] = current_pill

    return citation_pills

The #:~:text strip matters more than it looks. Google appends scroll-to-text fragments, so the same article comes back as several distinct URLs and your citation counts inflate quietly.

The sources panel needs its own selector per layout. Web-results pages expose it one way, the dialog another. Playwright's actionability checks handle the waiting, so log and continue on timeout rather than killing the run.

sources_selector = (
    '[data-container-id="rhs-col"] [role="dialog"] a'
    if not is_web_results_page
    else "a.ZbQNgf"
)

Converting to markdown without losing the sources

Hand the raw HTML to html2text and every pill becomes a dead button. Rewrite the pills into real anchors first, using the UUID map you already built, then convert.

soup = BeautifulSoup(html_content, "html.parser")
for button in soup.find_all("button", attrs={"data-icl-uuid": True, "data-amic": "true"}):
    uuid = button.get("data-icl-uuid")
    for link_data in reversed(citation_pills.get(uuid, [])):
        anchor = soup.new_tag("a", href=link_data.get("url"))
        anchor.string = link_data.get("label")
        button.insert_after(anchor)
    button.decompose()

h = html2text.HTML2Text()
h.ignore_links = False
h.body_width = 0
markdown = h.handle(str(soup)).strip()

What breaks at scale

A non-200 is usually a CAPTCHA, not a dead page, so solve before you give up. One blocked request should not tank a batch.

response = await page.goto(search_url, timeout=20_000)
if not is_http_success(response.status):
    if not await solve_captcha(page, page_interceptor):
        raise Exception(f"HTTP error: {response.status} (probably captcha)")

The real running cost is selector drift. AI Mode's DOM changes without notice, and a scraper that worked last month returns empty rather than throwing. Alert on success rate, not on exceptions. A sudden drop is a layout change, and you will find it weeks late if you are only watching for errors. You will also want a proxy pool and fingerprint rotation before this runs at any volume.

Or skip it

cloro maintains this pipeline as a managed endpoint, which is the disclosure to weigh this section against. One POST, parsed text and sources back, P50 under 8 seconds.

response = requests.post(
    "https://api.cloro.dev/v1/monitor/aimode",
    headers={"Authorization": "Bearer sk_live_your_api_key"},
    json={"prompt": "What do you know about Tesla's latest updates?",
          "country": "US",
          "include": {"markdown": True}},
)
result = response.json()["result"]
print(f"{len(result['sources'])} citations, fan-out: {result['searchQueries']}")

Built in-house, this runs $5,000 to $10,000 a month once you count engineering time, browser instances and proxies. Below a few thousand queries a month that maths does not favour us, and you should build it. Above that it does, and cloro's AI Mode endpoint absorbs the selector drift.

If you want the dashboard rather than the pipeline, the tools that track this surface are compared separately.