DEV Community: Ricardo Batista

How to benchmark a SERP API: six axes and a cost model that survives n=100

Ricardo Batista — Fri, 21 Aug 2026 06:04:00 +0000

Key takeaways

Score six axes separately and never average them. A provider can be fast and shallow, or excellent on head terms and blind on local packs.
Compute cost at your own depth. DataForSEO is half cloro's price at n=10 and level with it at n=100 with AI Overview, and SerpApi's gap widens tenfold.
Score a missing block and an absent block differently, or your trend lines encode the parser's blind spots as real changes in Google.

Most SERP API comparisons publish one number: a price per 1,000 calls, or a median latency off a handful of queries. Both are true and both describe the easy case.

A SERP API is a parser sitting on scraping infrastructure. It can be fast and shallow. It can be cheap at ten results and ruinous at a hundred. It can be excellent on commercial head terms and blind on local packs. Average those into one score and you have hidden every failure mode that will actually bite you.

Run it in this order:

Build the query set. Fifty queries, evenly split across the four classes. Freeze the list, because you will re-run it against every provider and again in three months.
Fix the request shape. Same country, language, depth and AIO flag everywhere. Any provider-specific parameter is a variable you can no longer attribute.
Capture ground truth by hand for ten of them, so a parser miss is distinguishable from an absent block.
Score the six axes as six columns. Do not average.
Compute cost at your depth using each vendor's own unit, at both n=10 and the depth you actually need.
Measure latency under concurrent load, reporting p50 and p95. A sequential median tells you nothing about queue behaviour.
Ask the two questions the harness cannot answer: how schema changes are versioned, and where the data comes from.

Six axes, scored separately

Never collapse them into a single number. What you weight depends on your workload: a rank tracker running nightly batches cares about cost at depth, an agent doing grounding cares about latency, a GEO platform cares almost entirely about AI Overview field fidelity.

Axis	What you are actually testing
AI Overview fidelity	Not "does it return AIO" but in what shape. Cited sources with URLs and positions, a text-only summary, or raw HTML you parse yourself. Only the first survives a Google DOM change without work on your side
Multi-engine coverage	Whether each surface is a real endpoint or a wrapper around the same Google call. Coverage claims are cheap
Envelope completeness	Ads, People Also Ask, related searches, knowledge panel, shopping. Ten of the twelve we tested return the full envelope, so this rarely separates the field. It does catch the ones quietly returning organic only
Geolocation precision	Country, city, or UULE-precise. This is where a local SEO tracker gets burned late, because a national average and a metro result diverge on exactly the queries a client checks by hand
True cost per 1,000	At your depth and feature set, not the advertised base rate. Its own section below
Latency p50 and p95	The p95 is the one that matters. Dispatch faster than the queue clears and the slowest 5% balloon past the median. A 2-second median hides a 15-second tail

Fifty queries, four classes

Same request shape through every provider. Commercial head terms, which stress the envelope. AI-Overview-eligible informational queries, which separate real parsing from a text summary. Local and maps-heavy, which stress geolocation. Long tail, where thin SERPs show up as ambiguous nulls rather than errors.

The classes matter more than the count. A provider that looks identical to its competitors on head terms can be missing a third of AI Overviews on informational ones, and an aggregate success rate will never show you that.

One thing worth stealing: score a missing block and an absent block differently. When a provider returns nothing for an AI Overview, you cannot tell whether the SERP had no Overview or the parser missed it. Capture ground truth by hand for ten queries so you can tell the two apart. Skip that and your trend lines encode the parser's blind spots as real changes in Google.

The cost model, where most evaluations go wrong

Three things decide your bill, and the headline rate is none of them.

Depth. Google removed &num=100 on September 11, 2025, so everyone paginates at 10 results a page. Locomotive Agency put the infrastructure cost increase at roughly 10x at top-100 depth. If you need 100 results you are buying ten pages, and how a vendor bills those pages is now the dominant term.

AI Overview enrichment. Usually a surcharge, not an included field. DataForSEO gates it behind load_async_ai_overview at $0.0006 per keyword, roughly doubling the first-page price. Others include it. Credit-pool vendors charge extra credits.

Billing unit. Per-call, per-successful-result, and per-credit are not comparable until you do the arithmetic at your own depth. A credit pool turns a fixed cost into a variable one. Per-successful-result billing turns your failure rate into a price.

Provider	Unit	n=10	n=100 + AIO
cloro (Hobby, $0.40/1k credits)	3 credits at n=10, 23 at n=100 with AIO	$1.20/1k	$9.20/1k
DataForSEO (Standard queue)	per call, +$2/1k for AIO on page one	$0.60/1k	$9.20/1k
SerpApi (Big Data tier)	per search, one search = one 10-result page (pricing)	$9.17/1k	$91.70/1k

The ranking inverts between those two columns. DataForSEO is half cloro's price at n=10 and level with it at n=100 with AIO. SerpApi's gap widens by an order of magnitude, because each batch of ten bills as its own search. Run this before you shortlist. The provider that wins your evaluation at n=10 can be the most expensive thing you run in production, and the whole category is priced this way here.

Two things the harness cannot tell you

Whether the schema will hold. A benchmark captures one moment. It says nothing about whether the vendor renames a field next quarter and breaks your parsers. You cannot measure this from outside, so ask: how are schema changes versioned, and what is the deprecation window? No answer means a maintenance cost you have not priced.

Where the data comes from. Google sued SerpApi on December 19, 2025, alleging circumvention of anti-bot protections under DMCA 1201 (announcement, Search Engine Land has the detail). SerpApi moved to dismiss in February 2026, arguing it returns only what a signed-out user sees. Unresolved as of August 2026. No court has ruled that scraping public results is unlawful and the suit targets circumvention rather than scraping, but provenance is still a procurement question: official feed, scraped results pages, or the vendor's own index. Where the legal line actually sits is worth reading before you pick on this axis.

Both official routes are closing anyway. Bing's search APIs retired in August 2025, and Google's Custom Search JSON API is closed to new customers with a January 1, 2027 shutdown per Google's own docs. What is actually being retired and what is not is worth checking, because most write-ups get it wrong.

If you want the filled-in version, 12 providers scored on all six axes with the cost table at four depths, it is in Best SERP APIs 2026. Check vendor pricing pages yourself before committing. The category has repriced twice since num=100 and will again.

LLM visibility tracking tools: 15 compared, and when to build instead

Ricardo Batista — Wed, 19 Aug 2026 06:02:00 +0000

Key takeaways

LLM observability and LLM visibility are unrelated products sold under overlapping search terms. This is about the second one.
Citation overlap between engines on the same prompt runs under 30%, so a pooled six-engine average can hide being invisible on the engine your category actually uses.
Build-versus-buy flips around 50,000 queries a month or 10 tracked clients. Below 5,000 a dashboard wins on time-to-value alone.

First, a disambiguation that wastes a lot of people's time. LLM observability is telemetry for models your application calls: tokens, latency, traces, cost. Datadog, LangSmith, Helicone. LLM visibility is what models say about your brand: whether ChatGPT names you, which sources it cites, how you move against competitors. Same search terms, unrelated products. This is about the second one.

Search Console will not show you any of it, because the surface that decides whether ChatGPT names your brand is the answer generated before anyone clicks.

What do LLM visibility tracking tools actually measure?

Mention rate is the share of sampled answers naming your brand at all. Everything else is a cut of this number. A platform that shows a composite "visibility score" without the mention count underneath is asking you to trust a black box, and that score is not comparable to anyone else's.

Share of voice is your mention rate against the competitors named on the same prompts. It only means something if you control the competitor list, so check whether you define it or the platform infers it from your category.

Citation rate and position is whether the answer links your domain and where you sit in the source list. Mention and citation move independently. Your brand can be named in an answer that cites someone else entirely, and a tool that collapses the two hides the more actionable half.

Sentiment and accuracy is whether the answer describes your product correctly. Newest of the four, least standardised across vendors, so it deserves the most scrutiny in a demo.

Per-engine, not pooled. In our own cross-engine monitoring, citation overlap between engines on the same prompt runs under 30%. A single average across six engines can hide being invisible on the one that matters to your category.

Two numbers for scale before the tools. Ahrefs measured position-1 organic CTR falling roughly 58% on queries that trigger an AI Overview, and per a Globe Newswire industry report only 14% of marketers track AI citations while 89% of brands already appear in AI answers.

The 15

Tool	From	Best for	Notable
Gauge	$599/mo	B2B SaaS doing GEO	Prompt intelligence, Action Center recommendations
Profound	$5,000/mo	Enterprise compliance	10+ engines, sentiment, ISO-certified
Peec AI	$2,000/mo	Competitor depth	Real-time citations, clickstream correlation
Brandlight	$2,000/mo	Accuracy alerting	Brand accuracy monitoring, health scoring
DemandSphere	Custom	Layering AI onto an existing search programme	AI citations on the same footing as SERP share of voice
Nightwatch	Contact	Agencies already rank tracking	Citation Intelligence, unlimited seats, white-label on all plans
SE Ranking	$119/mo	SMBs on a budget	AIO and AI Mode trackers, competitor citation gap
Semrush AI Toolkit	$99/mo add-on	Existing Semrush users	25 prompts, up to 9 competitors
AthenaHQ	$199/mo	Growing SaaS and agencies	Clean dashboard, share of voice, weekly trends
OtterlyAI	$29/mo	Monitoring plus optimisation	ChatGPT Ads and Shopping tracking, GEO audit
Scrunch	Custom	AI-native reporting	Answer-first reporting, competitor benchmarking
Evertune	Custom	Prompt-level analytics	Share of voice broken down per prompt
aiclicks.io	$39/mo promo	Tracking plus content in one place	Built-in writer, GSC integration, prompt database
Authoritas Visibility Explorer	$99/mo	Multi-client agencies	Daily difference reports, 30+ markets
Ahrefs Brand Radar	$129/mo	Existing Ahrefs users	243M+ monthly prompts, joined to Ahrefs SEO data

Metric sets cluster by product focus rather than price. Evertune and AthenaHQ are built around share of voice. Peec AI and Nightwatch lead on citation-level detail. Profound and Brandlight are the two treating sentiment and brand accuracy as first-class rather than as add-ons. DemandSphere is the one that puts AI citations on the same foundation as the search-visibility number you already report to the business.

The gap that shows up in testing is engine coverage versus claimed engine coverage. Most cover ChatGPT, Perplexity, Gemini and AI Overview. Copilot and AI Mode are spottier, and depth varies even where the logo appears on the pricing page. Ask whether each engine is queried through an official API or browser automation, because API-only coverage structurally misses AI Overview and Perplexity's web-search surface, which are the citation-heavy ones.

Worth knowing before you read anyone's citation-gap report: per Otterly's analysis of over a million citations, 73% of sites carry technical barriers (robots.txt blocks, CDN rules, JS-only content) that stop AI crawlers reaching the page at all. Some "citation gap" findings are crawler-access findings wearing a different hat.

How we tested

One B2B SaaS brand and one consumer-product brand, 25 commercial-investigation queries each, four weeks, spanning comparison, definitional, recommendation and how-to intents. We captured ground truth by hand across ChatGPT with web search, Perplexity, the Gemini app, Copilot, AI Overviews and AI Mode, then compared what each tool reported for the same queries on the same days. Scored on engine coverage, citation fidelity, update frequency, reporting depth, pricing fairness, and methodology transparency.

This roundup is published by cloro, a data provider rather than a dashboard, so none of the 15 tools above is a competitor. cloro appears once below, in the infrastructure section, and is not scored against them.

A roundup is editorial judgement on one date. The AI visibility platforms leaderboard is the standing measurement beside it, re-scored weekly and free to cite.

The API question, which most comparisons get wrong

Vendors use "API" for three different things:

Scheduled export. CSV drops, Sheets, sometimes a Looker Studio connector. Fine for reporting, useless for a product feature. This is the most common thing sold as integration.
Read API over their stored data. Query the vendor's results for your tracked prompts. Good for pulling their numbers into your BI stack, still bounded by the prompts, engines and cadence your plan allows.
Raw data API. Submit any prompt to any surface, get the parsed response. Different product category, not a dashboard feature.

Two things a comparison table cannot hold. API availability moves between pricing tiers often enough that any published list is stale within a quarter. And an API on a dashboard product is normally rate-limited against the prompt allowance you already bought, so "has an API" does not mean "can back a product feature."

When building your own starts to make sense

Three bands, from watching teams do this:

Under ~5,000 queries a month. Buy a dashboard. Anything you build will take longer to stand up than the value it returns.
Over ~50,000 queries a month, or 10+ tracked clients. Per-seat dashboard pricing crosses per-call API pricing, and an in-house build pays back in two to four months.
In between. It depends entirely on whether you already have BI and ops infrastructure to host the output. If the answer is a CSV landing in a warehouse, you are paying for a UI layer and throwing it away.

The builds themselves are small. Three patterns cover most of it: nightly scheduler into Postgres into Metabase with Slack alerts; per-client query lists with per-tenant keys and white-label reports; or a daily batch into BigQuery joined against GA4, Search Console and CRM data. Each is one to two weeks of engineering on top of a managed data layer.

What is not small is the parsing layer. Scraping it yourself runs $500-2,000 a month in proxies and browser infrastructure plus two to three months of build, and the hard part is not the scraping. It is that a citation parser working cleanly in January can miss 30% of them by April after a UI change, silently. Official model APIs avoid that and cannot help either, because they do not expose citation data the way a live search surface does.

cloro is one such layer: a single endpoint across ChatGPT, Perplexity, Gemini, Copilot, AI Overview, AI Mode, Google Search and Google News, returning parsed source URLs with position and label, plus query fan-out terms, as JSON rather than screenshots. Credits are deducted only on successful extraction. Hobby is $100/month for 250,000 credits ($0.40 per 1,000), scaling to $0.34 at the top tier, with 500 free credits to test and docs rather than an onboarding call. Async is the right call for nightly batches; sync costs two extra credits and returns inline.

It is not a dashboard, and if what you want is a dashboard then one of the 15 above is the better purchase.

What to pressure-test in a demo

Engine coverage method. Official API or browser automation, per engine. Vagueness here is the answer.
Citation format. Structured queryable URLs, or screenshots. This decides whether you can compute anything yourself.
Query control. Can you import prompts from Search Console and define your own, or does it generate them from your domain?
Cost at your real cadence. Affordable at 50 queries a week is a different product at 500. Most teams find they need weekly rather than daily, which changes the maths a lot.

If you want the ChatGPT surface specifically rather than all six, monitoring ChatGPT mentions is covered end to end separately.

Best ChatGPT scraper in 2026: what breaks, and what 8 tools cost

Ricardo Batista — Mon, 17 Aug 2026 05:57:00 +0000

The best ChatGPT scraper for brand monitoring is a managed API that returns parsed citations, and cloro is the only one of the eight below that ships that parsing rather than handing back HTML. For general-purpose scraping at volume, Bright Data's Scraping Browser is the stronger pick. For zero budget with engineering capacity, it is Playwright and a proxy bill. Which of those three you are depends on the table further down, not on the verdict.

Key takeaways

The official OpenAI API cannot answer brand-monitoring questions, because citations, shopping cards and the search-versus-memory decision only exist in the web UI.
At 1,000 queries a day, DIY Playwright runs $980-2,140 a month once proxies, CAPTCHA and 8-15 engineer hours are counted, against $100-300 for a managed API.
Selector maintenance is the hidden line item. OpenAI's dynamic CSS class names change roughly weekly, and the failure is silent rather than loud.

There are two ChatGPTs. The API developers build on, and the web interface 900 million weekly users actually see. Only the second one browses the web, cites sources, renders shopping cards, and decides per-query whether to search or answer from memory.

If you want to know whether ChatGPT recommends your product or names your competitor instead, the API structurally cannot tell you. You have to read the rendered UI, which means scraping one of the better-defended properties on the public web.

What makes chatgpt.com hard to scrape?

1. Cloudflare fingerprints your TLS handshake

Per Scrapfly's analysis, the 2026 stack starts with JA4 TLS fingerprinting. Default requests or urllib fail on the handshake, before any of your logic runs. Then behavioral profiling of timing and movement, JavaScript challenges a static client cannot execute, and Turnstile CAPTCHAs on anything that smells automated.

Datacenter IPs are the fastest way to get nowhere. proxies.sx testing has real mobile IPs surviving 50-100+ queries while datacenter ranges get blocked in the first few requests.

2. The response streams

ChatGPT does not return HTML. It streams tokens over Server-Sent Events, so you keep the connection alive, parse event: message frames as they land, assemble the answer, and detect the end signal. Static HTTP clients see nothing. Headless browsers handle it transparently and charge you in compute and latency.

3. Class names change between deploys

OpenAI's React build emits dynamic classes like ._a4b3f. Anything selecting on them breaks roughly weekly. Semantic selectors (ARIA labels, roles, text content) survive longer, and you still want fallback chains. This is the line item every DIY estimate underprices.

4. Login, 2FA, and session persistence

Most useful workflows need an account, most accounts have 2FA, and the login flow sits behind the same Cloudflare checks. Three realistic paths: export cookies after a manual login and refresh on a schedule, pay someone to hold the session, or run headful Playwright with stealth plugins and accept the breakage.

5. Proxy economics

Databay's 2026 breakdown puts residential proxies at $3-15/GB, with the mobile IPs ChatGPT tolerates at the top of that range. At 1,000 queries a day you burn roughly 3-8 GB a month, plus CAPTCHA credits at $1-3 per 1,000 challenges. That line item alone often exceeds a managed subscription. There is a fuller treatment of proxy selection for SERP work elsewhere.

The eight options

Tool	What it solves	What you still write	Starting price
cloro	Access, SSE assembly, citation parsing, fan-out, auth	Nothing	$100/mo, 500 free credits
Apify	Access and compute, via marketplace actors	Cookie refresh; parsing varies by actor	$49/mo + compute
Bright Data	Access at any scale, 72M+ IPs, CAPTCHA	SSE assembly and all parsing	~$1.50/1k at volume
Browserbase	Persistent sessions, stealth, agent ergonomics	SSE assembly, parsing, selectors	$50/mo, free tier
Browserless	Headless Chrome, self-hostable	SSE assembly, parsing, selectors, some evasion	$50/mo, free Docker image
ScrapingBee	Cloudflare bypass with a clean API	SSE assembly, parsing, selectors	$49/mo, 100k credits
ZenRows	Cloudflare bypass, cheaper tier	SSE assembly, parsing, selectors	$69/mo, 250k credits
Playwright	Nothing. It is the toolkit	Everything	Free, plus your time

This roundup is published by cloro, which is one of the eight tools in it. The prices and capabilities attributed to the other seven come from their own documentation, linked where they are not obvious.

The split that matters is the third column. Every tool except the managed APIs leaves you owning SSE assembly and selector maintenance, which is the part that fails weekly rather than the part that fails loudly.

A few notes the table cannot hold. Apify's actors are community-maintained more often than not, so reliability tracks whoever still cares about that actor. Bright Data's Scraping Browser is genuinely hard for Cloudflare to block and overkill below serious volume. Browserbase is built for agents and its session persistence solves the auth problem cleanly, which is a real advantage here. ScrapingBee's team published their own ChatGPT scraper roundup, which is a decent signal they understand the target.

Out of scope on purpose: the official OpenAI API (different surface), wrappers like LiteLLM and OpenRouter (they wrap the API, not the UI), Cloudflare-bypass libraries like Camoufox and FlareSolverr (they solve the WAF and leave you the rest), and answer APIs like Tavily and Exa (synthesized answers, not the rendered UI).

What it costs at 1,000 queries a day

Tool	Subscription	Proxies/credits	CAPTCHA	Eng hrs/mo	Total $/mo
cloro	$30-300	included	included	0	$30-300
Apify (official actor)	$49	$30-80	included	2	$280-410
ScrapingBee	$49-249	included	included	4	$449-849
ZenRows	$69-249	included	included	4	$469-849
Browserless	$50-100	$100-300	$30-90	6	$580-890
Bright Data	pay-as-you-go	$200-500	included	4	$600-900
Browserbase	$50-200	$50-150	$30-90	6	$730-1,040
Playwright (DIY)	$0	$150-450	$30-90	8-15	$980-2,140

Assumptions, because a cost table without them is decoration: engineer time at $100/hour fully loaded, CAPTCHA solver at $2 per 1,000 challenges firing on 5% of requests, mobile residential proxies at $10/GB, requests averaging 4-8 KB.

Two things fall out. The DIY gap widens with volume, because proxy bandwidth scales linearly while per-call rates flatten or improve with tiers. And the browser-infrastructure tier sits in an awkward middle: cheaper than Bright Data, more expensive than a managed API once you price the parsing work.

Change the engineer-hours assumption and the ranking moves. If your team already runs Playwright infrastructure and selector patches cost you an hour a month rather than ten, DIY wins outright. That is a real case, not a strawman.

Picking

Already on Apify, ChatGPT is one target of several. Use the official actor rather than a community one.
Volume is the binding constraint. Bright Data, and budget for writing the parsing layer.
Building an agent that needs a live session. Browserbase.
You want Cloudflare handled and nothing else. ScrapingBee, or ZenRows if the budget is tight.
Zero budget, real engineering capacity. Playwright with stealth plugins and mobile proxies. Plan for 8-15 hours a month.
You want parsed citations and fan-out with no maintenance. cloro is built for that case and is a monitoring tool rather than a general scraper. If you need arbitrary browser automation, buy browser infrastructure instead.

One legal note, since it comes up in every thread: scraping your own session or publicly visible content is generally fine, while bypassing authentication or pulling other people's data is not. Where the line sits is covered properly elsewhere.

If you are doing this for brand monitoring rather than data collection, the tools built for that job are a different category and probably the better starting point.

How to scrape Google AI Mode: citation pills + place cards

Ricardo Batista — Fri, 14 Aug 2026 05:49:00 +0000

Key takeaways

AI Mode and AI Overview cite the same URLs only 13.7% of the time, so scraping one tells you nothing reliable about the other.
Source URLs live in HTML comments keyed by UUID, not in anchor tags, so a scraper reading <a href> returns an empty citation list rather than an error.
Strip #:~:text scroll-to-text fragments before counting, or one article reads as several distinct URLs and inflates your citation numbers.

Google AI Mode does not put its sources in anchor tags. It renders each one as a button, and hides the actual URL in an HTML comment keyed by UUID. A scraper that reads <a href> finds nothing. That one design choice is most of the work in this post.

The rest is that nothing is in the initial HTML at all. The shell loads, then the answer streams in over a separate call to /async/folwr. Parse on page load and you get an empty container.

Build the URL with udm=50 and aep=11 to force the interface.
Register a network interceptor on https://www.google.com/async/folwr before navigating.
Navigate, and treat a non-200 as a CAPTCHA rather than a dead page.
Poll until the intercepted response lands, with a hard timeout.
Read the answer text off data-session-thread-id, not off a snapshot.
Collect the citation buttons, then rejoin them to their HTML-comment metadata by UUID.
Rewrite the pills into anchors before converting anything to markdown.

Steps 6 and 7 are where the work is. The rest is plumbing.

Why bother with this surface specifically

AI Mode is not AI Overview with a different skin. Per ALM Corp's analysis of 1.3 million AI Mode citations, the two cite the same URLs only 13.7% of the time on semantically similar queries, and google.com self-cites in 17.42% of AI Mode answers, more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. Scraping one tells you nothing reliable about the other.

It is also where the comparison queries go. Seer Interactive's 2026 trigger data puts AI surfaces on 95.4% of comparison queries and 85.9% of question-format ones, and AI Mode passed 75 million daily active users by late 2025 per Alphabet's Q3 2025 earnings call. The query fan-out behind it, where Google splits your question into subtopics and runs many searches at once, is why one answer stitches together so many cited pages.

Surface	Trigger	Citation style	Difficulty
AI Mode	`udm=50`	Pills, metadata in HTML comments	High: async streaming, drifting selectors
AI Overview	Organic query, varies by class	Inline source list above the SERP	Medium: single fetch
Gemini	gemini.google.com	Grounded sources panel	High: separate auth domain
ChatGPT Search	chatgpt.com with search on	Numbered citations	Highest: SSE plus Cloudflare

Force the interface

udm=50 is the switch. aep=11 goes with it.

search_url = build_url_with_params(
    "https://www.google.com/search",
    {
        "udm": 50,        # AI Mode
        "aep": 11,
        "q": prompt,
        "hl": google_params["hl"],
        "gl": google_params["gl"],
    },
)

Wait for the stream, then parse

We drive Playwright and capture the async call, then poll until it lands. Sixty seconds, then fail loudly. A hard timeout beats a silent partial parse, because a half-streamed answer looks like a real answer with fewer citations.

page_interceptor = PlaywrightInterceptor(do_not_block_resources=True)
page_interceptor.add_capture_urls(["https://www.google.com/async/folwr"])
await page_interceptor.setup_page_interceptor(page)

for _ in range(120):                      # 500ms intervals, 60s cap
    if len(page_interceptor.captured_responses):
        break
    await sleep(500)
else:
    raise Exception("Never received AI Mode response after 60 seconds")

The answer text hangs off data-session-thread-id. Target that and read its parent, because AI Mode rewrites its own container several times while streaming and a locator survives that where a snapshot does not.

thread_element = page.locator("[data-session-thread-id]")
text = await thread_element.locator("..").text_content() or ""

The citation pills

Here is the shape you are working against. A UUID ties the visible button to a comment block carrying the real URLs.

<!--Sv6Kpe[["uuid-12345",["label","description"],["https://example.com","source2"]]]-->
<button data-icl-uuid="uuid-12345" data-amic="true">[1]</button>

So: find the buttons, read their UUIDs, regex the comments back out of the page HTML, and filter Google's own domains out of the URL list. One pill can carry several sources.

async def extract_aimode_citation_pills(page: Page) -> Dict[str, List[LinkData]]:
    citation_pills: Dict[str, List[LinkData]] = {}
    pill_locators = page.locator('button[data-icl-uuid][data-amic="true"]')
    page_html = html.unescape(await page.content())

    for i in range(await pill_locators.count()):
        pill_button = pill_locators.nth(i)
        if not await pill_button.is_visible():
            continue
        uuid = await pill_button.get_attribute("data-icl-uuid")
        if not uuid:
            continue

        pattern = rf'<!--Sv6Kpe\[\["{re.escape(uuid)}".*?]]-->'
        current_pill: List[LinkData] = []

        for content in re.findall(pattern, page_html, re.DOTALL):
            desc_match = re.search(
                rf'"{re.escape(uuid)}"\s*,\s*\[\s*"[^"]+"\s*,\s*"([^"]+)"', content
            )
            url = next(
                (u for u in re.findall(r'"(https://[^"]+)"', content)
                 if not any(skip in u for skip in
                            ["google.com", "gstatic.com", "encrypted-tbn"])),
                None,
            )
            if url:
                url = url.split("#:~:text")[0]
                url = url.replace("\\u003d", "=").replace("\\u0026", "&")
                current_pill.append(LinkData(
                    position=len(current_pill) + 1,
                    label=f"Source {len(current_pill) + 1}",
                    url=url,
                    description=desc_match.group(1) if desc_match else None,
                ))

        if current_pill:
            citation_pills[uuid] = current_pill

    return citation_pills

The #:~:text strip matters more than it looks. Google appends scroll-to-text fragments, so the same article comes back as several distinct URLs and your citation counts inflate quietly.

The sources panel needs its own selector per layout. Web-results pages expose it one way, the dialog another. Playwright's actionability checks handle the waiting, so log and continue on timeout rather than killing the run.

sources_selector = (
    '[data-container-id="rhs-col"] [role="dialog"] a'
    if not is_web_results_page
    else "a.ZbQNgf"
)

Converting to markdown without losing the sources

Hand the raw HTML to html2text and every pill becomes a dead button. Rewrite the pills into real anchors first, using the UUID map you already built, then convert.

soup = BeautifulSoup(html_content, "html.parser")
for button in soup.find_all("button", attrs={"data-icl-uuid": True, "data-amic": "true"}):
    uuid = button.get("data-icl-uuid")
    for link_data in reversed(citation_pills.get(uuid, [])):
        anchor = soup.new_tag("a", href=link_data.get("url"))
        anchor.string = link_data.get("label")
        button.insert_after(anchor)
    button.decompose()

h = html2text.HTML2Text()
h.ignore_links = False
h.body_width = 0
markdown = h.handle(str(soup)).strip()

What breaks at scale

A non-200 is usually a CAPTCHA, not a dead page, so solve before you give up. One blocked request should not tank a batch.

response = await page.goto(search_url, timeout=20_000)
if not is_http_success(response.status):
    if not await solve_captcha(page, page_interceptor):
        raise Exception(f"HTTP error: {response.status} (probably captcha)")

The real running cost is selector drift. AI Mode's DOM changes without notice, and a scraper that worked last month returns empty rather than throwing. Alert on success rate, not on exceptions. A sudden drop is a layout change, and you will find it weeks late if you are only watching for errors. You will also want a proxy pool and fingerprint rotation before this runs at any volume.

Or skip it

cloro maintains this pipeline as a managed endpoint, which is the disclosure to weigh this section against. One POST, parsed text and sources back, P50 under 8 seconds.

response = requests.post(
    "https://api.cloro.dev/v1/monitor/aimode",
    headers={"Authorization": "Bearer sk_live_your_api_key"},
    json={"prompt": "What do you know about Tesla's latest updates?",
          "country": "US",
          "include": {"markdown": True}},
)
result = response.json()["result"]
print(f"{len(result['sources'])} citations, fan-out: {result['searchQueries']}")

Built in-house, this runs $5,000 to $10,000 a month once you count engineering time, browser instances and proxies. Below a few thousand queries a month that maths does not favour us, and you should build it. Above that it does, and cloro's AI Mode endpoint absorbs the selector drift.

If you want the dashboard rather than the pipeline, the tools that track this surface are compared separately.

SERP API reliability at scale: uptime, latency, concurrency

Ricardo Batista — Wed, 12 Aug 2026 05:46:10 +0000

Key takeaways

Two of the four headline reliability numbers in this category are not uptime at all: DataForSEO publishes 99.95% with no credit terms, and Bright Data's 99.99% is a success rate from a benchmark with a disclosed vested interest.
At 10,000 requests a day the P95 decides your architecture, not the median. Bright Data measures 2.61s P50 against 4.92s P95, and a 2-second median can hide a 15-second tail once queuing starts.
Google's &num=100 removal plus DataForSEO's depth-based pricing took the Live tier from $2.00 to $17.50 per 1,000 at 100 results with AI Overview, a ninefold jump from a change you did not make.

Below a few hundred requests a day, every SERP API looks fine. The failure modes only show up somewhere past 10,000: queue latency, rate-limit rejections, concurrency ceilings. By then you have built on top of one.

Three things decide it, and only one of them appears on a pricing page.

This comparison is published by cloro, which is one of the four providers in it. Every figure attributed to another vendor is sourced below; the cloro rows are self-reported and should be read as such.

What does a SERP API uptime SLA actually commit to?

Provider	Uptime SLA	Median response	Rate limits
cloro	99.9% with service credits	Sub-second (async queue)	Parallel lanes, no hard cap
SerpApi	None published	1.96s	Hard monthly cap, request queuing
DataForSEO	99.95%, no credit terms stated	Live fastest; ~5-min Standard queue	High default RPS, async batching
Bright Data	None published	2.61s standard, 0.89s premium	Configurable 5-20+, no documented ceiling

Two of those four numbers are not what they look like.

DataForSEO's 99.95% is higher than ours and the documentation states no credit or penalty terms. That makes it a target rather than a remedy. A published SLA without credits tells you what a provider aims for, not what you recover when they miss.

Bright Data's widely quoted 99.99% is a success rate, not uptime, and it comes from a benchmark whose author disclosed a vested interest. Success rate measures request completion. Uptime measures whether the endpoint answers at all. They are different failures and only one of them wakes you up.

So when you compare this column, check three things: is it uptime or success rate, is it contractual or measured, and does anything happen when they miss it.

Latency: the median is the wrong number

Almost every published latency figure is measured sequentially, which is the easy case. What decides production behaviour is what happens when you dispatch faster than the provider's queue clears.

The best public numbers come from that same 30-day benchmark of nine providers: Bright Data's standard P50 at 2.61s against a P95 of 4.92s, in a field ranging from Serper at 2.23s to DataForSEO at 4.54s. Premium routing takes the P50 to 0.89s, and that is the vendor-adjacent figure, so discount it.

At 10,000 daily requests the P95 is the number that matters. A 2-second median hides a 15-second tail once queuing kicks in. An overnight rank tracker can absorb that tail. Brand monitoring cannot, because a five-minute delay on an incident query defeats the point.

DataForSEO's Standard queue is the clearest case: one tester measured async callbacks averaging 12 seconds for 100 keywords, stretching toward five minutes at production volume. That is fine for scheduled work and useless for anything real-time.

Concurrency is an architecture choice, not a setting

A 10k batch survives or fails on this, and each provider picked a different model.

SerpApi queues. Hit the rate limit and requests wait behind one another rather than running in parallel, so latency climbs with volume. Exhaust the monthly cap and requests fail outright until the next billing cycle. If your volume is variable you overprovision for the worst day and pay for headroom you rarely use.

DataForSEO batches asynchronously. Submit, wait, retrieve. It absorbs thousands of tasks without per-request rate errors, and you pay for it in freshness. No requests-per-second ceiling appears in the docs.

Bright Data hands you the dial. Its own rank-tracking demo defaults to 5 concurrent and recommends 5 to 10, documenting a 1 to 20+ range with no published ceiling. Past the recommended range you are tuning against undocumented limits and watching for rate errors yourself.

cloro runs parallel lanes behind an async job queue. You POST a batch, get a job ID back in milliseconds, and results arrive by webhook as workers finish. Request 10,000 does not wait behind the other 9,999, so adding volume widens the fan-out instead of lengthening the wait. Concurrency scales by tier rather than capping at a fixed lane count.

The honest summary of that difference: SerpApi and DataForSEO both trade real-time responsiveness for predictable batch cost, in opposite directions. If your workload is nightly and cost-sensitive, DataForSEO's queue is the right trade and cloro is overkill.

AI Overview coverage, where the gap is measurable

For GEO work the question is whether you get AI Overview data as consistent structured fields, not whether the provider returns it at all.

SerpApi detected AI Overviews in 68% of qualifying queries in a published 25-query benchmark. Roughly a third of pages that should show an Overview returned nothing detectable, and that is the ugly kind of gap: a missing Overview reads identically to an absent one, so your trend lines encode the parser's blind spots as changes in Google.

DataForSEO makes it opt-in with load_async_ai_overview at $0.0006 per keyword, which doubles the Live tier to about $4.00 per 1,000 at ten results, and $17.50 per 1,000 at 100 results with AIO. Structured, but you submit and poll rather than reading it inline.

cloro returns Overview text, cited sources and position in the same payload as organic results. The reason that matters is not convenience: Google changed the Overview DOM during its January 2025 JS-rendering update, and anyone parsing raw HTML rebuilt their extraction. Provider-managed fields absorb that server-side. It is the strongest argument for buying rather than building, covered at length in this build-versus-buy breakdown.

Cost at volume, which moved twice in 2025

Model this at your depth, not the advertised rate.

SerpApi is subscription with hard caps and no pay-as-you-go: roughly $2,500 at 100k monthly requests, $25,000 at 1M. DataForSEO is the cheapest at raw volume, about $0.60 per 1,000 on Standard, so 300k requests a month lands near $180. Bright Data prices per call near $0.005 with a minimum commitment cited around $499/month.

Then two changes reset the arithmetic. DataForSEO moved to depth-based pricing on September 19, 2025, and Google deprecated &num=100 on September 11, 2025, forcing pagination at 10 results a page. If you need 100 results plus AI Overview, that Live tier goes from $2.00 to $17.50 per 1,000. A ninefold jump, from a parameter change you did not make. A fuller pricing breakdown runs the same numbers at four depths.

Pick by workload

Nightly batches, cost-sensitive: DataForSEO. The queue latency is real and irrelevant to you, and $0.60 per 1,000 is hard to argue with. Watch the depth pricing.
City-level geo-targeting across many markets: Bright Data. 195 countries with city-level targeting is the widest coverage published, and you manage concurrency yourself.
Low volume, many engines, want it managed: SerpApi. Live inside a tier and the caps never bite.
Real-time monitoring past 10k/day: cloro is built for this shape, and its constraint is that it is API-first. If you want a dashboard rather than endpoints, buy a dashboard.

Whatever you pick, run your own 10k-request test against it under concurrent load before you commit. Every number above, cloro's included, was measured by someone with an interest in the result.

🚀 Launching FidForward! 🚀

Ricardo Batista — Mon, 08 Jul 2024 16:54:15 +0000

Today, Bernardo and I are excited to launch FidForward in private beta!

We're looking for companies with 10-50 employees willing to pilot the next generation of performance management.

With two pilot programs already running successfully (achieving a 30-50% improvement in eNPS), we’re eager to expand FidForward to more companies to validate these results further.

If you’re a founder seeking to implement performance management or a larger company aiming to elevate your current system, please reach out to me!

🔗 Learn more about FidForward at FidForward.com

We were tired of bureaucracy, so we built an opensource repo for the best guides

Ricardo Batista — Sun, 23 Jun 2024 21:09:11 +0000

Today we are launching Tramitit, a shared database on getting all those poor-UX local procedures done in a much simpler way!

How it works: The community provides detailed walkthroughs on every possible bureaucratic process you might go through.
For the community by the community: The content on this website is curated, verified, and rated by the community.
Welcoming providers that can help our users: Although our guides are as simple as possible, users may still prefer to delegate tasks to a curated provider.

We are in a bit a chicken & egg problem: for the traffic to pick up and have contributors, we need to start by adding some content - some of which we have already gone through and fixed it, other not yet there. We will get there over time, and work with providers that can also bring the quality up.

You can find our GitHub repo here, where you can contribute as well!
Our website is also open-source, so feel free to add anything.

Getting started with Open SaaS

Ricardo Batista — Sun, 21 Apr 2024 07:34:17 +0000

When building Article Generation and Meeting Reminders, I decided to get started as fast as possible. So I looked for a free boilerplate and stumbled upon Open SaaS, which used YC-backed Wasp. It is a full-stack React + NodeJS + Prisma that takes 8 hours to get started with.

Their documentation is brilliant, and great support over Discord. I decided to list here the full walkthrough on how to set up the development environment so that it is a simple copy-paste for you.

For frontend deployment, I used Netlify (for the generous free package) and the recommended fly.io for server + database (also cheap package).

Run development commands

Start development database:

cd app && wasp start db

View development database in Prisma:

wasp db studio

Start development app:

cd app && wasp start

Start development blog:

cd blog && npm install && npm run dev

Run frontend deployment commands

Locate the folder:

cd .wasp/build/web-app

Install dependencies:

npm install && REACT_APP_API_URL={your-server-URL} npm run build

Deploy preview:

netlify deploy

Deploy to production:

netlify deploy --prod

Every time there is a Wasp app rebuild (with wasp build), it will remove your .wasp/build/ directory. Until the GitHub issue is fixed, you need to update netlify.toml’s publish line to:

  publish = ".wasp/build/web-app/build"

Run backend & db deployment commands

Locate the folder:

cd .wasp/build/

Deploy preview:

flyctl deploy --remote-only --config ../../src/shared/fly-server.toml

Every time there is a Wasp app rebuild (with wasp build), it will remove your .wasp/build/ directory. That’s why there is a copy of the file in app/src/shared/.

Run blog deployment commands

Locate the folder:

cd blog

Build the app:

npm run build

Deploy preview:

netlify deploy

Deploy to production:

netlify deploy --prod

Good luck with building, hopefully it will be faster now!