The $450 Headless Chrome Bill That Made Me Try a SERP API

I spent the last quarter scraping Google with headless Chrome. I run a small price-monitoring service for a few e-commerce clients, and the original plan was to use Playwright on one server and keep the monthly cost near zero. The API route felt like paying for something I could build myself. Ninety days later the infrastructure bill hit $450 and I still couldn't reliably get a SERP back.

It started small. One VPS, a few cron jobs, Playwright with a real browser profile. Week one was fine. Week two, Google started showing the consent wall to datacenter IPs. Week three, one of the client's keywords came back with a CAPTCHA about a third of the time. By the end of the first month I'd bought a proxy pool, added a CAPTCHA solver, and was still losing requests.

Here's where the money actually went.

The month-by-month breakdown

Month	What broke	What I added	Cost
1	consent walls, intermittent blocks	rotating proxies	$120
2	CAPTCHA rate climbed to ~20%	solver + retry loop	$80
3	Google changed the SERP layout	parser rewrite, bigger server	$250
Total			$450

Month three was the worst. Google changed the results page layout, my CSS selectors all broke, and I spent two weeks rewriting the parser while the proxy bill kept running. That's the $250 month.

The server line is sneaky. Every one of those items is a fixed cost that keeps charging whether you succeed or not. My best week was 87% success. That means 13 out of every 100 checks came back empty, which meant retries, which meant more proxy traffic, which meant more money.

What the switch looked like

I kept expecting the API route to be painful. It wasn't. One POST to /google/search with an X-API-Key header, and you get the SERP back as JSON. No browser, no proxy rotation, no consent walls.

curl -X POST "https://api.serpbase.dev/google/search" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $SERPBASE_API_KEY" \
  -d '{"q": "wireless earbuds", "hl": "en", "gl": "us"}'

The response has an organic array with rank, title, link, and snippet per result — the exact fields my rank checker needed. It also includes people_also_ask, knowledge_graph, and ai_overview when Google shows them, which my headless parser was ignoring anyway.

The migration was a 40-line diff. The old code parsed HTML selectors that broke every few weeks. The new code reads a JSON key. That's the entire difference.

import requests

def check_rank(keyword, key):
    r = requests.post(
        "https://api.serpbase.dev/google/search",
        headers={"X-API-Key": key, "Content-Type": "application/json"},
        json={"q": keyword, "hl": "en", "gl": "us"},
    )
    r.raise_for_status()
    organic = r.json().get("organic", [])
    return [(i + 1, item["title"], item["link"])
            for i, item in enumerate(organic)]

print(check_rank("wireless earbuds", "your_api_key"))

The cost math

My workload is about 100,000 searches a month — a few thousand keywords checked daily, plus price checks. On the Growth pack that's $50 for 125,000 searches at $0.40/1k, so around $40/month. On Pro it drops to about $35/month at $0.35/1k. Credits don't expire, which matters when a client pauses and I stop querying for two weeks — the money stays.

Search, news, and videos are 1 credit per call; images and both Maps endpoints are 2. Failed dispatches get refunded automatically, so a timeout doesn't cost me anything. That single rule is worth more than the price difference. Under headless Chrome, a timeout cost me proxy bandwidth and retry time. Here it costs nothing.

P50 latency is about 1.4 seconds and the SLA is 99.9%. I don't have to think about the infrastructure anymore, and that's the line I never budgeted for before: the hours.

Where headless Chrome still wins

The browser still has a place. If you need the raw DOM — parsing a widget Google only renders for certain sessions, or testing against the exact page your scraper targets — headless is the right tool. If your setup already handles captchas and layout drift and you're happy with the maintenance, there's no reason to rip it out.

The math only tips when you'd rather spend the hours on your product than on keeping a scraper alive.

What I'd do differently

If I were starting over, I'd skip the browser stage entirely. Buy 100 searches on the free signup to test the response shape, then size a pack to actual volume. The entry pack is $3 for 10,000 searches, expiring one month after purchase, available once per account per month. I'll never discount maintenance hours again.

Source: SERP API comparison, updated Apr 23, 2026.

Full parameter and response reference: serpbase.dev.

DEV Community