DEV Community

Cover image for What breaks when you run your own Google scraper, and how many engineering hours does it cost?
Ricardo Batista for cloro

Posted on

What breaks when you run your own Google scraper, and how many engineering hours does it cost?

At 100,000 Google searches a month, a self-run scraper costs about $2,450 to $5,150 a month in the model below, and engineer hours are 86% to 90% of that. The four SERP APIs priced below do the same volume for $200 to $725 a month.

  • What breaks: a default headless browser got Google's /sorry/ page on 9 of 9 requests in our test, and two newer failures return HTTP 200 with no usable data.
  • Engineering time: about 20 to 46 hours a month to keep the scraper running, plus about 80 hours to build it. These hours are assumptions, and the arithmetic is below.
  • Proxies alone: residential bandwidth can cost less than API credits. Bandwidth plus browsers plus hours does not.

What actually breaks when you build a Google scraper in-house?

Eight things, in about the order a new scraper meets them: the /sorry/ page, CAPTCHAs, the verify page, poisoned results, the end of num=100, the AI Overview, layout changes and proxy cost. Three of them (the verify page, poisoned results and layout changes) give HTTP 200, so the scraper reports success while the data is wrong.

1. A default headless browser gets the /sorry/ page

On November 19, 2026 we sent 3 queries to google.com/search from one residential connection, with Python requests and with Playwright. Python requests got Google's cookie-consent page on 3 of 3. Playwright's headless Chromium, with no stealth changes, got redirected to /sorry/ on 9 of 9 requests in three setups, starting with the first request. Its user agent included HeadlessChrome.

The blocked responses still used bandwidth: 81 KB to 838 KB each. A residential proxy bills for those bytes too.

2. CAPTCHAs follow the fingerprint, not only the IP

Google scores more than the IP. Anti-bot systems read the TLS handshake before any header, with JA3 and JA4 fingerprints, and reCAPTCHA v3 returns a risk score for each request. A clean residential IP on a badly configured client still gets a challenge.

The solve itself is not expensive. 2Captcha charges €0.99 to €2.80 per 1,000 reCAPTCHA solves. The cost is the retry state machine around it.

3. The "Verifying your request" page returns HTTP 200

From June 25, 2026, Google started to answer some flagged sessions with a JavaScript verification page. It returns HTTP 200 and a valid HTML document with no results, so a status-code check logs a success. cloro's write-up of the rollout records that a customer found it because the AI Overview data came back empty while the success rate stayed flat. To get through, you must run Google's script in a real browser, in the same session that got flagged. A new IP does not help.

4. Poisoned SERPs look correct and are wrong

Since late September 2026, Google answers some automated searches with a complete page of the wrong sites: ten organic results, ads, People also ask and an AI Overview. Nozzle named this a poisoned SERP on October 10, 2026. In cloro's measurement, 71.5% of its own Google pages were poisoned at the worst point (September 30 to October 12). On October 26 at 20:00 UTC, Google changed something again, and all four SERP APIs in that benchmark lost the full page in the same hour. cloro fell to 11.1% full-page reads on October 27.

So a SERP API does not remove this failure. It moves the work to the vendor. Ask any vendor how it detects a poisoned page, because a status code and a result count will not.

5. num=100 is gone

Google stopped honoring &num=100 around September 10, 2025 (Search Engine Journal, September 15, 2025). The top 100 results now take 10 page loads of 10 results each. That is 10 times the requests, 10 times the bandwidth and 10 times the chances to get blocked for each keyword.

6. The AI Overview loads separately

cloro's SERP feature census found an AI Overview on 85.4% of 69,817 Google results across 41 markets. The sample leans toward question-shaped queries. The AI Overview is drawn client-side, so a plain HTTP fetch does not get it, and a browser must wait for it. The APIs have the same problem. SerpApi's docs say Google "sometimes requires an additional request" for it. DataForSEO needs load_async_ai_overview, which adds $0.002 per request.

7. The layout changes without notice

Google's class names are obfuscated and change without an announcement. A selector that works today can return nothing next week. The page does not throw an error. Your parser returns an empty list, and you find out from a downstream chart.

8. Proxies cost per GB and per country

Residential bandwidth, read on November 19, 2026:

Provider Price per GB
Decodo $3.00 at 50 GB ($150 a month), $2.50 at 250 GB ($625), $4.00 pay as you go
Oxylabs $3.50 up to 142 GB, $2.00 from 1,250 GB
Bright Data $8.00 pay as you go list price, $4.00 with a 3-month promo code
Webshare $3.50 at 1 GB, $1.40 at 3,000 GB

Google uses your location, which it can read from the IP address, to localize results, so rank data for 30 countries needs healthy IPs in 30 countries.

How much engineering time does maintaining a Google scraper really take?

About 20 to 46 hours a month, or $1,644 to $3,862 at a loaded US median rate, plus about 80 hours to build it. No public study measures this for Google scrapers, so these numbers are assumptions. Each one is stated, so you can put your own number in.

Item Assumption Hours a month
Routine upkeep: selectors, proxy pool health, alerts 4 to 8 hours a week 17.3 to 34.7
Google-side changes 4 changes in 14 months, 8 to 40 hours each 2.3 to 11.4
Running total 19.6 to 46.1
First build: queue, browser workers, parser, monitoring 80 hours, spread over 12 months 6.7

The 4 changes are items 3, 4 and 5 above, plus the October 26 change, from September 2025 to November 2026. The 8-hour low end is measured: cloro shipped its fix for the verify page 8 hours after it appeared, on a stack that already ran real browsers. A team without that stack starts further back. The 40-hour high end is one engineer-week, and it is an assumption.

To price an hour: the US median pay for software developers, QA analysts and testers was $134,040 in May 2025 (BLS). With a 1.3x loaded-cost multiplier, which is an assumption, that is $174,252 a year. Divided by 2,080 working hours, one hour costs $83.78.

  • 19.6 hours x $83.78 = $1,644 a month
  • 46.1 hours x $83.78 = $3,862 a month
  • 80 build hours x $83.78 = $6,702, or $558 a month over 12 months

Is it cheaper to scrape Google myself with proxies or to use a SERP API?

At 100,000 searches a month, a SERP API is cheaper: $200 to $725 against $2,451 to $5,150 for the DIY model. The DIY lines, for the top 10 results in one country:

Line Low case High case Basis
Residential bandwidth $150 $625 0.5 MB or 2 MB a search (assumption), so 50 GB or 200 GB, on Decodo's 50 GB or 250 GB plan
Browser time $99 $105 10 or 20 seconds a search (assumption), so 278 or 556 hours, on Browserbase Startup: $99 for 500 hours, then $0.10 an hour
Engineering, running $1,644 $3,862 19.6 or 46.1 hours at $83.78
Engineering, build $558 $558 80 hours over 12 months
Total $2,451 $5,150
CAPTCHA solves €10 €280 10% or 100% of searches solved at €0.99 or €2.80 per 1,000

The bandwidth per search is the least certain input. Our test could not measure a full results page, because every browser request got the /sorry/ page. A /sorry/ page alone was 81 KB to 838 KB.

Engineer hours are 86% to 90% of the total. The proxy bill is the line people quote, and it is the small one.

Are SERP API credits cheaper than running residential proxies?

On bandwidth alone, proxies can be cheaper. At 0.5 MB a search and $3.00 per GB, bandwidth costs $1.50 per 1,000 searches. That is less than most of the API prices below. The comparison changes when you add browser time and hours, because the API price includes both.

The four SERP APIs below are the four in cloro's poisoned-SERP benchmark. Each row is a live (real-time) request with the AI Overview, at 100,000 searches a month:

API List rate per 1,000 Bill for 100,000 a month Plan and notes
DataForSEO (Live) $2.00 to $4.00 $200 to $400 $2 per 1,000 SERPs, plus $0.002 for load_async_ai_overview, refunded when no AI Overview loads; pay as you go
cloro $1.92 $250 Starter: $250 for 650,000 credits; 5 credits a search with the AI Overview, 3 without; failed requests are not billed
Oxylabs $1.15 $500 Google JS results: $1.30 per 1,000 up to 76,000 results, $1.15 up to 434,000; plans are $100 and $500 of monthly credit, so 100,000 needs the $500 plan; AI Overview not stated on the page
SerpApi $7.25 $725 Searcher: 100,000 searches; failed searches are not counted; the AI Overview sometimes needs an additional request

At this volume every API in the table costs less than the low DIY case, by $1,700 or more a month.

For the top 100 results, the price changes for everyone. cloro bills 21 credits for 10 pages (3 for the first, 2 for each next page). DataForSEO bills each 10-result SERP. A DIY scraper loads 10 pages.

When does running your own Google scraper make sense?

  • Compliance. Your rules forbid sending queries through a third party.
  • No outbound access. The pipeline runs on-premises or in a VPC that cannot call an external API.
  • Very high volume with cheap bandwidth. In this model, at 0.5 MB a search, Webshare's $1.40 per GB and 10 browser seconds a search, DIY costs $0.98 per 1,000 searches before hours. Against cloro's custom-contract rate of $1.25 per 1,000 searches with the AI Overview, the $0.27 difference pays for the low-case hours ($2,202 a month) at about 8 million searches a month. That assumes the hours do not grow with volume, and they usually do.
  • Your team already runs this stack. If selector and block fixes cost you 2 hours a month and not 20, set the hours to 2 and the DIY case gets much closer.

How this was checked

Proxy, browser, CAPTCHA and API prices come from each vendor's pricing page or docs, read on November 19, 2026, and each one is linked above. Only live rates are shown; queued and batch rates are left out. The pay figure is from BLS. The hours, the MB per search, the browser seconds and the 1.3x multiplier are assumptions, and the table shows where each one goes. The blocking test was 3 queries, sent with requests and with Playwright's headless Chromium from one residential connection on one day: 3 requests and 9 browser requests. It shows what a default setup gets, and it is too small to give a block rate.

This post is published by cloro, which sells a SERP API and is one of the four APIs in the price table. Its rows come from its own pricing page and should be read as self-reported. The poisoned-SERP figures come from cloro's own benchmark.

Top comments (0)