DEV Community

dodou
dodou

Posted on

Scrape Google Search Results with a SERP API in Python

A while back I needed Google search data for a side project. My first instinct was scraping the HTML with BeautifulSoup. That worked for exactly one afternoon, then the block pages started. Between CAPTCHAs, consent walls, and the fact that Google's markup changes weekly, I gave up on the DIY route and looked for a SERP API.

This post shows the minimal setup that works: one POST request, one JSON response, no browser.

Why not just scrape

Scraping Google's HTML directly is a losing game because:

  • IP-based rate limiting kicks in after a handful of requests
  • Google serves different markup to different devices and sessions
  • Consent and CAPTCHA pages return 200 with zero useful data
  • You spend your time maintaining selectors, not building your product

A SERP API solves the "maintain the scraper" part. You send a query, get structured JSON back.

The request

SerpBase is the one I'm using here — it runs Google Search, Images, News, Videos, and Maps behind a single endpoint. The base URL is https://api.serpbase.dev, auth is a plain X-API-Key header, and every endpoint accepts a POST JSON body.

Here's the whole thing in Python with only requests:

import requests

API_KEY = "your_api_key"   # from https://serpbase.dev/register
BASE_URL = "https://api.serpbase.dev"

resp = requests.post(
    f"{BASE_URL}/google/search",
    headers={
        "Content-Type": "application/json",
        "X-API-Key": API_KEY,
    },
    json={
        "q": "python asyncio",
        "hl": "en",
        "gl": "us",
        "page": 1,
        "device": "default",
    },
    timeout=30,
)
resp.raise_for_status()
data = resp.json()
Enter fullscreen mode Exit fullscreen mode

That's it. Two headers, a JSON body with q plus optional hl/gl/page/device, and you get the SERP back.

What comes back

Every SerpBase endpoint returns the same top-level envelope, so your error handling and logging stay consistent across endpoints:

{
  "status": 0,
  "request_id": "req_01Hxxxx",
  "elapsed_ms": 1420,
  "credits_charged": 1,
  "search_type": "search",
  "query": "python asyncio",
  "page": 1,
  "organic": [
    {
      "rank": 1,
      "title": "asyncio — Asynchronous I/O — Python 3.x documentation",
      "link": "https://docs.python.org/3/library/asyncio.html",
      "display_url": "docs.python.org",
      "snippet": "Asynchronous programming with the async/await syntax..."
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Useful fields to know:

  • status — 0 means success. Everything else is an error, also returned as JSON.
  • elapsed_ms — gateway latency, handy for logging and alerting.
  • credits_charged — what this request cost after refund logic is applied.
  • organic — the results array, each item with rank, title, link, display_url, snippet, and optional sitelinks.

When Google shows them, the same response also carries featured_snippet, people_also_ask, top_stories, knowledge_graph, and related_searches — all parsed, none of your parser code needed.

Pulling out the top 10 links

A common task is "give me the top N results as a clean list". With the envelope above:

results = []
for item in data.get("organic", []):
    results.append(
        {
            "rank": item["rank"],
            "title": item.get("title", ""),
            "url": item.get("link", ""),
            "snippet": item.get("snippet", ""),
        }
    )

for r in results[:10]:
    print(f"{r['rank']:>2}. {r['title']} — {r['url']}")
Enter fullscreen mode Exit fullscreen mode

Checking cost before you build

SERP requests are metered per successful call. On SerpBase, /google/search, /google/news, and /google/videos cost 1 credit each; /google/images and both Maps endpoints cost 2 credits. Failed dispatches and upstream timeouts get refunded automatically, so you're not paying for retries that didn't deliver.

For a hobby budget, the pricing is refreshing: 100 free searches on signup (no card), and a $3 Starter Boost gets you 10,000 searches at the lowest rate. Standard credit packs start at $10 for 20k searches and never expire. Full details are on the pricing page.

Where to go next

  • Run the snippet above against your own key and print the organic array — that's the whole loop.
  • Read the endpoint reference in the SerpBase documentation for the Images, News, Videos, and Maps endpoints.
  • If you want the agent-style route, SerpBase also ships an MCP server and a portable skill for Claude, Codex, Cursor, and opencode.

The fastest way to decide whether this fits is to run one request and look at the JSON. The whole pipeline is shorter than the CSS selector you were about to write.

Top comments (0)