DEV Community: dodou

Async Batch Crawling with a SERP API in Python (asyncio)

dodou — Wed, 02 Sep 2026 14:57:39 +0000

A batch of 500 keywords, crawled sequentially, takes 500 × latency. With asyncio you can overlap those requests and finish the batch in roughly one latency unit — while keeping concurrency under control so you don't hit rate limits. This post shows a clean asyncio pattern for SERP crawling, using httpx (async HTTP) and a semaphore.

The API I'm using is SerpBase (https://api.serpbase.dev): POST JSON, X-API-Key header, structured JSON back.

The async client

httpx is the async HTTP library of choice here. First, the basic single-request function:

import httpx

BASE = "https://api.serpbase.dev"
HEADERS = {"Content-Type": "application/json"}

async def fetch_serp(client: httpx.AsyncClient, api_key: str, keyword: str) -> dict:
    resp = await client.post(
        f"{BASE}/google/search",
        headers={**HEADERS, "X-API-Key": api_key},
        json={"q": keyword, "hl": "en", "gl": "us"},
        timeout=30,
    )
    return resp.json()

Concurrency with a semaphore

You don't want 500 simultaneous requests — that's how you trip QPS limits. An asyncio.Semaphore caps how many requests are in flight:

import asyncio
import httpx

async def crawl_batch(api_key: str, keywords: list[str], max_concurrent: int = 10) -> dict[str, dict]:
    sem = asyncio.Semaphore(max_concurrent)
    results: dict[str, dict] = {}

    async def worker(keyword: str) -> None:
        async with sem:
            data = await fetch_serp(client, api_key, keyword)
            results[keyword] = data

    async with httpx.AsyncClient() as client:
        await asyncio.gather(*(worker(kw) for kw in keywords))

    return results

The semaphore limits in-flight requests to max_concurrent.
asyncio.gather fans out all keywords; the semaphore keeps the actual concurrency bounded.
Start with max_concurrent=10 and tune down if you see rate-limit errors.

Timeout and retry

A robust batch needs per-request timeout plus bounded retries. SerpBase refunds credits for failed dispatches and upstream timeouts, so retrying is safe — no balance burn:

import asyncio
import httpx

async def fetch_with_retry(client, api_key, keyword, max_retries=3):
    for attempt in range(max_retries):
        try:
            data = await fetch_serp(client, api_key, keyword)
            if data.get("status") == 0:
                return data
        except (httpx.TimeoutException, httpx.NetworkError):
            pass
        await asyncio.sleep(0.5 * 2 ** attempt)   # 0.5s, 1s, 2s
    return {"keyword": keyword, "status": -1, "error": "failed"}

The retry is bounded and exponential — individual failures don't block the batch, and a permanently failing keyword is marked rather than retried forever.

Streaming results to disk (keep memory flat)

Batching all results into one dict is fine for hundreds of keywords, but for tens of thousands you want to stream: process each result as it completes, don't accumulate.

async def crawl_and_save(api_key, keywords, out_file, max_concurrent=10):
    sem = asyncio.Semaphore(max_concurrent)

    async def worker(keyword):
        async with sem:
            data = await fetch_with_retry(client, api_key, keyword)
            # append one line per result as it completes
            with open(out_file, "a", encoding="utf-8") as f:
                for r in data.get("organic", []):
                    f.write(f"{keyword}\t{r.get('rank')}\t{r.get('title')}\t{r.get('link')}\n")
        return keyword

    async with httpx.AsyncClient() as client:
        await asyncio.gather(*(worker(kw) for kw in keywords))

Appending to a file per completion keeps peak memory flat regardless of batch size — the "stream, don't accumulate" rule for long-running crawls.

Cancellation and shutdown

Long batches can be interrupted. asyncio.gather + a global timeout gives you an overall cap:

try:
    await asyncio.wait_for(
        asyncio.gather(*(worker(kw) for kw in keywords)),
        timeout=600,   # whole batch capped at 10 minutes
    )
except asyncio.TimeoutError:
    print("batch timed out — rerun for unfinished keywords")

Pair that with a task-state table (pending/done/failed) and re-runs only cover what's missing.

Cost note

/google/search is 1 credit per request. A 500-keyword daily batch is 15k requests/month — a few dollars on a standard pack. Async doesn't change the cost per request; it just finishes the batch much faster.

Wrapping up

The async pattern for SERP crawling is: httpx.AsyncClient for the requests, a Semaphore for concurrency, bounded exponential retries, and stream-to-disk for large batches. Five components, no threads, no callback soup.

The full response schema is in the SerpBase documentation. Start with the semaphore + gather pattern — it turns a 10-minute sequential crawl into a ~1-minute concurrent one.

Type-Safe Google Search Results in TypeScript

dodou — Mon, 31 Aug 2026 08:58:08 +0000

If you're building a Node backend in TypeScript, the last thing you want is any[] spilling out of your search integration. The SERP API I use (SerpBase, https://api.serpbase.dev) returns a well-defined JSON envelope, which means you can type the whole integration and get editor autocomplete + compile-time safety for free.

This post builds a typed TypeScript client: response types, a small client class, and a CLI. Node 18+ with global fetch — no extra dependencies.

The response types

First, model the response. The envelope is consistent across endpoints: status, request_id, elapsed_ms, credits_charged, plus endpoint-specific data.

// serp.ts
interface OrganicResult {
  rank: number;
  position?: number;           // alias for rank, optional
  title: string;
  link: string;
  url?: string;
  display_url?: string;
  snippet?: string;            // optional
  sitelinks?: { title: string; link: string }[];
}

interface SearchResponse {
  status: number;              // 0 = success
  request_id: string;
  elapsed_ms: number;
  credits_charged: number;
  search_type: string;
  query: string;
  organic?: OrganicResult[];
  featured_snippet?: {
    title?: string;
    answer?: string;
    link?: string;
  };
  people_also_ask?: { question: string; answer?: string }[];
  related_searches?: string[];
}

Marking snippet, display_url, position as optional reflects the real schema — those fields only appear when Google renders them. TypeScript's ? is exactly the right tool for "this field may be absent."

The typed client

const BASE = "https://api.serpbase.dev";

export class SerpClient {
  constructor(private apiKey: string) {}

  async search(
    query: string,
    opts: { hl?: string; gl?: string; page?: number; device?: "default" | "pc" | "mobile" } = {},
  ): Promise<SearchResponse> {
    const resp = await fetch(`${BASE}/google/search`, {
      method: "POST",
      headers: { "Content-Type": "application/json", "X-API-Key": this.apiKey },
      body: JSON.stringify({ q: query, ...opts }),
    });

    const data = (await resp.json()) as SearchResponse;
    if (data.status !== 0) {
      throw new Error(`Search failed: ${data.status} (${data.request_id})`);
    }
    return data;
  }
}

The device option is typed as a union, so passing "tablet" is a compile error instead of a runtime surprise.

Using it with autocomplete

Now callers get full typing:

const client = new SerpClient(process.env.SERPBASE_API_KEY!);

const data = await client.search("typescript generics", { hl: "en", gl: "us" });

for (const r of data.organic ?? []) {
  console.log(`${r.rank}. ${r.title}`);
  console.log(`   ${r.link}`);
}

// optional fields are typed as optional — safe to guard
if (data.featured_snippet) {
  console.log("Snippet:", data.featured_snippet.answer);
}

No any, no casts to guess at — the editor knows organic is OrganicResult[] | undefined and nudges you to handle the empty case.

A typed CLI wrapper

Small wrapper to make it a one-liner:

// cli.ts
const [, , query = "serp api"] = process.argv;
const client = new SerpClient(process.env.SERPBASE_API_KEY!);
const data = await client.search(query, { hl: "en", gl: "us" });
for (const r of data.organic ?? []) {
  console.log(`${r.rank}. ${r.title}\n   ${r.link}`);
}

npx tsx cli.ts "typescript generics"

Adding a typed cache layer

Since the response is typed, a typed cache wrapper is straightforward:

export class CachedSerpClient {
  private cache = new Map<string, { data: SearchResponse; expireAt: number }>();

  constructor(
    private inner: SerpClient,
    private ttlMs = 15 * 60 * 1000,
  ) {}

  async search(query: string, opts: Parameters<SerpClient["search"]>[1] = {}) {
    const key = `${opts.gl ?? "us"}:${query}`;
    const hit = this.cache.get(key);
    if (hit && hit.expireAt > Date.now()) return hit.data;

    const data = await this.inner.search(query, opts);
    this.cache.set(key, { data, expireAt: Date.now() + this.ttlMs });
    return data;
  }
}

Same method signature, so swapping in the cache is a one-line change — and the types make sure you don't accidentally use a string where a SearchResponse belongs.

The optional-fields pattern to remember

The schema has fields that appear only when Google shows them (snippet, position, sitelinks, featured_snippet). The TS pattern that matches this:

Mark them optional with ?
Guard with data.field !== undefined or ?? []
Never assume — the type system enforces what the docs state

Cost note

/google/search costs 1 credit per request. A small rank-check job running 50 keywords daily is ~1,500 requests/month — around a dollar or two on a standard pack. The free 100 searches on signup cover your build week.

Wrapping up

A typed integration turns "parse the JSON and hope" into "the compiler checks my field access." The full schema (including all optional modules) is in the SerpBase documentation — type the fields you actually use and let ? carry the "only when Google shows it" semantics.

Google Search Results in Go with a SERP API

dodou — Tue, 25 Aug 2026 22:59:42 +0000

Go doesn't get enough love in the "scrape search results" space — most tutorials are Python or Node. But Go is a genuinely great fit for SERP work: you often want to fire off a batch of queries concurrently, and goroutines make that trivial. This post is the Go version: one request, concurrent batch crawling, JSON parsing with encoding/json, no browser, no HTML parsing.

The request

The API I'm using is SerpBase (https://api.serpbase.dev). Auth is a plain X-API-Key header, every endpoint takes a POST JSON body. The standard library's net/http is all you need:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "os"
)

const baseURL = "https://api.serpbase.dev"

type SearchResult struct {
    Rank  int    `json:"rank"`
    Title string `json:"title"`
    Link  string `json:"link"`
}

type SerpResponse struct {
    Status  int            `json:"status"`
    Organic []SearchResult `json:"organic"`
}

func search(apiKey, query string) (SerpResponse, error) {
    body, _ := json.Marshal(map[string]any{
        "q": query, "hl": "en", "gl": "us",
    })

    req, _ := http.NewRequest(http.MethodPost, baseURL+"/google/search", bytes.NewReader(body))
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-API-Key", apiKey)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return SerpResponse{}, err
    }
    defer resp.Body.Close()

    var data SerpResponse
    if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
        return SerpResponse{}, err
    }
    return data, nil
}

func main() {
    data, err := search(os.Getenv("SERPBASE_API_KEY"), "go concurrency")
    if err != nil {
        panic(err)
    }
    for _, r := range data.Organic {
        fmt.Printf("%d. %s\n   %s\n", r.Rank, r.Title, r.Link)
    }
}

The json struct tags map the response fields directly. Struct-based parsing is one of Go's quiet advantages here — no dynamic dict juggling.

Concurrent batch crawling with goroutines

The payoff of Go shows up when you have a keyword list. Each keyword is an independent HTTP call, so they parallelize cleanly:

func crawlBatch(apiKey string, keywords []string) []SerpResponse {
    results := make([]SerpResponse, len(keywords))
    var wg sync.WaitGroup
    var mu sync.Mutex

    for i, kw := range keywords {
        wg.Add(1)
        go func(i int, kw string) {
            defer wg.Done()
            data, err := search(apiKey, kw)
            if err != nil {
                mu.Lock()
                results[i] = SerpResponse{Status: -1} // mark as failed
                mu.Unlock()
                return
            }
            mu.Lock()
            results[i] = data
            mu.Unlock()
        }(i, kw)
    }

    wg.Wait()
    return results
}

Notes on this pattern:

Results are written by index (not appended), so order is preserved.
A mutex guards the slice because multiple goroutines write it.
A failed request is marked with Status: -1 so the caller can retry just those.

If your QPS matters, wrap the loop with a small semaphore to cap concurrency (a buffered channel works great):

sem := make(chan struct{}, 10) // max 10 concurrent
for i, kw := range keywords {
    sem <- struct{}{} // acquire
    go func(i int, kw string) {
        defer func() { <-sem }() // release
        // ...same search logic
    }(i, kw)
}

Retrying safely

SerpBase refunds credits for failed dispatches and upstream timeouts, so retries don't burn your balance. In Go, a small retry wrapper is easy:

func searchWithRetry(apiKey, query string, maxRetries int) (SerpResponse, error) {
    var data SerpResponse
    var err error
    for attempt := 0; attempt < maxRetries; attempt++ {
        data, err = search(apiKey, query)
        if err == nil && data.Status == 0 {
            return data, nil
        }
        time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)
    }
    return data, fmt.Errorf("failed after %d retries: %w", maxRetries, err)
}

Handling optional fields

The response schema has optional fields (snippet, display_url, position). In Go, use pointers or omitempty-friendly structs so missing fields decode to zero values instead of errors:

type SearchResult struct {
    Rank      int     `json:"rank"`
    Position  *int    `json:"position,omitempty"`  // optional alias
    Title     string  `json:"title"`
    Link      string  `json:"link"`
    Snippet   *string `json:"snippet,omitempty"`   // may be absent
}

A *string field that's absent decodes to nil, which you can check with if r.Snippet != nil.

Cost before you scale

/google/search costs 1 credit per request. A batch of 500 keywords, checked daily, is ~15k requests/month — on a standard pack (~$0.50/1k) that's around $7.50/month. The free 100 searches on signup cover your first build week.

Wrapping up

Go + SERP API is a small, boring, fast stack: standard library HTTP, struct-based JSON, goroutines for concurrency. No cheerio, no puppeteer, no selector maintenance.

The full response schema (including all optional modules like featured_snippet and people_also_ask) is in the SerpBase documentation. Run the single-query version first, then add the concurrent batch — it's the difference between a script and a tool.

Price & Product Intelligence with a SERP API

dodou — Sun, 23 Aug 2026 22:47:09 +0000

If you sell online, your "pricing page" isn't your pricing page — it's the Google SERP. Competitors' product listings, ad slots, and even the knowledge panel all show up when someone searches for your product category. Tracking how that SERP shifts is price intelligence, and it's embarrassingly cheap to build with a SERP API.

This post builds a small price-intelligence monitor: snapshot the SERP for a product keyword, extract the competing listings, and diff it over time.

What price intelligence actually tracks

For an ecommerce brand, three things matter in the SERP:

Organic product listings — who ranks for "wireless earbuds" and where
Shopping ads / product results — which products Google surfaces as paid
Price changes in the listings themselves — snippets often contain prices ("$49.99")

All three come back from a single search request when Google renders them.

The setup

The API I'm using is SerpBase (https://api.serpbase.dev). Its /google/search endpoint returns organic results plus rich modules in one JSON response. Auth is a plain X-API-Key header, POST JSON.

import requests, sqlite3, datetime

API_KEY = "your_api_key"
BASE = "https://api.serpbase.dev"
YOUR_DOMAIN = "yourstore.com"

def snapshot_serp(keyword, gl="us", hl="en"):
    resp = requests.post(
        f"{BASE}/google/search",
        headers={"Content-Type": "application/json", "X-API-Key": API_KEY},
        json={"q": keyword, "hl": hl, "gl": gl},
        timeout=30,
    )
    return resp.json()

data = snapshot_serp("wireless earbuds")
for r in data.get("organic", [])[:5]:
    print(r["rank"], r["title"], "|", r.get("snippet", "")[:60])

Extracting the price signal

Snippets in ecommerce SERPs frequently include prices. A regex pulls them out for a rough price-parity check:

import re

def extract_price(snippet):
    if not snippet:
        return None
    m = re.search(r"[\$€£]\s?\d+(?:[.,]\d{2})?", snippet)
    return m.group(0) if m else None

rows = []
for r in data.get("organic", []):
    rows.append({
        "rank": r["rank"],
        "title": r["title"],
        "link": r["link"],
        "price": extract_price(r.get("snippet")),
        "domain": r.get("display_url", ""),
    })

Now you have a per-keyword list of competitors with any prices Google chose to show. That's your baseline snapshot.

Persisting snapshots for diffs

Price intelligence is a time-series problem: the value is in the diff. Store each snapshot:

con = sqlite3.connect("pi.db")
con.execute("""CREATE TABLE IF NOT EXISTS snapshots (
    keyword TEXT, day TEXT, rank INTEGER,
    title TEXT, link TEXT, price TEXT, domain TEXT,
    PRIMARY KEY (keyword, day, rank, link)
)""")

for r in rows:
    con.execute(
        "INSERT OR REPLACE INTO snapshots VALUES (?, ?, ?, ?, ?, ?, ?)",
        ("wireless earbuds", datetime.date.today().isoformat(),
         r["rank"], r["title"], r["link"], r["price"], r["domain"]),
    )
con.commit()

Alerting on the interesting changes

The signals worth alerting on:

You dropped out of the top N for your own product keyword
A competitor's snippet gained a price (they're running a promo)
New domains appeared in the top 5

today = con.execute(
    "SELECT link, rank, price FROM snapshots WHERE day = ?",
    (datetime.date.today().isoformat(),),
).fetchall()
yesterday = dict(con.execute(
    "SELECT link, rank FROM snapshots WHERE day = ?",
    (datetime.date.today() - datetime.timedelta(days=1),),
).fetchall())

for link, rank, price in today:
    if link and YOUR_DOMAIN in link and rank > 5:
        print(f"ALERT: your listing fell to #{rank}")
    prev = yesterday.get(link)
    if prev is not None and rank < prev:
        print(f"UP: {link} #{prev} -> #{rank}")
    if price and yesterday.get(link) is None and rank <= 5:
        print(f"NEW with price: {link} {price}")

Hook those prints to Slack and you have a promo-detection bot.

Cost of running this

A product catalog of 100 keywords, checked daily:

100 searches/day × 30 = 3,000 searches/month
SerpBase /google/search costs 1 credit each; standard pack ~$0.50/1k → around $1.50/month

The free 100 searches on signup cover your first build week. The cost is low enough that "check daily" isn't a budget decision.

Going further

Add gl markets: the same keyword in de, jp, in gives you cross-market price positioning.
Track shopping modules: when Google renders product/carousel modules, inspect shopping/related fields for sponsored product data.
Combine with Maps: for local retail, /google/maps/search snapshots which stores surface for a category+city pair.

Wrapping up

Price intelligence isn't an enterprise tool problem — it's a "run one search per product keyword, diff daily" problem. The response schema for search (including all optional modules) is documented at serpbase.dev/docs.

Run a snapshot for your own best-selling keyword today. You'll be surprised what the snippet prices reveal about your competitors.

Tracking Google Featured Snippets with a SERP API

dodou — Sun, 23 Aug 2026 01:16:27 +0000

Featured snippets are the "position zero" of Google — the answer card above the organic results. If you own it, you win the click without being #1. If you lose it, your traffic drops overnight.

The annoying part: snippets come and go, and Google doesn't tell you when. This post shows how to track them programmatically with a SERP API, so you get an alert the day your snippet changes.

The key insight

Most search APIs return organic results, but the interesting part for snippet tracking is the featured_snippet module — which is only present when Google actually shows a snippet for that query. That presence/absence is itself the signal you want to watch.

The API I'm using here is SerpBase (https://api.serpbase.dev). Its /google/search endpoint returns featured_snippet alongside organic when Google renders one:

import requests

resp = requests.post(
    "https://api.serpbase.dev/google/search",
    headers={"Content-Type": "application/json", "X-API-Key": "your_api_key"},
    json={"q": "python asyncio", "hl": "en", "gl": "us"},
)
data = resp.json()

snippet = data.get("featured_snippet")
if snippet:
    print("Snippet found:", snippet.get("title"))
    print("Answer:", snippet.get("answer") or snippet.get("snippet"))
    print("Source:", snippet.get("link"))
else:
    print("No snippet for this query right now")

What a snippet response looks like

When present, featured_snippet carries the answer card content:

{
  "title": "asyncio — Asynchronous I/O",
  "answer": "asyncio is a library to write concurrent code using the async/await syntax...",
  "link": "https://docs.python.org/3/library/asyncio.html",
  "source": "docs.python.org"
}

Note that some queries have no snippet at all — a plain "did we win position zero or not" signal, which is exactly what you want to track day over day.

Building the tracker

The pattern: run the query daily, record whether a snippet exists, and alert on changes.

import sqlite3, datetime, requests

API_KEY = "your_api_key"
BASE = "https://api.serpbase.dev"
YOUR_DOMAIN = "example.com"

KEYWORDS = ["python asyncio", "serp api", "google search api"]

con = sqlite3.connect("snippet.db")
con.execute("""CREATE TABLE IF NOT EXISTS snippet_state (
    keyword TEXT, day TEXT,
    snippet_owned INTEGER,           -- 1 = we own the snippet
    snippet_present INTEGER,         -- 1 = someone has a snippet
    snippet_source TEXT,
    PRIMARY KEY (keyword, day)
)""")

def check_snippet(kw):
    resp = requests.post(
        f"{BASE}/google/search",
        headers={"Content-Type": "application/json", "X-API-Key": API_KEY},
        json={"q": kw, "hl": "en", "gl": "us"},
    ).json()
    sn = resp.get("featured_snippet")
    present = 1 if sn else 0
    owned = 1 if (sn and YOUR_DOMAIN in (sn.get("link", ""))) else 0
    return present, owned, (sn or {}).get("link")

for kw in KEYWORDS:
    present, owned, source = check_snippet(kw)
    con.execute(
        "INSERT INTO snippet_state VALUES (?, ?, ?, ?, ?)",
        (kw, datetime.date.today().isoformat(), owned, present, source),
    )
con.commit()

Alerting on change

The value comes from the diff: today vs yesterday.

today = dict(con.execute(
    "SELECT keyword, snippet_owned FROM snippet_state WHERE day = ?",
    (datetime.date.today().isoformat(),),
).fetchall())

yesterday = dict(con.execute(
    "SELECT keyword, snippet_owned FROM snippet_state WHERE day = ?",
    (datetime.date.today() - datetime.timedelta(days=1),),
).fetchall())

for kw, owned_now in today.items():
    owned_prev = yesterday.get(kw)
    if owned_prev is not None and owned_now != owned_prev:
        action = "WON" if owned_now else "LOST"
        print(f"[{action}] {kw} featured snippet — notify the team")   # hook to Slack here

Now you know the day you lost (or won) position zero, instead of discovering it in next month's traffic report.

Beyond snippets: the same response has more

The featured_snippet module is one of several that come back when Google renders them:

people_also_ask — the PAA box questions, useful for content gaps
related_searches — keyword expansion material
knowledge_graph — entity panel data

All in the same response, no extra requests.

Cost note

/google/search costs 1 credit per request. Tracking 20 keywords daily = 600 requests/month, which on a standard pack (~$0.50/1k) is around $0.30/month. The free 100 searches on signup cover your first test week, and there's no monthly fee to carry while you build.

Wrapping up

Featured snippet tracking is just "run the query, read the featured_snippet module, diff day over day." Ten lines of code, and you finally get notified when position zero changes hands.

The full response schema (including all optional modules) is in the SerpBase documentation. Run the snippet check against your own keywords and see who owns position zero today.

Get Google Search Results in Node.js with a SERP API

dodou — Fri, 21 Aug 2026 13:14:30 +0000

Most "scrape Google" tutorials are Python. Fine, but half the products I touch are Node. This post is the Node.js version: one POST request, clean JSON back, no cheerio, no puppeteer.

Why not scrape Google's HTML directly? Because you'll spend your time fighting CAPTCHAs, consent walls, and weekly markup changes instead of building. A SERP API hands you the parsed results so your code stays boring and stable.

The request

The API I'm using here is SerpBase (https://api.serpbase.dev). Auth is a plain X-API-Key header, every endpoint takes a POST JSON body, and it runs Google Search, Images, News, Videos, and Maps behind one key. Node 18+ has global fetch, so there's no dependency to install:

const API_KEY = "your_api_key";   // from https://serpbase.dev/register
const BASE = "https://api.serpbase.dev";

async function googleSearch(query, { hl = "en", gl = "us", page = 1 } = {}) {
  const resp = await fetch(`${BASE}/google/search`, {
    method: "POST",
    headers: { "Content-Type": "application/json", "X-API-Key": API_KEY },
    body: JSON.stringify({ q: query, hl, gl, page, device: "default" }),
  });
  if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
  return resp.json();
}

const data = await googleSearch("nodejs async");
console.log(data.organic?.[0]);

That's the whole call. Two headers, a JSON body, and you get the SERP back.

What comes back

Every SerpBase endpoint returns the same top-level envelope, so logging and error handling stay consistent:

{
  "status": 0,
  "request_id": "req_01Hxxxx",
  "elapsed_ms": 1420,
  "credits_charged": 1,
  "search_type": "search",
  "query": "nodejs async",
  "page": 1,
  "organic": [
    {
      "rank": 1,
      "title": "Node.js v22.x Documentation: Async Hooks",
      "link": "https://nodejs.org/api/async_hooks.html",
      "display_url": "nodejs.org",
      "snippet": "The node:async_hooks module provides an API..."
    }
  ]
}

Key fields:

status — 0 means success; anything else is an error, also returned as JSON.
elapsed_ms — gateway latency, handy for logging.
credits_charged — what the request cost after refund logic.
organic — the results array with rank, title, link, display_url, snippet.

When Google shows them, the response also carries featured_snippet, people_also_ask, knowledge_graph, and related_searches — all already parsed.

Mapping it to a clean array

A common task: "give me the top 10 results as a plain list." One map does it:

const top10 = (data.organic ?? []).slice(0, 10).map((r) => ({
  rank: r.rank,
  title: r.title,
  url: r.link,
  snippet: r.snippet ?? "",
}));

console.table(top10);

Handling the fields that are optional

Not every result has every field — snippet, display_url, and sitelinks can be absent. Always use ?. and ?? rather than assuming:

const first = data.organic?.[0];
if (first) {
  const snippet = first.snippet ?? "(no snippet)";
  const sitelinks = first.sitelinks?.length ?? 0;
  console.log(`${first.title} — ${sitelinks} sitelinks`);
}

This keeps the parser from throwing on a lean response.

Cost before you commit

SERP requests are metered per successful call. On SerpBase, /google/search, /google/news, and /google/videos cost 1 credit each; /google/images and both Maps endpoints cost 2. Failed dispatches and upstream timeouts are refunded automatically, so retries don't silently drain your balance.

For a side project, the pricing is refreshing: 100 free searches on signup (no card), and a $3 Starter Boost gets you 10,000 searches at the lowest rate. Standard packs start at $10 for 20k and never expire.

A tiny CLI wrapper

To make it reusable, wrap it in a one-liner CLI:

// search.mjs
const [,, query = "serp api"] = process.argv;
const data = await googleSearch(query);
for (const r of data.organic ?? []) {
  console.log(`${r.rank}. ${r.title}\n   ${r.link}`);
}

node search.mjs "nodejs async"

Where to go next

Swap hl/gl to simulate other markets: { hl: "de", gl: "de" } returns German Google.
Read the endpoint reference in the SerpBase documentation for Images, News, Videos, and Maps.
If you want the agent route, SerpBase also ships an MCP server and a portable skill for Claude, Codex, Cursor, and opencode.

Run one request against your own key and inspect organic — the whole loop is shorter than the cheerio selector you were about to write.

Scrape Google Search Results with a SERP API in Python

dodou — Tue, 18 Aug 2026 07:19:04 +0000

A while back I needed Google search data for a side project. My first instinct was scraping the HTML with BeautifulSoup. That worked for exactly one afternoon, then the block pages started. Between CAPTCHAs, consent walls, and the fact that Google's markup changes weekly, I gave up on the DIY route and looked for a SERP API.

This post shows the minimal setup that works: one POST request, one JSON response, no browser.

Why not just scrape

Scraping Google's HTML directly is a losing game because:

IP-based rate limiting kicks in after a handful of requests
Google serves different markup to different devices and sessions
Consent and CAPTCHA pages return 200 with zero useful data
You spend your time maintaining selectors, not building your product

A SERP API solves the "maintain the scraper" part. You send a query, get structured JSON back.

The request

SerpBase is the one I'm using here — it runs Google Search, Images, News, Videos, and Maps behind a single endpoint. The base URL is https://api.serpbase.dev, auth is a plain X-API-Key header, and every endpoint accepts a POST JSON body.

Here's the whole thing in Python with only requests:

import requests

API_KEY = "your_api_key"   # from https://serpbase.dev/register
BASE_URL = "https://api.serpbase.dev"

resp = requests.post(
    f"{BASE_URL}/google/search",
    headers={
        "Content-Type": "application/json",
        "X-API-Key": API_KEY,
    },
    json={
        "q": "python asyncio",
        "hl": "en",
        "gl": "us",
        "page": 1,
        "device": "default",
    },
    timeout=30,
)
resp.raise_for_status()
data = resp.json()

That's it. Two headers, a JSON body with q plus optional hl/gl/page/device, and you get the SERP back.

What comes back

Every SerpBase endpoint returns the same top-level envelope, so your error handling and logging stay consistent across endpoints:

{
  "status": 0,
  "request_id": "req_01Hxxxx",
  "elapsed_ms": 1420,
  "credits_charged": 1,
  "search_type": "search",
  "query": "python asyncio",
  "page": 1,
  "organic": [
    {
      "rank": 1,
      "title": "asyncio — Asynchronous I/O — Python 3.x documentation",
      "link": "https://docs.python.org/3/library/asyncio.html",
      "display_url": "docs.python.org",
      "snippet": "Asynchronous programming with the async/await syntax..."
    }
  ]
}

Useful fields to know:

status — 0 means success. Everything else is an error, also returned as JSON.
elapsed_ms — gateway latency, handy for logging and alerting.
credits_charged — what this request cost after refund logic is applied.
organic — the results array, each item with rank, title, link, display_url, snippet, and optional sitelinks.

When Google shows them, the same response also carries featured_snippet, people_also_ask, top_stories, knowledge_graph, and related_searches — all parsed, none of your parser code needed.

Pulling out the top 10 links

A common task is "give me the top N results as a clean list". With the envelope above:

results = []
for item in data.get("organic", []):
    results.append(
        {
            "rank": item["rank"],
            "title": item.get("title", ""),
            "url": item.get("link", ""),
            "snippet": item.get("snippet", ""),
        }
    )

for r in results[:10]:
    print(f"{r['rank']:>2}. {r['title']} — {r['url']}")

Checking cost before you build

SERP requests are metered per successful call. On SerpBase, /google/search, /google/news, and /google/videos cost 1 credit each; /google/images and both Maps endpoints cost 2 credits. Failed dispatches and upstream timeouts get refunded automatically, so you're not paying for retries that didn't deliver.

For a hobby budget, the pricing is refreshing: 100 free searches on signup (no card), and a $3 Starter Boost gets you 10,000 searches at the lowest rate. Standard credit packs start at $10 for 20k searches and never expire. Full details are on the pricing page.

Where to go next

Run the snippet above against your own key and print the organic array — that's the whole loop.
Read the endpoint reference in the SerpBase documentation for the Images, News, Videos, and Maps endpoints.
If you want the agent-style route, SerpBase also ships an MCP server and a portable skill for Claude, Codex, Cursor, and opencode.

The fastest way to decide whether this fits is to run one request and look at the JSON. The whole pipeline is shorter than the CSS selector you were about to write.

Video Keyword Research With the /google/videos Endpoint

dodou — Mon, 17 Aug 2026 03:19:17 +0000

The Google Videos tab is a keyword mine almost nobody works. Video keyword research tools exist, but they're one more subscription. The raw source is the Videos tab itself, and one POST returns it as JSON. This post builds a small video keyword research script on /google/videos at 1 credit per call, then turns the output into a YouTube content calendar.

The pattern: for each candidate keyword, pull the video results, keep the title, source, and duration, and bucket them by how competitive the niche looks. If three big channels from the same network own the first page, the keyword is hard. If the videos are old or from small channels, there is room.

The request

/google/videos takes the same envelope as the rest of the API. POST with the X-API-Key header to https://api.serpbase.dev/google/videos:

curl -X POST "https://api.serpbase.dev/google/videos" \
  -H "X-API-Key: $SERPBASE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q": "raspberry pi home assistant", "hl": "en", "gl": "us", "num": 10}'

The response includes a videos array with title, link, source, duration, and thumbnail per result, wrapped in the standard envelope: status, request_id, elapsed_ms, credits_charged.

The script

A keyword file, one keyword per line, then this:

import json, requests

def video_serp(keyword: str, api_key: str) -> list:
    r = requests.post(
        "https://api.serpbase.dev/google/videos",
        headers={"X-API-Key": api_key, "Content-Type": "application/json"},
        json={"q": keyword, "hl": "en", "gl": "us", "num": 10},
        timeout=10,
    )
    r.raise_for_status()
    return r.json().get("videos", [])

def score(keyword: str, videos: list) -> dict:
    channels = {v["source"] for v in videos}
    avg_dur = sum(_parse_seconds(v.get("duration", "0:00")) for v in videos) / max(len(videos), 1)
    return {
        "keyword": keyword,
        "result_count": len(videos),
        "unique_channels": len(channels),
        "avg_duration_s": int(avg_dur),
        "tough": len(channels) <= 2 and len(videos) >= 8,
    }

def _parse_seconds(d: str) -> int:
    parts = d.split(":")
    if len(parts) == 2:
        return int(parts[0]) * 60 + int(parts[1])
    if len(parts) == 3:
        return int(parts[0]) * 3600 + int(parts[1]) * 60 + int(parts[2])
    return 0

For every keyword the script writes one row: result count, unique channels, average duration, and a tough flag when the same two channels own the whole first page. That flag is the "hard keyword, skip it" signal.

The content calendar

Sort the scored keywords by tough=False and by avg_duration_s. Short average duration on a competitive-looking keyword means people watch short explainers — a scripted 5-minute video has a shot. Long average duration with few unique channels means the niche rewards depth and is already owned; write an article instead.

A week of topic ideas, computed in a single script run: 200 keywords × 1 call = 200 credits. Weekly runs for a month = 800 calls ≈ $0.24 on the Starter Boost pack ($3 / 10,000 searches, expires one month after purchase, once per account per month). At daily runs (6,000 calls/month) the Growth tier ($50 / 125,000 searches, credits never expire) costs about $2.40/month.

What this gives you that a manual search cannot

A manual Videos-tab search shows you the top of the page. The API gives you the same page as JSON you can diff week over week: when a keyword's first-page channels change, that is a content opportunity appearing in real time. The elapsed_ms and request_id fields keep the loop auditable — the same envelope pattern across /google/search, /google/news, and /google/videos at 1 credit each. That consistency is why the pipeline stays at ~30 lines instead of turning into a scraping project.

Honest limits

duration is parsed from Google's compact metadata, so a few results can miss it. The _parse_seconds fallback returns 0 for those.
hl/gl change which videos and channels show. Fix one locale per content strategy.
The tough heuristic is a first pass, not a verdict. Two dominant channels with 8+ results is a strong smell, but check the actual videos before dropping a keyword.

Source: SERP API comparison, updated Apr 23, 2026.

Full parameter and response reference: serpbase.dev/docs.

The $450 Headless Chrome Bill That Made Me Try a SERP API

dodou — Fri, 14 Aug 2026 00:47:14 +0000

I spent the last quarter scraping Google with headless Chrome. I run a small price-monitoring service for a few e-commerce clients, and the original plan was to use Playwright on one server and keep the monthly cost near zero. The API route felt like paying for something I could build myself. Ninety days later the infrastructure bill hit $450 and I still couldn't reliably get a SERP back.

It started small. One VPS, a few cron jobs, Playwright with a real browser profile. Week one was fine. Week two, Google started showing the consent wall to datacenter IPs. Week three, one of the client's keywords came back with a CAPTCHA about a third of the time. By the end of the first month I'd bought a proxy pool, added a CAPTCHA solver, and was still losing requests.

Here's where the money actually went.

The month-by-month breakdown

Month	What broke	What I added	Cost
1	consent walls, intermittent blocks	rotating proxies	$120
2	CAPTCHA rate climbed to ~20%	solver + retry loop	$80
3	Google changed the SERP layout	parser rewrite, bigger server	$250
Total			$450

Month three was the worst. Google changed the results page layout, my CSS selectors all broke, and I spent two weeks rewriting the parser while the proxy bill kept running. That's the $250 month.

The server line is sneaky. Every one of those items is a fixed cost that keeps charging whether you succeed or not. My best week was 87% success. That means 13 out of every 100 checks came back empty, which meant retries, which meant more proxy traffic, which meant more money.

What the switch looked like

I kept expecting the API route to be painful. It wasn't. One POST to /google/search with an X-API-Key header, and you get the SERP back as JSON. No browser, no proxy rotation, no consent walls.

curl -X POST "https://api.serpbase.dev/google/search" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $SERPBASE_API_KEY" \
  -d '{"q": "wireless earbuds", "hl": "en", "gl": "us"}'

The response has an organic array with rank, title, link, and snippet per result — the exact fields my rank checker needed. It also includes people_also_ask, knowledge_graph, and ai_overview when Google shows them, which my headless parser was ignoring anyway.

The migration was a 40-line diff. The old code parsed HTML selectors that broke every few weeks. The new code reads a JSON key. That's the entire difference.

import requests

def check_rank(keyword, key):
    r = requests.post(
        "https://api.serpbase.dev/google/search",
        headers={"X-API-Key": key, "Content-Type": "application/json"},
        json={"q": keyword, "hl": "en", "gl": "us"},
    )
    r.raise_for_status()
    organic = r.json().get("organic", [])
    return [(i + 1, item["title"], item["link"])
            for i, item in enumerate(organic)]

print(check_rank("wireless earbuds", "your_api_key"))

The cost math

My workload is about 100,000 searches a month — a few thousand keywords checked daily, plus price checks. On the Growth pack that's $50 for 125,000 searches at $0.40/1k, so around $40/month. On Pro it drops to about $35/month at $0.35/1k. Credits don't expire, which matters when a client pauses and I stop querying for two weeks — the money stays.

Search, news, and videos are 1 credit per call; images and both Maps endpoints are 2. Failed dispatches get refunded automatically, so a timeout doesn't cost me anything. That single rule is worth more than the price difference. Under headless Chrome, a timeout cost me proxy bandwidth and retry time. Here it costs nothing.

P50 latency is about 1.4 seconds and the SLA is 99.9%. I don't have to think about the infrastructure anymore, and that's the line I never budgeted for before: the hours.

Where headless Chrome still wins

The browser still has a place. If you need the raw DOM — parsing a widget Google only renders for certain sessions, or testing against the exact page your scraper targets — headless is the right tool. If your setup already handles captchas and layout drift and you're happy with the maintenance, there's no reason to rip it out.

The math only tips when you'd rather spend the hours on your product than on keeping a scraper alive.

What I'd do differently

If I were starting over, I'd skip the browser stage entirely. Buy 100 searches on the free signup to test the response shape, then size a pack to actual volume. The entry pack is $3 for 10,000 searches, expiring one month after purchase, available once per account per month. I'll never discount maintenance hours again.

Source: SERP API comparison, updated Apr 23, 2026.

Full parameter and response reference: serpbase.dev.

Verify LLM Claims With a $3 Search Budget

dodou — Mon, 10 Aug 2026 04:04:05 +0000

An LLM confidently stating a wrong fact is worse than a blank answer — it sounds the same as a right one. The cheapest reliable fix is a search-backed verification pass: extract the claims, search each one, and mark unsupported claims.

At 1 credit per search, a verification pass is cheap. SerpBase's Starter Boost is $3 for 10,000 searches — enough for thousands of claim checks.

The verification loop

import requests

API = "https://api.serpbase.dev"
KEY = "your_api_key"

def verify_claim(claim):
    """Search a claim, return whether the top results support it."""
    r = requests.post(
        f"{API}/google/search",
        headers={"X-API-Key": KEY},
        json={"q": claim, "hl": "en", "gl": "us"},
        timeout=10,
    )
    r.raise_for_status()
    organic = r.json().get("organic", [])
    if not organic:
        return {"claim": claim, "supported": False, "reason": "no results"}
    top = organic[0]
    return {
        "claim": claim,
        "supported": True,
        "top_title": top.get("title"),
        "top_link": top.get("link"),
    }

How to extract claims

Ask the LLM to enumerate its own factual claims before verification:

List the factual claims in your answer, one per line, as standalone search queries.

Each claim becomes a query. Run the loop, and any claim whose top result doesn't relate to it gets flagged.

What this catches

Fabricated statistics ("revenue grew 23% last year" with no source).
Confused entities ("the CEO of X is Y" when Y left in 2024).
Stale facts that changed since the training cut-off.

It won't catch subtle misreads, but it removes the loudest failures.

Cost

A 10-claim verification = 10 searches. At 1 credit each, the $3 Starter Boost covers ~10,000 claim checks. Even at full Starter rate, verification is a rounding error next to LLM inference tokens.

Honest limits

A top result is not proof; it's evidence. For high-stakes claims, pull a few results and read snippets, not just the top link.
Verification adds latency. Run it async for non-interactive pipelines.
The model can game the pass if you let it pick its own queries; a fixed extractor is more reliable.

Full parameter and response reference: serpbase.dev/docs.

Localized Rank Tracking: hl + gl for 200+ Countries

dodou — Sun, 09 Aug 2026 01:01:50 +0000

Rank tracking gets complicated when your keyword is "coffee" and your customers are in Shanghai, London, and Tokyo. The same query ranks differently per country and language. That's what hl and gl are for.

SerpBase supports 200+ countries via hl (interface language) + gl (country), plus lat/lng/zoom for Maps. This post shows a rank tracker that monitors one keyword across several locales.

The code

import requests
from datetime import datetime

API = "https://api.serpbase.dev"
KEY = "your_api_key"

LOCALES = [
    ("zh-CN", "cn"),
    ("en", "us"),
    ("ja", "jp"),
    ("ko", "kr"),
    ("de", "de"),
    ("fr", "fr"),
]

def track(query, locales=LOCALES):
    rows = []
    for hl, gl in locales:
        r = requests.post(
            f"{API}/google/search",
            headers={"X-API-Key": KEY},
            json={"q": query, "hl": hl, "gl": gl},
            timeout=10,
        )
        r.raise_for_status()
        data = r.json()
        organic = data.get("organic", [])
        rows.append({
            "query": query,
            "hl": hl,
            "gl": gl,
            "rank": organic[0].get("rank") if organic else None,
            "top_link": organic[0].get("link") if organic else None,
            "request_id": data.get("request_id"),
        })
    return rows

print(track("coffee beans"))

What you get per locale

For each hl/gl, the response includes organic with rank per result. Tracking the same keyword across locales shows which markets you rank for and which you don't.

Cost

6 locales × 100 keywords × 1x/day = 600 searches/day. At the Growth rate ($0.40/1k), that's about $7/month. Multi-market monitoring is cheap because each locale is one 1-credit search.

Honest caveats

rank varies by location AND language. Track the same hl/gl pair consistently, or numbers aren't comparable.
Localized results change more than a single-locale view; daily polling is usually enough.
For local (Maps) ranking, use the Maps endpoints with lat/lng/zoom — search alone covers organic only.

Full parameter and response reference: serpbase.dev/docs.

Wiring a SERP API Into Claude/Cursor via MCP

dodou — Sat, 08 Aug 2026 08:14:03 +0000

When an agent needs fresh Google data, handing it a raw curl instruction makes the agent guess at API details. Model Context Protocol (MCP) fixes that: the search becomes a tool the agent calls with structured arguments.

SerpBase ships an MCP server that exposes all six endpoints as agent tools. Here's how to wire it in.

Install and configure

pip install serpbase-mcp
export SERPBASE_API_KEY=your_api_key

Point an MCP-capable client at it:

{
  "mcpServers": {
    "serpbase": {
      "command": "python",
      "args": ["-m", "serpbase_mcp"],
      "env": { "SERPBASE_API_KEY": "your_api_key" }
    }
  }
}

Claude Desktop, Cursor, and Codex all pick up this config. The tools appear in the agent's tool list.

What the agent sees

Each endpoint is a tool with an input schema. /google/search takes query (required) plus optional hl, gl, page, device. The result is structured — organic with rank, title, link, snippet per result — so the agent can cite a rank and a link instead of guessing from text.

A grounded flow

Agent gets a question needing current data.
It calls google_search with the query.
The tool returns organic results.
The agent summarizes, citing links.
Optional verify step: check each claimed fact maps to a returned snippet.

Credits, honestly

Every successful tool call costs credits: 1 for search/news/videos, 2 for images and the Maps endpoints. If the agent rephrases the same query a few times, costs add up. Two mitigations:

Instruct the agent to reuse a result instead of re-searching the same intent.
Cache the tool result per session for identical queries.

When not to use MCP

MCP is great for interactive desktop/IDE agents. For a server-side pipeline that just needs results, call the HTTP API directly — simpler, no transport layer.

Takeaway

An MCP server turns SERP data into a first-class agent tool: structured input, structured output, one config. SerpBase ships one so you don't hand-roll the glue. Full details: serpbase.dev/docs.