Async Batch Crawling with a SERP API in Python (asyncio)

A batch of 500 keywords, crawled sequentially, takes 500 × latency. With asyncio you can overlap those requests and finish the batch in roughly one latency unit — while keeping concurrency under control so you don't hit rate limits. This post shows a clean asyncio pattern for SERP crawling, using httpx (async HTTP) and a semaphore.

The API I'm using is SerpBase (https://api.serpbase.dev): POST JSON, X-API-Key header, structured JSON back.

The async client

httpx is the async HTTP library of choice here. First, the basic single-request function:

import httpx

BASE = "https://api.serpbase.dev"
HEADERS = {"Content-Type": "application/json"}

async def fetch_serp(client: httpx.AsyncClient, api_key: str, keyword: str) -> dict:
    resp = await client.post(
        f"{BASE}/google/search",
        headers={**HEADERS, "X-API-Key": api_key},
        json={"q": keyword, "hl": "en", "gl": "us"},
        timeout=30,
    )
    return resp.json()

Concurrency with a semaphore

You don't want 500 simultaneous requests — that's how you trip QPS limits. An asyncio.Semaphore caps how many requests are in flight:

import asyncio
import httpx

async def crawl_batch(api_key: str, keywords: list[str], max_concurrent: int = 10) -> dict[str, dict]:
    sem = asyncio.Semaphore(max_concurrent)
    results: dict[str, dict] = {}

    async def worker(keyword: str) -> None:
        async with sem:
            data = await fetch_serp(client, api_key, keyword)
            results[keyword] = data

    async with httpx.AsyncClient() as client:
        await asyncio.gather(*(worker(kw) for kw in keywords))

    return results

The semaphore limits in-flight requests to max_concurrent.
asyncio.gather fans out all keywords; the semaphore keeps the actual concurrency bounded.
Start with max_concurrent=10 and tune down if you see rate-limit errors.

Timeout and retry

A robust batch needs per-request timeout plus bounded retries. SerpBase refunds credits for failed dispatches and upstream timeouts, so retrying is safe — no balance burn:

import asyncio
import httpx

async def fetch_with_retry(client, api_key, keyword, max_retries=3):
    for attempt in range(max_retries):
        try:
            data = await fetch_serp(client, api_key, keyword)
            if data.get("status") == 0:
                return data
        except (httpx.TimeoutException, httpx.NetworkError):
            pass
        await asyncio.sleep(0.5 * 2 ** attempt)   # 0.5s, 1s, 2s
    return {"keyword": keyword, "status": -1, "error": "failed"}

The retry is bounded and exponential — individual failures don't block the batch, and a permanently failing keyword is marked rather than retried forever.

Streaming results to disk (keep memory flat)

Batching all results into one dict is fine for hundreds of keywords, but for tens of thousands you want to stream: process each result as it completes, don't accumulate.

async def crawl_and_save(api_key, keywords, out_file, max_concurrent=10):
    sem = asyncio.Semaphore(max_concurrent)

    async def worker(keyword):
        async with sem:
            data = await fetch_with_retry(client, api_key, keyword)
            # append one line per result as it completes
            with open(out_file, "a", encoding="utf-8") as f:
                for r in data.get("organic", []):
                    f.write(f"{keyword}\t{r.get('rank')}\t{r.get('title')}\t{r.get('link')}\n")
        return keyword

    async with httpx.AsyncClient() as client:
        await asyncio.gather(*(worker(kw) for kw in keywords))

Appending to a file per completion keeps peak memory flat regardless of batch size — the "stream, don't accumulate" rule for long-running crawls.

Cancellation and shutdown

Long batches can be interrupted. asyncio.gather + a global timeout gives you an overall cap:

try:
    await asyncio.wait_for(
        asyncio.gather(*(worker(kw) for kw in keywords)),
        timeout=600,   # whole batch capped at 10 minutes
    )
except asyncio.TimeoutError:
    print("batch timed out — rerun for unfinished keywords")

Pair that with a task-state table (pending/done/failed) and re-runs only cover what's missing.

Cost note

/google/search is 1 credit per request. A 500-keyword daily batch is 15k requests/month — a few dollars on a standard pack. Async doesn't change the cost per request; it just finishes the batch much faster.

Wrapping up

The async pattern for SERP crawling is: httpx.AsyncClient for the requests, a Semaphore for concurrency, bounded exponential retries, and stream-to-disk for large batches. Five components, no threads, no callback soup.

The full response schema is in the SerpBase documentation. Start with the semaphore + gather pattern — it turns a 10-minute sequential crawl into a ~1-minute concurrent one.

DEV Community