“serp api python” is one of the highest-intent developer searches in web data, and most tutorials answer it with a hello-world and stop — before pagination, before rate discipline, before the geo and device parameters that make results actually correct. This is the tutorial we wanted when we were the one searching: a complete, runnable pattern with the traps marked in the places they bite.
The Setup (Two Minutes, No Ceremony)
Thordata’s SERP API is an HTTP endpoint with a key — no SDK required, though the thordata-sdk package wraps it for Python if you prefer. The direct version, so you can see the shape:
import requests
def serp(query, api_key, location="United States", device="mobile",
engine="google", fmt="json"):
r = requests.post(
"https://api.thordata.com/serp",
json={"q": query, "engine": engine, "location": location,
"device": device, "output": fmt},
headers={"Authorization": f"Bearer {api_key}"},
timeout=30,
)
r.raise_for_status()
return r.json()
One design note before the traps: the location and device defaults are doing real work. Search results vary by geography and device class — a national-desktop scrape tells you about a world that most of your users don’t live in. Pricing context: responses bill per successful request at roughly $0.70–$1.20 per 1,000 by volume, and trial credits exist for the evaluation phase — so the tutorial below is genuinely free to run for a new account.https://www.thordata.com/products/serp-api/?op=mary&from=dev
Trap One: The First Page Is a Lie (Pagination)
SERPs return a page of results, and the page you get is not “the results” — it’s one window. The trap bites in two directions: teams that treat page one as complete, and teams that paginate by naively re-requesting with an incremented start offset and discover inconsistent result sets (results shift between fetches; offsets aren’t stable snapshots).
The working pattern:
def serp_all(query, api_key, location, max_pages=3):
results = []
seen_urls = set() # dedup across pages: offsets drift
for page in range(max_pages):
payload = serp(query, api_key, location=location)
organic = payload.get("organic", [])
if not organic:
break # empty page: stop, don't guess
for item in organic:
if item["url"] not in seen_urls:
seen_urls.add(item["url"])
results.append(item)
return results
The seen_urls set is the quiet correctness fix: cross-page dedup handles the drift that offset-based pagination pretends doesn’t exist.
Trap Two: Parsing the JSON You Were Handed (Don’t)
The second trap is psychological: teams receive clean, structured JSON — organic array, positions, titles, URLs, features — and then write a “parser just to normalize a few things.” Don’t. The structured response is the normalization; every downstream parser is a schema-drift liability the next time the provider updates fields. Read the documented schema, consume it directly, and pin your dependency on the documented field names (raise loudly on absence rather than defaulting to None silently — a missing position should be an event, not a mystery).
def extract_rank(payload, target_domain):
for item in payload.get("organic", []):
if target_domain in item.get("url", ""):
return item["position"]
return None # absent from page one — a fact, not an error
Trap Three: Rate Discipline (Be a Client, Not a Flood)
The third trap is the retry storm: a timeout triggers a retry, the retry triggers the same timeout, and a burst of concurrency “to speed things up” turns a transient failure into a rate-shaped one. The working discipline for per-request-billed APIs:
Sequential by default, concurrent by measurement — parallelize only where you’ve measured the win
Timeouts short and honest (30s), retries few and backed-off (two, exponential)
Failures are data — log the query, the market, the error class; your block-shaped and quota-shaped failures have different signatures and different fixes
Trap Four: The Agent-Shaped Future of This Exact Tutorial
The last trap is temporal: the tutorial above teaches you to drive the API from Python — but if the consumer of your search results is an LLM agent, the modern integration isn’t Python code at all. The Model Context Protocol endpoint exposes the same SERP machinery as agent-discoverable tools: the Thordata MCP server gives Claude Desktop, Cursor, Cline, Windsurf, or any MCP/JSON-RPC runtime live search, engine listings, request history, and usage statistics — with the agent reading the engine schemas and choosing its own parameters. New accounts get 500 free calls and the MCP tier draws on the same response balance (no separate subscription). The same tutorial, two consumption shapes: your code calls serp(), or your agent calls the tool.
The Complete Working Pattern
Assembled — a monitoring-grade loop with every trap handled:
def track_keyword(query, market, api_key, store, max_pages=2):
payload_pages = [serp(query, api_key, location=market)
for _ in range(max_pages)]
organic = dedup_pages(payload_pages) # trap one
record = {
"query": query, "market": market,
"target_rank": extract_rank(payload_pages[0], "yourdomain.com"),
"observed_at": utc_now_iso(), # freshness is a field, not hope
"result_count": len(organic),
}
store.append(record) # schema consumed directly — trap two
return record
Wrap it in a scheduler, store observations as events, and you have the minimum honest rank tracker — geo-pinned, timestamped, deduped, parser-free. Add the MCP endpoint when an agent joins the pipeline, and the same backend serves both consumption shapes.
Top comments (0)