Your data pipeline may not be "down",but it is quietly accumulating compliance debt

Marysha — Thu, 05 Nov 2026 08:24:56 +0000

From conversations with many teams operating long-running data pipelines,we've observed a rarely discussed trend:

Few teams check their IP pool health while their pipelines are still functional.Troubleshooting only kicks in when scraping success rates plummet or target sites block an entire IP subnet with challenge pages. By that point, the problem may have been accumulating for weeks.

This issue cannot be reliably detected with a one-time benchmark. It resembles gradual reputational decay:a handful of IPs get flagged by targets for abnormal activity. Left undetected and unquarantined,they keep cycling within the pool and tainting subsequent requests.When your monitoring fires an alert,the damage footprint is likely much larger than the initial bad IPs.

Internally,we call this state compliance debt.While the pipeline continues to produce data outwardly,the health and auditability of the underlying IP assets are being eroded.

Below are some signals worth adding to your regular inspection checklist.If your pipeline has been operating for some time,feel free to review against these.

Gradual,modest success-rate decay,not an abrupt collapse. Sharp drops tend to have an obvious trigger.Slow drift is too often dismissed as target-site updates,hiding underlying IP degradation.

Concentrated failures tied to a particular ASN or geo block.This signals the problem is not uniform but confined to one resource slice,which your rotation logic may not be quarantining.

Captcha and challenge page frequency climbs for long sessions, even as short sessions work fine.This subtle sign is frequently overlooked. It means the target is building models around how session identity ties to identity—not just blocking single requests.

These are not problems you can resolve by swithing proxies. The root cause is not per-request behavior,but sustained health governance of your resource pool.

This is why we built Thordata around compliance as a foundational design principle, not a bolt-on fix. When IP sourcing, rotation rules and isolation controls aren’t built into the stack from the start, retrofitted compliancechecks can only detect contamination after the fact — they cannot prevent bad IPs from getting into the pool.

For teams operating long-term data pipelines: Which signal do you typically notice first before pipeline reliability worsens? Have you ever run an investigation and discovered the issue wasn’t originating from the target site, but your own IP fleet?https://www.thordata.com/solutions/video-data-use-cases?op=mary&from=dev

SERP API in Python: A Working Developer’s Tutorial (With the Traps Marked)

Marysha — Tue, 03 Nov 2026 06:49:26 +0000

“serp api python” is one of the highest-intent developer searches in web data, and most tutorials answer it with a hello-world and stop — before pagination, before rate discipline, before the geo and device parameters that make results actually correct. This is the tutorial we wanted when we were the one searching: a complete, runnable pattern with the traps marked in the places they bite.

The Setup (Two Minutes, No Ceremony)
Thordata’s SERP API is an HTTP endpoint with a key — no SDK required, though the thordata-sdk package wraps it for Python if you prefer. The direct version, so you can see the shape:
import requests
def serp(query, api_key, location="United States", device="mobile",
engine="google", fmt="json"):
r = requests.post(
"https://api.thordata.com/serp",
json={"q": query, "engine": engine, "location": location,
"device": device, "output": fmt},
headers={"Authorization": f"Bearer {api_key}"},
timeout=30,
)
r.raise_for_status()
return r.json()
One design note before the traps: the location and device defaults are doing real work. Search results vary by geography and device class — a national-desktop scrape tells you about a world that most of your users don’t live in. Pricing context: responses bill per successful request at roughly $0.70–$1.20 per 1,000 by volume, and trial credits exist for the evaluation phase — so the tutorial below is genuinely free to run for a new account.https://www.thordata.com/products/serp-api/?op=mary&from=dev

Trap One: The First Page Is a Lie (Pagination)
SERPs return a page of results, and the page you get is not “the results” — it’s one window. The trap bites in two directions: teams that treat page one as complete, and teams that paginate by naively re-requesting with an incremented start offset and discover inconsistent result sets (results shift between fetches; offsets aren’t stable snapshots).

The working pattern:

def serp_all(query, api_key, location, max_pages=3):
results = []
seen_urls = set() # dedup across pages: offsets drift
for page in range(max_pages):
payload = serp(query, api_key, location=location)
organic = payload.get("organic", [])
if not organic:
break # empty page: stop, don't guess
for item in organic:
if item["url"] not in seen_urls:
seen_urls.add(item["url"])
results.append(item)
return results
The seen_urls set is the quiet correctness fix: cross-page dedup handles the drift that offset-based pagination pretends doesn’t exist.

Trap Two: Parsing the JSON You Were Handed (Don’t)
The second trap is psychological: teams receive clean, structured JSON — organic array, positions, titles, URLs, features — and then write a “parser just to normalize a few things.” Don’t. The structured response is the normalization; every downstream parser is a schema-drift liability the next time the provider updates fields. Read the documented schema, consume it directly, and pin your dependency on the documented field names (raise loudly on absence rather than defaulting to None silently — a missing position should be an event, not a mystery).

def extract_rank(payload, target_domain):
for item in payload.get("organic", []):
if target_domain in item.get("url", ""):
return item["position"]
return None # absent from page one — a fact, not an error
Trap Three: Rate Discipline (Be a Client, Not a Flood)
The third trap is the retry storm: a timeout triggers a retry, the retry triggers the same timeout, and a burst of concurrency “to speed things up” turns a transient failure into a rate-shaped one. The working discipline for per-request-billed APIs:

Sequential by default, concurrent by measurement — parallelize only where you’ve measured the win
Timeouts short and honest (30s), retries few and backed-off (two, exponential)
Failures are data — log the query, the market, the error class; your block-shaped and quota-shaped failures have different signatures and different fixes
Trap Four: The Agent-Shaped Future of This Exact Tutorial
The last trap is temporal: the tutorial above teaches you to drive the API from Python — but if the consumer of your search results is an LLM agent, the modern integration isn’t Python code at all. The Model Context Protocol endpoint exposes the same SERP machinery as agent-discoverable tools: the Thordata MCP server gives Claude Desktop, Cursor, Cline, Windsurf, or any MCP/JSON-RPC runtime live search, engine listings, request history, and usage statistics — with the agent reading the engine schemas and choosing its own parameters. New accounts get 500 free calls and the MCP tier draws on the same response balance (no separate subscription). The same tutorial, two consumption shapes: your code calls serp(), or your agent calls the tool.

The Complete Working Pattern
Assembled — a monitoring-grade loop with every trap handled:

def track_keyword(query, market, api_key, store, max_pages=2):
payload_pages = [serp(query, api_key, location=market)
for _ in range(max_pages)]
organic = dedup_pages(payload_pages) # trap one
record = {
"query": query, "market": market,
"target_rank": extract_rank(payload_pages[0], "yourdomain.com"),
"observed_at": utc_now_iso(), # freshness is a field, not hope
"result_count": len(organic),
}
store.append(record) # schema consumed directly — trap two
return record
Wrap it in a scheduler, store observations as events, and you have the minimum honest rank tracker — geo-pinned, timestamped, deduped, parser-free. Add the MCP endpoint when an agent joins the pipeline, and the same backend serves both consumption shapes.

DEV Community: Marysha

Your data pipeline may not be "down",but it is quietly accumulating compliance debt

SERP API in Python: A Working Developer’s Tutorial (With the Traps Marked)