Getting search results into a Python script is a useful milestone.
It is not the same thing as building a reliable search-data pipeline.
Once those results feed a dashboard, a scheduled report, or an AI application, a different set of questions matters:
What exactly did we search for?
Which location and device settings did we use?
When did we collect the results?
Does an empty list mean “no results,” or did our integration break?
Can we change providers without rewriting the application?
This article uses Thordata’s SERP API as a concrete example. Its documentation describes an HTTP interface for retrieving search results, with options such as search engine, location, and device. The interesting engineering work starts after that request returns.
- Define the observation before writing the request Consider a small application that tracks which pages appear for a set of product-related searches.
The tempting data model is:
text
keyword → list of URLs
That leaves out the context needed to interpret the list.
A more useful model is:
text
query + search context + collection time → observed results
A search observation should preserve the settings that produced it. Thordata’s own tutorial highlights geography, device settings, and collection timestamps as important parts of interpreting SERP data.
For an application like this, I would start with an internal record shaped roughly like this:
json
{
"query": "project management software",
"engine": "google",
"location": "United States",
"device": "desktop",
"requested_at": "2026-11-17T10:00:00Z",
"received_at": "2026-11-17T10:00:02Z",
"provider": "thordata",
"results": []
}
This is a proposed application schema, not a copy of Thordata’s response format.
That distinction matters: your storage model should describe what your application needs to remember, rather than inherit every detail of a vendor’s JSON structure.
- Keep provider-specific parsing in one place The shortest integration passes the API response directly to the rest of the application:
python
payload = fetch_search_results(...)
update_dashboard(payload)
For a prototype, that can be enough.
For a longer-lived application, I would introduce a boundary:
text
Provider response
↓
Provider-specific adapter
↓
Application-owned search record
↓
Database, dashboard, or downstream consumer
Here is a minimal internal model:
from dataclasses import dataclass
from typing import Optional
@dataclass(frozen=True)
class SearchResult:
position: int
title: str
url: str
snippet: Optional[str] = None
The adapter is responsible for mapping the provider’s documented fields into this model.
It is also where I would decide what happens when a required field is missing:
class UnexpectedSearchResponse(ValueError):
pass
The important design choice is not the exception’s name. It is refusing to quietly convert every unexpected response into an empty list.
“No organic results were returned” and “our parser no longer understands the response” should be different states.
- Treat pagination as a collection problem Pagination looks simple until the underlying results change between requests.
Thordata’s Python tutorial explicitly calls out shifting result sets and duplicate URLs across search-result pages. That makes a multi-page collection different from reading slices of a fixed database snapshot.
For this application, I would use three rules:
Use the provider’s documented pagination mechanism.
Preserve the page and position associated with each result.
Deduplicate only in the downstream view that needs unique URLs.
That last rule prevents a subtle loss of information.
A research tool may want one entry per URL. A ranking-analysis tool may need to retain every observed position. Those are different views of the same collected data.
Repeatedly sending the same request is not pagination. The adapter must actually advance the page or offset according to the current API documentation.
- Make retries a budget decision A retry policy should answer two questions:
Is another attempt likely to help?
How much additional work are we willing to pay for?
Thordata’s tutorial recommends bounded retries, backoff, and measured rather than automatic concurrency for per-request-billed API usage.
For this project, I would handle failures according to their cause:
- Invalid credentials: Stop the job and surface a configuration error. Retrying with the same credentials will not fix the problem.
- Invalid request parameters: Review and correct the request before trying again.
- Rate-limit responses: Follow the provider’s documented retry guidance and set a maximum number of attempts.
- Temporary upstream failures: Retry within a limited attempt count and an overall time budget.
- Unexpected response structures: Preserve relevant diagnostic metadata and investigate instead of silently treating the response as an empty result. These are application-design choices, not a claim about Thordata’s exact error contract. The implementation should use its current documentation to map actual responses to these categories.
I would also avoid letting every layer retry independently. A scheduler, worker, and HTTP client should not each assume they own the full retry budget.
- Measure usable output, not just successful requests For this application, I would track:
Whether the HTTP request succeeded.
Whether the response passed validation.
How many usable results were extracted.
How many attempts the collection required.
How long the complete collection took.
Whether the requested search context was recorded.
A request returning successfully is only one part of the job.
The application needs an observation it can interpret, store, and compare later.
During development, I would retain a small, access-controlled sample of raw responses alongside the normalized records. That gives the adapter something concrete to test against when a field disappears or a response shape changes. Retention and redaction would be deliberate choices, not an instruction to log everything.
- Decide what you are outsourcing A SERP API moves search-result retrieval and parsing behind a service boundary. Thordata is one option in that category; its overview describes returning structured search data for queries with contextual parameters such as country, language, or device.
For this project, I would evaluate it with a representative set of queries rather than a single successful demo.
The evaluation would ask:
Does it support the search contexts the application needs?
Are the required fields consistently usable?
Is pagination behavior clear enough for our collection strategy?
Can we distinguish valid empty results from failures?
What does a complete collection cost after retries and pagination?
I would also keep the adapter small enough that testing another provider does not require changing the dashboard, database schema, and business logic at the same time.
That is not an argument against using a managed API. It is a way to make the integration easier to reason about.
The useful boundary
For a prototype, the first JSON response may be the finish line.
For a maintained application, I would treat it as the input boundary.
Let the search API handle retrieval. Keep search context, validation, retry budgets, storage, and downstream meaning explicit in your own code.
That is the difference between a script that fetches results and a pipeline whose results you can explain.
Top comments (0)