DEV Community

Mary
Mary

Posted on

How to Build a Localized SERP Monitoring Pipeline with Python


Manually checking search results works when you have a few keywords. It stops working when you need to track dozens of keywords across multiple cities, languages, and search engines—especially when rankings, ads, and SERP features change daily.

This article walks through a practical architecture for a SERP monitoring pipeline: collecting search results, storing them in a comparable format, analyzing changes over time, and optionally feeding the results into an AI workflow.

This article was created with the help of AI.

Why SERP Monitoring Matters
A search engine results page, or SERP, is not just a list of blue links. It can include organic results, paid ads, featured snippets, local packs, image results, video results, and other modules.

For a business, the same keyword can mean very different things in different markets. A product page may rank well in one city but be invisible in another. A competitor may dominate a local search result. A new content format may start taking attention away from traditional organic listings.

A monitoring pipeline helps you turn those observations into repeatable data:

Track keyword rankings over time.

Compare results across cities, countries, and languages.

Detect new competitors or changes in the search landscape.

Monitor branded search results and potential reputation issues.

Study which SERP features appear for high-value queries.

Provide current search context to AI agents or RAG systems.

The goal is not to replace human judgment. It is to remove repetitive data collection so the team can focus on interpretation and action.

A Simple Pipeline Architecture
A useful SERP monitoring system does not need to be complicated. At a minimum, it needs four parts:

*Keyword configuration *— the queries, locations, languages, and schedules you want to monitor.

*Collection layer *— a scheduled job that requests search results and saves the raw response.

Parsing layer — code that converts the response into structured records.

Storage and reporting — a database, dashboard, or alerting system that makes trends visible.

A minimal architecture could look like this:

text
Keyword config
      |
      v
Scheduler
      |
      v
SERP request layer
      |
      v
Raw response storage
      |
      v
Parser
      |
      v
Structured database
      |
      v
Dashboard / alerts / AI workflow
Enter fullscreen mode Exit fullscreen mode

The most important design decision is to always retain the raw response. Parsers change, search engines change their page structure, and you may later discover a data point you did not originally plan to extract.

Design the Data Model First
Before writing code, decide what you want to compare.

A common mistake is storing only the keyword and its ranking position. That makes it impossible to answer useful questions later, such as whether a result changed because of location, device, language, or time.

A practical table might include:

text
search_requests
- id
- keyword
- search_engine
- country
- city
- language
- device
- requested_at
- raw_response_path
- status
- error_message

serp_results
- id
- search_request_id
- result_type
- position
- title
- url
- domain
- description
- serp_feature
- parsed_at
Enter fullscreen mode Exit fullscreen mode

result_type can distinguish organic results from ads, local results, images, videos, or featured snippets. serp_feature can record whether the result appeared inside a special module, such as a local pack or People Also Ask section.

This structure lets you answer questions like:

Which keywords lost visibility this week?

Which domains consistently appear for a target query?

Do results differ between New York and London?

Did a competitor begin appearing in paid results?

Which queries trigger local packs or video results?

Collect Results with Python
The example below shows the collection layer in a provider-agnostic way. The exact request format depends on the SERP API you use, but the surrounding logic remains similar.

python
import os
import time
import json
import requests
from pathlib import Path

RAW_DIR = Path("raw_responses")
RAW_DIR.mkdir(exist_ok=True)

API_URL = os.environ["SERP_API_URL"]
API_TOKEN = os.environ["SERP_API_TOKEN"]


def fetch_serp(
    keyword: str,
    country: str = "us",
    city: str = None,
    language: str = "en",
    device: str = "desktop",
) -> dict:
    payload = {
        "q": keyword,
        "gl": country,
        "hl": language,
        "device": device,
    }

    if city:
        payload["location"] = city

    response = requests.post(
        API_URL,
        headers={
            "Authorization": f"Bearer {API_TOKEN}",
            "Content-Type": "application/json",
        },
        json=payload,
        timeout=30,
    )

    response.raise_for_status()
    return response.json()


def save_raw_response(keyword: str, data: dict) -> Path:
    timestamp = time.strftime("%Y%m%d-%H%M%S")
    safe_keyword = "".join(
        char if char.isalnum() else "_" for char in keyword.lower()
    )

    file_path = RAW_DIR / f"{safe_keyword}-{timestamp}.json"
    file_path.write_text(json.dumps(data, ensure_ascii=False, indent=2))

    return file_path
Enter fullscreen mode Exit fullscreen mode

In production, avoid storing API credentials directly in code. Use environment variables, a secrets manager, or your deployment platform’s secret storage.

Parse the Results
A SERP response may contain many modules. Start with the data you will actually use; do not try to parse everything on day one.

For a basic rank tracker, organic results are usually the first priority:

python
def parse_organic_results(data: dict, search_request_id: int):
    results = []

    for item in data.get("organic_results", []):
        results.append(
            {
                "search_request_id": search_request_id,
                "result_type": "organic",
                "position": item.get("position"),
                "title": item.get("title"),
                "url": item.get("url"),
                "domain": extract_domain(item.get("url")),
                "description": item.get("description"),
                "serp_feature": None,
            }
        )

    return results


def extract_domain(url: str) -> str | None:
    if not url:
        return None

    from urllib.parse import urlparse

    parsed = urlparse(url)
    return parsed.netloc
Enter fullscreen mode Exit fullscreen mode

If your provider returns a different schema, write a small adapter for it. Keeping parsing logic separate from collection logic makes it easier to switch providers or support multiple formats later.

Store Results in SQLite
SQLite is a good starting point for a small internal tool. It requires no separate database server and is easy to inspect locally.

python
import sqlite3

DB_PATH = "serp_monitor.db"


def init_db():
    connection = sqlite3.connect(DB_PATH)

    connection.execute(
        """
        CREATE TABLE IF NOT EXISTS search_requests (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            keyword TEXT NOT NULL,
            search_engine TEXT NOT NULL,
            country TEXT,
            city TEXT,
            language TEXT,
            device TEXT,
            requested_at TEXT NOT NULL,
            raw_response_path TEXT,
            status TEXT NOT NULL,
            error_message TEXT
        )
        """
    )

    connection.execute(
        """
        CREATE TABLE IF NOT EXISTS serp_results (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            search_request_id INTEGER NOT NULL,
            result_type TEXT NOT NULL,
            position INTEGER,
            title TEXT,
            url TEXT,
            domain TEXT,
            description TEXT,
            serp_feature TEXT,
            parsed_at TEXT NOT NULL,
            FOREIGN KEY (search_request_id)
                REFERENCES search_requests (id)
        )
        """
    )

    connection.commit()
    connection.close()
Enter fullscreen mode Exit fullscreen mode

For a larger system, PostgreSQL is usually a better fit. It offers stronger concurrency, better tooling, and more options for analytical queries.

Run a Daily Job
You can start with a simple scheduled script. Once the volume grows, move to a task queue or a workflow engine such as Celery, Temporal, or Airflow.

python
from datetime import datetime, timezone

KEYWORDS = [
    {"keyword": "project management software", "city": "New York"},
    {"keyword": "best crm for startups", "city": "London"},
    {"keyword": "email marketing tools", "city": "Singapore"},
]


def run_monitoring_job():
    init_db()

    for item in KEYWORDS:
        try:
            data = fetch_serp(
                keyword=item["keyword"],
                city=item["city"],
            )

            raw_path = save_raw_response(item["keyword"], data)

            connection = sqlite3.connect(DB_PATH)
            cursor = connection.cursor()

            cursor.execute(
                """
                INSERT INTO search_requests (
                    keyword, search_engine, country, city,
                    language, device, requested_at,
                    raw_response_path, status
                )
                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
                """,
                (
                    item["keyword"],
                    "google",
                    "us",
                    item["city"],
                    "en",
                    "desktop",
                    datetime.now(timezone.utc).isoformat(),
                    str(raw_path),
                    "success",
                ),
            )

            request_id = cursor.lastrowid
            parsed = parse_organic_results(data, request_id)

            cursor.executemany(
                """
                INSERT INTO serp_results (
                    search_request_id, result_type, position,
                    title, url, domain, description,
                    serp_feature, parsed_at
                )
                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
                """,
                [
                    (
                        row["search_request_id"],
                        row["result_type"],
                        row["position"],
                        row["title"],
                        row["url"],
                        row["domain"],
                        row["description"],
                        row["serp_feature"],
                        datetime.now(timezone.utc).isoformat(),
                    )
                    for row in parsed
                ],
            )

            connection.commit()
            connection.close()

        except Exception as error:
            print(f"Failed to collect {item['keyword']}: {error}")


if __name__ == "__main__":
    run_monitoring_job()
Enter fullscreen mode Exit fullscreen mode

This is intentionally simple. Once it works, add retry logic, request throttling, structured logging, and alerting.

Make Localized Comparisons Reliable
Localization is where many SERP monitoring projects become misleading.

If you compare a result collected in New York with one collected in London, you are not comparing the same search experience. Location, language, device, and even the time of collection can affect what the user sees.

Always store the full request context:

Keyword

Search engine

Country and city

Language

Device type

Request timestamp

API version or parser version

Then compare like with like. For example, compare “New York, English, desktop” results over time—not a mix of different locations and devices.

Add Alerts Without Creating Noise
A rank drop is not automatically an incident. Search results can fluctuate for many reasons, including personalization, regional variation, SERP layout changes, and normal ranking movement.

A useful alert should answer: what changed, by how much, and does it matter?

For example:

A target domain falls out of the top 10 for a priority keyword.

A competitor enters the top three for several related keywords.

A local pack appears where it previously did not.

A high-value page disappears from the first page for more than 24 hours.

The number of paid ads on a keyword increases sharply.

Avoid alerting on every one-position change. Instead, define thresholds based on business impact and keyword priority.

Use SERP Data in AI Workflows
SERP data can be useful input for AI systems, but it should not be treated as final truth.

A research agent might use search results to identify relevant sources, then visit the original pages, extract content, and generate an answer with citations. A RAG system can use search results to supplement an internal knowledge base with current public information.

A useful pattern is:

text
User question
      |
      v
Generate search queries
      |
      v
Retrieve SERP results
      |
      v
Select promising URLs
      |
      v
Fetch and extract page content
      |
      v
Cross-check and summarize
      |
      v
Answer with citations
Enter fullscreen mode Exit fullscreen mode

Keep the source URL, title, snippet, retrieval timestamp, and search query with every answer. This makes the output easier to audit and helps users understand where the information came from.

Engineering Pitfalls to Avoid
A few issues tend to appear in almost every SERP monitoring project:

Mixing locations in one dataset. Store location and language as first-class fields.

Overwriting historical results. Keep a time-series record so you can analyze trends.

Parsing only the happy path. Search pages and API responses change; add validation and error handling.

Ignoring SERP features. Organic position alone may not explain visibility changes.

Requesting too aggressively. Use sensible schedules, caching, and rate limits.

Treating rankings as absolute truth. Use rankings as one signal alongside traffic, conversions, and business outcomes.

Collecting more data than needed. Define a clear purpose for every field and retention period.

A Good Starting Point
Start small. Choose 10 to 20 keywords that genuinely affect your business, monitor them in two or three priority markets, and store the raw responses.

Once you can reliably answer questions such as “Which pages changed this week?” or “Which markets show a different competitive landscape?”, expand the system. Add more keywords, more locations, additional SERP modules, and AI-driven analysis only when the basic pipeline is stable.

The most valuable SERP monitoring system is not the one that collects the most data. It is the one that helps your team notice meaningful change early enough to act.

Top comments (0)