DEV Community: Elowen

Build a SERP-Based Content Gap Finder for Existing Pages

Elowen — Wed, 22 Jul 2026 02:44:12 +0000

Updating an existing article is harder than writing a new one in one specific way:

You already have content, but you do not always know what to add.

A keyword tool can show related terms. A content brief can list possible sections. But a live SERP can show what searchers are currently being offered:

comparison sections
setup steps
examples
pricing context
implementation notes
common pitfalls
glossary explanations
evaluation criteria
tool alternatives

This post shows a small pattern for building a content gap finder from SERP result patterns. A SERP API such as TalorData SERP API can collect structured search results; your code can compare those signals with an existing page outline.

What the tool should do

Input:

target keyword
current page title
current page headings
optional current page summary
top SERP result titles, snippets, and URLs

Output:

recurring SERP topics
page formats that appear often
candidate missing angles
gaps that need human review
things not worth adding

The last point matters. A gap finder should not turn every SERP detail into an editing task.

A simple input format

Start with your existing page outline:

{
  "page_url": "https://example.com/blog/serp-api-guide",
  "target_keyword": "serp api guide",
  "title": "SERP API Guide for Developers",
  "headings": [
    "What is a SERP API?",
    "How search result APIs work",
    "Common use cases",
    "Basic API workflow"
  ]
}

Then normalize the SERP results:

{
  "query": "serp api guide",
  "location": "United States",
  "captured_at": "2026-07-22T09:00:00Z",
  "results": [
    {
      "position": 1,
      "title": "How to Use a SERP API for SEO Monitoring",
      "url": "https://example.com/blog/serp-api-seo-monitoring",
      "snippet": "Learn how to collect ranking data and monitor search results over time."
    }
  ]
}

The example is generic. Replace field paths with the actual response schema from your provider.

Normalize text

import re
from collections import Counter
from urllib.parse import urlparse


def normalize_text(value: str) -> str:
    value = value.lower()
    value = re.sub(r"[^a-z0-9]+", " ", value)
    return re.sub(r"\s+", " ", value).strip()


def tokens(value: str) -> set[str]:
    stopwords = {
        "a", "an", "the", "and", "or", "to", "of", "for", "in", "with",
        "how", "what", "why", "is", "are", "your", "from", "by"
    }
    return {token for token in normalize_text(value).split() if token not in stopwords}

This is intentionally simple. For a first version, readable rules are easier to review than a model-driven score nobody trusts.

Extract current page topics

def page_topic_tokens(page: dict) -> set[str]:
    text = " ".join([page.get("title", ""), *page.get("headings", [])])
    return tokens(text)

The outline is not the full article, but it is often enough to find obvious missing areas.

Extract SERP topics

def result_topic_tokens(result: dict) -> set[str]:
    text = " ".join([
        result.get("title", ""),
        result.get("snippet", ""),
        urlparse(result.get("url", "")).path.replace("-", " ")
    ])
    return tokens(text)


def serp_topic_counts(serp: dict) -> Counter:
    counts = Counter()

    for result in serp["results"][:10]:
        weight = max(1, 11 - result.get("position", 10))
        for token in result_topic_tokens(result):
            counts[token] += weight

    return counts

This gives higher weight to topics appearing in higher positions.

Find candidate gaps

def find_candidate_gaps(page: dict, serp: dict, min_score: int = 5) -> list[dict]:
    page_tokens = page_topic_tokens(page)
    serp_counts = serp_topic_counts(serp)

    gaps = []
    for topic, score in serp_counts.most_common():
        if score < min_score:
            continue
        if topic in page_tokens:
            continue

        gaps.append({
            "topic": topic,
            "score": score,
            "reason": "Appears in SERP result titles, snippets, or URL paths but not in the current page outline"
        })

    return gaps[:20]

The output is a candidate list, not an editing plan.

Add page format detection

Content gaps are not only words. They can also be formats.

def detect_page_format(result: dict) -> str:
    text = normalize_text(" ".join([result.get("title", ""), result.get("url", "")]))

    if "pricing" in text:
        return "pricing_context"
    if "alternative" in text or "compare" in text or "comparison" in text:
        return "comparison"
    if "guide" in text or "tutorial" in text or "how to" in text:
        return "guide"
    if "docs" in text or "documentation" in text:
        return "documentation"
    if "review" in text:
        return "review"
    return "unknown"


def format_counts(serp: dict) -> Counter:
    return Counter(detect_page_format(result) for result in serp["results"][:10])

A page can be missing a topic, but it can also be missing a useful format. For example, an article may explain a concept but never include comparison criteria, implementation steps, or review questions.

Build a review output

def content_gap_report(page: dict, serp: dict) -> dict:
    return {
        "page_url": page.get("page_url"),
        "target_keyword": page.get("target_keyword"),
        "candidate_topic_gaps": find_candidate_gaps(page, serp),
        "serp_page_formats": dict(format_counts(serp)),
        "review_note": "Treat these as candidates. Review against search intent, page purpose, and editorial focus before adding sections."
    }

This kind of output is easy to pass to an editor or SEO reviewer.

Add a reject list

Not every gap should be filled.

Add reasons to reject a candidate:

off-topic for the page purpose
belongs on a separate page
appears only because of one outlier result
too commercial for an educational page
too beginner-level for an advanced guide
outside product or editorial scope
not useful for the target reader

A useful gap finder should help reduce random additions, not encourage them.

Where the SERP API fits

The SERP API is the collection layer. Your comparison logic is the analysis layer.

One useful workflow:

Collect SERP results for the target keyword
Normalize titles, snippets, URLs, and positions
Extract recurring topics and page formats
Compare them with the current page outline
Generate candidate gaps
Review what to add, ignore, or move to another page

If you need structured search result data for this workflow, TalorData can support the SERP collection step while your team owns the gap rules and editorial decisions.

What I would not automate

I would not automatically rewrite the page from the gap report.

SERP patterns can reveal missing angles, but they do not know your page purpose, brand voice, product limits, or editorial priorities.

Use the finder to create a review list. Let a human decide whether each gap belongs in the update.

Final thought

A content gap is not just "something competitors mention."

A useful content gap is a missing angle that matches the search intent, helps the target reader, and fits the page you are updating.

The SERP can help you find candidates. The editing decision still needs judgment.

Track Brand Mentions Across Search Result Titles and Snippets

Elowen — Tue, 21 Jul 2026 02:17:12 +0000

Manual brand searches do not scale.

If you care about brand visibility in search, checking whether your homepage ranks for your brand name is only the beginning.

For non-brand queries, you may want to know:

does the brand appear in result titles?
does it appear in snippets?
does it appear in URLs?
is the mention on your own domain or a third-party page?
which query group produced the mention?
did the context look positive, neutral, or unclear?

This post shows a small pattern for detecting brand mentions across search result titles, snippets, and URLs. A SERP API such as TalorData SERP API can collect structured results; the matching and review logic belongs in your code.

What counts as a brand mention?

Start with a clear definition.

A mention can appear in:

title text
snippet text
visible URL or domain
page URL path

You may also want to distinguish:

owned mention: result is on your own domain
third-party mention: result is on another domain
brand-only mention: text contains only the brand name
contextual mention: text mentions the brand with category, comparison, review, or use-case language

For a first version, keep it simple.

Example normalized result

Use a normalized shape so your matching code does not depend on raw provider fields:

{
  "query": "serp api alternatives",
  "location": "United States",
  "captured_at": "2026-07-21T09:00:00Z",
  "results": [
    {
      "position": 3,
      "title": "Example SERP API Alternatives Compared",
      "url": "https://example.com/blog/serp-api-alternatives",
      "snippet": "A comparison of tools for collecting search result data."
    }
  ]
}

The code below is generic. Replace field paths with the actual response schema from your SERP provider.

Define brand aliases

Brand matching should handle spelling, casing, and common variants.

BRAND_ALIASES = [
    "talordata",
    "talor data api",
    "talordata serp api",
]

OWNED_DOMAINS = {
    "talordata.com",
    "www.talordata.com",
}

Only include aliases you are comfortable tracking. Too many loose aliases can create false positives.

Normalize text before matching

import re
from urllib.parse import urlparse


def normalize_text(value: str) -> str:
    value = value.lower()
    value = re.sub(r"[^a-z0-9]+", " ", value)
    return re.sub(r"\s+", " ", value).strip()


def clean_domain(url: str) -> str:
    domain = urlparse(url).netloc.lower()
    if domain.startswith("www."):
        domain = domain[4:]
    return domain

Normalization keeps matching consistent across titles, snippets, and URLs.

Match title, snippet, and URL

def contains_brand(text: str, aliases: list[str]) -> bool:
    normalized = normalize_text(text)
    normalized_aliases = [normalize_text(alias) for alias in aliases]
    return any(alias in normalized for alias in normalized_aliases)


def inspect_result(result: dict, aliases: list[str]) -> dict:
    title = result.get("title", "")
    snippet = result.get("snippet", "")
    url = result.get("url", "")
    domain = clean_domain(url)

    title_match = contains_brand(title, aliases)
    snippet_match = contains_brand(snippet, aliases)
    url_match = contains_brand(url, aliases)
    owned_match = domain in OWNED_DOMAINS

    return {
        "position": result.get("position"),
        "title": title,
        "url": url,
        "domain": domain,
        "title_match": title_match,
        "snippet_match": snippet_match,
        "url_match": url_match,
        "owned_domain": owned_match,
        "any_brand_mention": title_match or snippet_match or url_match,
        "third_party_mention": (title_match or snippet_match or url_match) and not owned_match,
    }

The difference between owned_domain and third_party_mention is important.

Your own ranking page is useful. A third-party page that mentions the brand is a different signal.

Process a SERP snapshot

def inspect_serp(serp: dict) -> list[dict]:
    inspected = []

    for result in serp["results"][:10]:
        row = inspect_result(result, BRAND_ALIASES)
        row["query"] = serp["query"]
        row["location"] = serp.get("location")
        row["captured_at"] = serp.get("captured_at")
        inspected.append(row)

    return inspected

Now every result has a few fields you can aggregate later.

Create query-level metrics

def summarize_query(rows: list[dict]) -> dict:
    mentions = [row for row in rows if row["any_brand_mention"]]
    third_party = [row for row in rows if row["third_party_mention"]]
    owned = [row for row in rows if row["owned_domain"]]

    return {
        "query": rows[0]["query"] if rows else None,
        "brand_mentioned": bool(mentions),
        "owned_domain_visible": bool(owned),
        "third_party_mentions": len(third_party),
        "title_mentions": sum(1 for row in rows if row["title_match"]),
        "snippet_mentions": sum(1 for row in rows if row["snippet_match"]),
        "url_mentions": sum(1 for row in rows if row["url_match"]),
        "best_mention_position": min((row["position"] for row in mentions), default=None),
    }

This gives you a compact summary for dashboards or reports.

Add a review queue

Do not fully trust string matching.

Send these cases to review:

third-party mentions on high-intent queries
snippets that mention the brand but not the owned domain
titles with comparison, alternative, review, or pricing language
mentions where the URL is unknown
sudden drops in owned-domain visibility
sudden increases in third-party mentions

The review queue prevents the monitor from becoming a noisy keyword counter.

Store enough context

For each mention, store:

query
query group
location
capture date
position
title
snippet
URL
domain
owned or third-party flag
matched field: title, snippet, or URL
review status

This makes the result auditable.

Where the SERP API fits

The SERP API handles collection. Your code handles interpretation.

One useful workflow:

Collect SERP results for target queries
Normalize title, snippet, URL, domain, and position
Match brand aliases across fields
Separate owned results from third-party mentions
Queue sensitive results for review
Track changes over time

If you need structured search result data for this workflow, TalorData can support the SERP collection layer while your team defines the brand matching rules.

What not to over-automate

I would avoid automatically labeling mention quality from a short snippet alone.

A snippet may be truncated. A title may be ambiguous. A third-party page may mention the brand in a list, a comparison, a complaint, or a tutorial.

Use the monitor to find the places worth reading, not to make every judgment automatically.

Final thought

Brand visibility in search is more than whether your own domain ranks.

Titles, snippets, URLs, and third-party pages can all shape what searchers see. A small mention monitor can help you notice those signals before they disappear into manual search checks.

If you track brand mentions in SERPs, I would be curious which matching rules helped reduce false positives.

Turn SERP Results into a Simple Market Map by Domain Category

Elowen — Mon, 20 Jul 2026 06:22:19 +0000

A search result page can tell you more than rankings.

For a market research workflow, the useful question is often not just:

Who ranks first?

It is:

What kinds of players show up when buyers search this topic?

For example, a SERP for a commercial keyword may contain:

product homepages
pricing pages
review sites
comparison blogs
directories
documentation pages
marketplaces
community discussions
analyst or media articles

That mix is a rough market map.

This post shows a simple way to turn SERP results into a categorized market map. A SERP API such as TalorData SERP API can collect the result data; your code can classify domains and page types for analysis.

What we want to build

Input:

a list of keywords
a target location
top organic results for each keyword

Output:

domains appearing across the keyword set
category for each domain
page type for each ranking URL
count of appearances
highest observed position
example queries where the domain appeared

This is not a full competitive intelligence platform. It is a practical first pass for understanding the shape of a market.

Example categories

Start with a small category set:

vendor
review_site
directory
marketplace
media
docs_or_education
community
unknown

You can make the taxonomy more specific later. The first version should be easy to review.

Example page types

For page types, use:

homepage
pricing
comparison
review
guide
documentation
community_thread
category_page
unknown

Domain category answers "who is this?"

Page type answers "what kind of page is ranking?"

Both are useful.

A normalized SERP result

Keep the SERP API response separate from your analysis model. Normalize the fields you need:

{
  "query": "serp api for seo monitoring",
  "location": "United States",
  "captured_at": "2026-07-20T09:00:00Z",
  "results": [
    {
      "position": 1,
      "title": "Example SERP API Platform",
      "url": "https://example.com/serp-api",
      "snippet": "Collect search result data for monitoring workflows."
    }
  ]
}

The example is intentionally generic. Replace the field paths with the response schema from your SERP provider.

Extract the domain

from urllib.parse import urlparse


def clean_domain(url: str) -> str:
    host = urlparse(url).netloc.lower()
    if host.startswith("www."):
        host = host[4:]
    return host

Classify page type with transparent rules

def classify_page_type(url: str, title: str) -> str:
    text = f"{url} {title}".lower()

    if "/pricing" in text or "pricing" in title.lower():
        return "pricing"
    if "compare" in text or "alternative" in text:
        return "comparison"
    if "review" in text or "ratings" in text:
        return "review"
    if "/docs" in text or "documentation" in text:
        return "documentation"
    if "how to" in text or "guide" in text or "tutorial" in text:
        return "guide"
    if "forum" in text or "community" in text or "reddit" in text:
        return "community_thread"
    if urlparse(url).path in ["", "/"]:
        return "homepage"

    return "unknown"

The goal is not perfect classification. The goal is a reviewable first pass.

Classify domain category

Some domain categories need a known list. For example:

REVIEW_DOMAINS = {"g2.com", "capterra.com", "trustpilot.com"}
COMMUNITY_DOMAINS = {"reddit.com", "stackoverflow.com", "news.ycombinator.com"}
MEDIA_DOMAINS = {"techcrunch.com", "forbes.com", "wired.com"}


def classify_domain(domain: str, page_type: str) -> str:
    if domain in REVIEW_DOMAINS:
        return "review_site"
    if domain in COMMUNITY_DOMAINS:
        return "community"
    if domain in MEDIA_DOMAINS:
        return "media"
    if page_type == "documentation":
        return "docs_or_education"
    if page_type in {"pricing", "homepage"}:
        return "vendor"
    return "unknown"

You can add a manual override file later:

{
  "example.com": "vendor",
  "example-directory.com": "directory"
}

Manual overrides are useful because domain classification is often market-specific.

Build the market map

from collections import defaultdict


def build_market_map(serps: list[dict]) -> list[dict]:
    rows = {}

    for serp in serps:
        query = serp["query"]

        for result in serp["results"][:10]:
            url = result.get("url", "")
            title = result.get("title", "")
            position = result.get("position", 99)

            domain = clean_domain(url)
            page_type = classify_page_type(url, title)
            domain_category = classify_domain(domain, page_type)

            if domain not in rows:
                rows[domain] = {
                    "domain": domain,
                    "category": domain_category,
                    "appearances": 0,
                    "best_position": position,
                    "page_types": defaultdict(int),
                    "example_queries": set(),
                }

            row = rows[domain]
            row["appearances"] += 1
            row["best_position"] = min(row["best_position"], position)
            row["page_types"][page_type] += 1
            row["example_queries"].add(query)

    output = []
    for row in rows.values():
        output.append({
            "domain": row["domain"],
            "category": row["category"],
            "appearances": row["appearances"],
            "best_position": row["best_position"],
            "page_types": dict(row["page_types"]),
            "example_queries": sorted(row["example_queries"])[:5],
        })

    return sorted(output, key=lambda item: (-item["appearances"], item["best_position"]))

What to inspect first

Once the map is built, look for questions like:

Are vendor domains dominating the SERP?
Are review and directory sites acting as gatekeepers?
Are community discussions ranking for commercial queries?
Are documentation pages visible for developer keywords?
Are the same domains appearing across many adjacent keywords?
Are top pages mostly category pages, comparison pages, or educational guides?

These questions are more useful than a plain rank export.

Store evidence

For each classified row, keep enough evidence for review:

domain
category
page type counts
sample URLs
sample titles
queries where it appeared
capture date
classification source: rule or manual override

If someone disagrees with a category, you want to fix the rule or override, not argue from memory.

Where the SERP API fits

The SERP API is the collection layer. The market map is your analysis layer.

A simple workflow:

Collect results for a keyword set
Normalize titles, URLs, snippets, and positions
Classify domains and page types
Aggregate by domain
Review unknown or high-impact domains
Use the map to guide research questions

If you need structured result data for this workflow, TalorData can support the SERP collection step while your team owns the classification logic.

Final thought

A SERP market map will not answer every strategy question.

But it can show who repeatedly appears in front of searchers, which page formats are common, and where review sites, communities, media, or vendor pages shape the market conversation.

That is a useful starting point before deeper research.

If you have built a similar map, I would be interested in how you classify domains that do not fit cleanly into one category.

Build a Keyword Intent Classifier from SERP Page Patterns

Elowen — Fri, 17 Jul 2026 01:57:29 +0000

Keyword intent is easy to guess and hard to verify.

Two keywords can look similar in a spreadsheet, but their result pages can tell very different stories:

one SERP is mostly product pages
one SERP is mostly how-to guides
one SERP is full of comparison pages
one SERP has local packs or marketplace pages
one SERP has documentation and developer tools

If you classify intent from keyword text alone, you are often classifying the phrase you see, not the demand Google is showing.

This post walks through a lightweight keyword intent classifier that uses SERP page patterns as input. A SERP API such as TalorData SERP API can provide structured result data; the classifier logic stays in your own code.

The classifier idea

Instead of asking what a keyword sounds like, ask what kind of pages are ranking for it.

For example:

"best proxy API" may show review pages, vendor pages, and comparison posts
"how to scrape SERP results" may show tutorials and documentation
"serp api pricing" may show pricing pages and vendor pages
"keyword rank tracker" may show product pages and listicles

The result page gives you a second signal.

A simple label set

Keep the first version small:

informational
commercial
transactional
navigational
mixed
unclear

You can always split labels later. A smaller label set makes review easier.

Features to extract from the SERP

Start with signals that are visible in ordinary organic results:

title words
URL path words
result position
repeated page formats
common modifiers such as "best", "review", "pricing", "tutorial", "docs"

You do not need a large model for the first pass. A transparent rule system is often easier to debug.

Example input

This example uses a normalized SERP shape. Adjust the fields to match your provider response.

{
  "query": "serp api pricing",
  "results": [
    {
      "position": 1,
      "title": "SERP API Pricing",
      "url": "https://example.com/pricing",
      "snippet": "Compare plans for search result data access."
    },
    {
      "position": 2,
      "title": "Best SERP APIs Compared",
      "url": "https://example.com/blog/best-serp-apis",
      "snippet": "A comparison of APIs for search result collection."
    }
  ]
}

Build a small feature extractor

import re
from collections import Counter
from urllib.parse import urlparse

COMMERCIAL_TERMS = {"best", "compare", "comparison", "review", "alternative", "pricing", "plans"}
TRANSACTIONAL_TERMS = {"buy", "signup", "trial", "demo", "pricing", "checkout"}
INFORMATIONAL_TERMS = {"how", "guide", "tutorial", "what", "why", "docs", "documentation", "example"}


def tokenize(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9-]+", text.lower()))


def path_tokens(url: str) -> set[str]:
    path = urlparse(url).path.replace("-", " ").replace("_", " ")
    return tokenize(path)


def score_result(result: dict) -> Counter:
    text = " ".join([result.get("title", ""), result.get("snippet", "")])
    tokens = tokenize(text) | path_tokens(result.get("url", ""))

    scores = Counter()
    scores["commercial"] += len(tokens & COMMERCIAL_TERMS)
    scores["transactional"] += len(tokens & TRANSACTIONAL_TERMS)
    scores["informational"] += len(tokens & INFORMATIONAL_TERMS)
    return scores

This is not meant to be perfect. It is meant to be readable enough that your SEO or content team can challenge the rules.

Aggregate scores across the SERP

Top-ranking pages should usually count more than lower results.

def classify_serp(serp: dict) -> dict:
    total = Counter()

    for result in serp["results"][:10]:
        position = result.get("position", 10)
        weight = max(1, 11 - position)
        total.update({
            intent: score * weight
            for intent, score in score_result(result).items()
        })

    if not total:
        return {"label": "unclear", "scores": {}, "reason": "No intent signals found"}

    top_label, top_score = total.most_common(1)[0]
    second_score = total.most_common(2)[1][1] if len(total) > 1 else 0

    if top_score == 0:
        label = "unclear"
    elif second_score and top_score / second_score < 1.4:
        label = "mixed"
    else:
        label = top_label

    return {"label": label, "scores": dict(total), "reason": f"Top SERP patterns leaned {label}"}

Add page-type hints

You can improve the first pass by detecting page types:

def page_type(url: str, title: str) -> str:
    text = f"{url} {title}".lower()

    if "/pricing" in text or "pricing" in title.lower():
        return "pricing_page"
    if "/docs" in text or "documentation" in title.lower():
        return "documentation"
    if "best" in title.lower() or "compare" in title.lower():
        return "comparison"
    if "how to" in title.lower() or "guide" in title.lower():
        return "guide"
    return "unknown"

For many keywords, page-type counts are easier to explain than raw text matching.

Keep a review queue

Do not auto-accept every label.

Send these cases to review:

mixed
unclear
low score difference between top labels
high-value keywords
keywords where SERP page types changed since the last crawl

The review queue matters because intent classification is a content decision, not just a parsing task.

Store evidence with each label

When you save the label, also save:

query
location
capture date
top result titles
top result URLs
detected page types
classifier scores
reviewer override, if any

That evidence is useful when someone asks why a keyword was labeled commercial instead of informational.

Where the SERP API fits

The SERP API is the collection layer. The classifier is your interpretation layer.

One possible workflow:

Pull SERP results for a keyword list
Normalize titles, URLs, snippets, and result positions
Extract page-pattern signals
Assign a first-pass intent label
Queue uncertain keywords for review
Save the final label with SERP evidence

If you need structured search result data for this kind of workflow, TalorData can support the SERP collection step while your classifier owns the labeling logic.

What I would not automate

I would avoid fully automating final labels for strategic pages, content format decisions, or search intent changes after major SERP shifts.

A classifier should reduce spreadsheet work. It should not hide judgment.

Final pattern

Treat SERP patterns as evidence, not truth.

The useful version of this classifier is not the one that claims perfect accuracy. It is the one that gives your team a repeatable starting point and shows enough evidence for a human to correct it.

If you have built something similar, I would be curious how you separate commercial, informational, and mixed intent in your own workflow.

Build a Simple SERP API Evaluation Checklist for Developers

Elowen — Wed, 15 Jul 2026 02:00:06 +0000

Choosing a SERP API only by looking at a quick demo can lead to surprises later.

The demo may work for one keyword.

The real workflow may need multiple locations, predictable schemas, retry behavior, pagination, stable result types, and enough metadata to debug failures.

This post is a practical checklist I would use before adding any SERP API to a developer workflow.

It is not a vendor ranking. It is a way to ask better engineering questions. A provider such as TalorData SERP API can be evaluated through the same checklist as any search result data source.

1. What search context can you control?

Start with the inputs.

Check whether the API supports the context your workflow needs:

query text
search engine
location
language
device
page or pagination
time or freshness controls if available

For SEO monitoring, location and language often matter.

For AI agents, freshness and traceability often matter.

For market research, repeatable query parameters matter.

If the API cannot represent the search context you care about, the rest of the integration will be fragile.

2. Is the response schema stable enough?

A SERP response can contain many result types.

Before integrating, inspect how the API represents:

organic results
ads
People Also Ask or related questions
local packs
videos
images
shopping results
news or top stories
related searches

Do not assume every provider uses the same field names.

Build an internal normalized shape so your application is not tightly coupled to the raw response.

Example:

{
  "keyword": "serp api for seo monitoring",
  "location": "United States",
  "checked_at": "2026-07-15T09:00:00Z",
  "organic_results": [
    {
      "position": 1,
      "title": "Example title",
      "url": "https://example.com/page",
      "snippet": "Short result summary"
    }
  ],
  "features": {
    "people_also_ask": true,
    "ads": false,
    "local_pack": false
  }
}

3. Can you debug failed requests?

A useful API integration needs observable failures.

Check whether you can easily log:

request parameters
request ID if available
status code
error message
retryability
latency
timestamp

Your job should distinguish between temporary failures and permanent configuration errors.

Retrying an invalid location parameter is not useful.

Retrying a temporary timeout may be useful.

4. Does it fit your batching model?

Many real workflows are batch workflows.

Examples:

200 keywords every morning
50 keyword-location pairs every week
a set of queries for agent evaluation
recurring competitor domain checks

Before choosing an API, ask:

How will I queue requests?
How will I retry failed items?
How will I detect missing snapshots?
How will I avoid duplicate records?
How will I resume after partial failure?

The API call is only one part of the job.

5. Is the output useful without a dashboard?

A good first integration should produce useful data even before a dashboard exists.

Try exporting:

CSV snapshots
JSON traces
weekly summary files
domain frequency reports
SERP feature change logs

If the data is hard to use outside a polished UI, the integration may be harder to validate.

6. Can it support your actual use case?

Different workflows need different strengths.

For SEO monitoring:

location support
SERP features
stable organic results
recurring snapshots
diff-friendly fields

For AI agents:

freshness
source traceability
titles, URLs, snippets
predictable schema
logging-friendly responses

For market research:

repeatable queries
domain and page type analysis
regional comparison
exportable data

Do not evaluate only the API. Evaluate the workflow it needs to support.

7. What would I test first?

I would run a small evaluation before committing:

Pick 10 real queries.
Include at least 2 locations if location matters.
Save raw responses.
Normalize the fields your workflow needs.
Run one retry test.
Export a small report.
Review whether the data supports an actual decision.

That is more useful than reading a feature list alone.

Where TalorData fits

For workflows that need structured search result data, TalorData can be evaluated as the collection layer behind SERP monitoring, AI agent search traces, competitor tracking, or market research workflows.

The key is to test the API against your real workflow, not a generic demo query.

Final note

If you have evaluated search APIs before, what broke first in practice: schema, location support, reliability, pricing assumptions, or downstream reporting?

Log Search Sources So AI Agent Answers Can Be Audited Later

Elowen — Tue, 14 Jul 2026 05:43:35 +0000

A search-aware AI agent should not only return an answer.

It should also preserve enough context to explain where that answer came from.

If the agent called a search tool, what query did it use? When did it search? Which results were returned? Which sources were passed into the model? Which ones were ignored?

Without that trace, debugging becomes guesswork.

This post shows a small pattern for logging search sources as part of an AI agent answer trace. A SERP API such as TalorData SERP API can provide structured search result input; your agent workflow should preserve the search context around that input.

What the trace should answer

A useful search trace should help answer:

What did the agent search?
When did it search?
Which search result data was returned?
Which URLs were used as context?
Which result titles and snippets were visible to the model?
Was the source fresh enough for the task?
Can the answer be reviewed later?

This is not only for compliance or audit.

It also helps day-to-day debugging.

A minimal trace shape

Start with a simple object:

{
  "answer_id": "ans_2026_07_14_001",
  "user_prompt": "What changed in search results for this topic?",
  "search_trace": {
    "query": "serp api seo monitoring",
    "searched_at": "2026-07-14T09:00:00Z",
    "location": "United States",
    "results_used": [
      {
        "position": 1,
        "title": "Example result title",
        "url": "https://example.com/page",
        "snippet": "Short result summary"
      }
    ]
  },
  "answer_summary": "Short answer generated by the agent"
}

You can expand this later, but the first version should preserve query, time, location, and source URLs.

Keep raw and selected sources separate

The search tool may return many results.

The model may only use a few.

Track both when possible:

{
  "raw_result_count": 10,
  "selected_result_count": 3,
  "selected_urls": [
    "https://example.com/page-a",
    "https://example.com/page-b",
    "https://example.com/page-c"
  ]
}

That distinction matters.

If the answer is weak, you need to know whether the search results were poor, the selection step was poor, or the final generation step was poor.

A small Python example

This example is intentionally generic. Replace endpoint, auth, parameters, and response fields with the real documentation for your SERP API provider.

import json
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

import requests


SERP_API_ENDPOINT = os.environ["SERP_API_ENDPOINT"]
SERP_API_KEY = os.environ["SERP_API_KEY"]
TRACE_DIR = Path("agent_traces")


def fetch_search_results(query: str, location: str) -> dict[str, Any]:
    response = requests.get(
        SERP_API_ENDPOINT,
        headers={"Authorization": f"Bearer {SERP_API_KEY}"},
        params={
            "q": query,
            "location": location,
        },
        timeout=30,
    )
    response.raise_for_status()
    return response.json()

Normalize the result before putting it into the trace:

def normalize_results(raw: dict[str, Any]) -> list[dict[str, Any]]:
    results = []

    # Adjust this path to match your SERP API response schema.
    for index, item in enumerate(raw.get("organic_results", [])[:10], start=1):
        results.append(
            {
                "position": item.get("position", index),
                "title": item.get("title"),
                "url": item.get("link") or item.get("url"),
                "snippet": item.get("snippet"),
            }
        )

    return results

Create an answer trace:

def build_answer_trace(
    answer_id: str,
    user_prompt: str,
    query: str,
    location: str,
    normalized_results: list[dict[str, Any]],
    answer_summary: str,
) -> dict[str, Any]:
    return {
        "answer_id": answer_id,
        "user_prompt": user_prompt,
        "search_trace": {
            "query": query,
            "searched_at": datetime.now(timezone.utc).isoformat(),
            "location": location,
            "results_used": normalized_results[:5],
            "raw_result_count": len(normalized_results),
        },
        "answer_summary": answer_summary,
    }

Save it as JSON:

def save_trace(trace: dict[str, Any]) -> Path:
    TRACE_DIR.mkdir(exist_ok=True)
    path = TRACE_DIR / f"{trace['answer_id']}.json"

    with path.open("w", encoding="utf-8") as file:
        json.dump(trace, file, indent=2, ensure_ascii=False)

    return path

What to log first

For a first version, log these fields:

answer ID
user prompt or prompt ID
search query
search timestamp
location or market
result URLs
titles and snippets
selected sources
answer summary
model or prompt version if relevant

Do not wait until the full observability system exists.

A basic trace is still much better than no trace.

Where the SERP API fits

The SERP API provides the structured search input.

The answer trace preserves how that input moved through the agent workflow.

For teams building search-aware AI agents, TalorData can provide the search result data layer. The agent application should still own trace storage, source selection records, and answer review workflows.

Final note

If you are building AI agents with search tools, what do you log today: search query, returned sources, selected sources, prompt version, or final answer only?

Add Retry and Backoff Around Search API Calls

Elowen — Sat, 11 Jul 2026 01:32:23 +0000

A batch SERP collection job usually fails in boring ways.

One request times out.

One response returns a temporary error.

One keyword fails halfway through a run.

If your script stops there, the problem is not the SERP API call itself. The problem is that the job has no reliability wrapper around the call.

This post shows a small pattern for adding retry, backoff, timeout, and error logging around search API calls. The same pattern works whether your collection layer uses TalorData SERP API or another provider.

What retry should handle

Retry is useful for temporary failures, not every failure.

Good retry candidates:

network timeout
temporary server error
connection reset
rate-limit response if your provider documentation says retrying later is appropriate
short-lived infrastructure issues

Bad retry candidates:

invalid API key
malformed query parameters
unsupported location value
missing required request field
permanent validation error

Retrying permanent failures only makes the job slower and noisier.

A simple retry wrapper

This example is intentionally generic. Replace endpoint, auth, parameters, and retryable status codes with the actual API documentation for your provider.

import os
import random
import time
from typing import Any

import requests


SERP_API_ENDPOINT = os.environ["SERP_API_ENDPOINT"]
SERP_API_KEY = os.environ["SERP_API_KEY"]

RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}


class SerpRequestError(Exception):
    pass


def fetch_serp_once(keyword: str, location: str) -> dict[str, Any]:
    response = requests.get(
        SERP_API_ENDPOINT,
        headers={"Authorization": f"Bearer {SERP_API_KEY}"},
        params={
            "q": keyword,
            "location": location,
        },
        timeout=30,
    )

    if response.status_code in RETRYABLE_STATUS_CODES:
        raise SerpRequestError(f"Retryable status code: {response.status_code}")

    response.raise_for_status()
    return response.json()

Add exponential backoff

Do not retry immediately in a tight loop.

Use backoff with a small amount of jitter:

def backoff_seconds(attempt: int, base: float = 1.0, cap: float = 30.0) -> float:
    delay = min(cap, base * (2 ** attempt))
    jitter = random.uniform(0, 0.5 * delay)
    return delay + jitter


def fetch_serp_with_retry(keyword: str, location: str, max_attempts: int = 4) -> dict[str, Any]:
    last_error: Exception | None = None

    for attempt in range(max_attempts):
        try:
            return fetch_serp_once(keyword, location)
        except (requests.Timeout, requests.ConnectionError, SerpRequestError) as error:
            last_error = error

            if attempt == max_attempts - 1:
                break

            time.sleep(backoff_seconds(attempt))

    raise SerpRequestError(
        f"SERP request failed after {max_attempts} attempts for keyword={keyword!r} location={location!r}"
    ) from last_error

This keeps temporary failures from breaking the full batch.

Log failures instead of hiding them

Retry should not make failures invisible.

If a keyword fails after all attempts, write a failure record:

import csv
from datetime import datetime, timezone
from pathlib import Path


FAILURE_LOG = Path("serp_failures.csv")


def log_failure(keyword: str, location: str, error: Exception) -> None:
    file_exists = FAILURE_LOG.exists()

    with FAILURE_LOG.open("a", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(
            file,
            fieldnames=["failed_at", "keyword", "location", "error"],
        )

        if not file_exists:
            writer.writeheader()

        writer.writerow(
            {
                "failed_at": datetime.now(timezone.utc).isoformat(),
                "keyword": keyword,
                "location": location,
                "error": str(error),
            }
        )

The failure log helps you rerun only the missing pieces later.

Use retry inside a batch job

def collect_batch(configs: list[dict[str, str]]) -> list[dict[str, Any]]:
    snapshots = []

    for config in configs:
        keyword = config["keyword"]
        location = config["location"]

        try:
            raw = fetch_serp_with_retry(keyword, location)
            snapshots.append(
                {
                    "keyword": keyword,
                    "location": location,
                    "raw": raw,
                }
            )
        except Exception as error:
            log_failure(keyword, location, error)

    return snapshots

This job completes even if a few keyword-location pairs fail.

That is usually better than losing the entire run.

What to add later

Once the basic retry wrapper works, add:

request IDs or job IDs
structured logs
retry count per request
failure reason categories
rerun command for failed keywords
dashboard or report section for missing snapshots
alert only when failure rate crosses a threshold

Do not start with a complex job system if a simple wrapper and failure log solves the first problem.

Where the SERP API fits

The SERP API provides structured search result data.

Your reliability layer makes collection repeatable enough to trust.

A tool like TalorData can provide the search data layer, while your application owns retry policy, logging, reruns, and reporting around failed jobs.

Final note

If you run scheduled SERP collection, what do you track first: retry count, failed keywords, missing snapshots, or total failure rate?

Build a Location-Aware SERP Check for Local SEO Experiments

Elowen — Fri, 10 Jul 2026 01:45:06 +0000

A keyword does not always have one SERP.

The result page for the same query can change by city, region, country, language, or device context. If your script only checks the default location, you may be reviewing a search page your target users never see.

This post shows a small pattern for building a location-aware SERP check.

The SERP collection layer can be a provider such as TalorData SERP API. The important part in your application is making location an explicit input instead of a hidden default.

Why location should be explicit

Location affects many search workflows:

local SEO monitoring
regional competitor research
market-specific content planning
location-sensitive product categories
queries with map, service, or retail intent
multi-country SERP comparison

Even if you are not doing local SEO, location can still change which domains, page types, and snippets appear.

A location-aware check helps you avoid treating one SERP as the universal result.

A simple input shape

Start with a query config:

{
  "keyword": "serp api for seo monitoring",
  "location": "New York, United States",
  "language": "en",
  "device": "desktop"
}

The exact supported location fields depend on your SERP API provider. Keep your internal shape stable, then map it to the provider parameters in one place.

Minimal Python example

This example is intentionally generic. Replace endpoint, auth, parameters, and response fields with the real API documentation for your provider.

import json
import os
from datetime import datetime, timezone
from pathlib import Path

import requests


SERP_API_ENDPOINT = os.environ["SERP_API_ENDPOINT"]
SERP_API_KEY = os.environ["SERP_API_KEY"]
OUTPUT_DIR = Path("location_serp_snapshots")


QUERY_CONFIGS = [
    {
        "keyword": "serp api for seo monitoring",
        "location": "New York, United States",
        "language": "en",
        "device": "desktop",
    },
    {
        "keyword": "serp api for seo monitoring",
        "location": "Austin, United States",
        "language": "en",
        "device": "desktop",
    },
]


def fetch_serp(config: dict) -> dict:
    response = requests.get(
        SERP_API_ENDPOINT,
        headers={"Authorization": f"Bearer {SERP_API_KEY}"},
        params={
            "q": config["keyword"],
            "location": config["location"],
            "language": config["language"],
            "device": config["device"],
        },
        timeout=30,
    )
    response.raise_for_status()
    return response.json()

Normalize the result

Keep location in every saved snapshot:

def normalize_snapshot(config: dict, raw: dict) -> dict:
    results = []

    # Adjust this path to match your SERP API response schema.
    for index, item in enumerate(raw.get("organic_results", [])[:10], start=1):
        results.append(
            {
                "position": item.get("position", index),
                "title": item.get("title"),
                "url": item.get("link") or item.get("url"),
                "snippet": item.get("snippet"),
            }
        )

    return {
        "keyword": config["keyword"],
        "location": config["location"],
        "language": config["language"],
        "device": config["device"],
        "checked_at": datetime.now(timezone.utc).isoformat(),
        "results": results,
    }

Save one file per keyword and location

def safe_name(value: str) -> str:
    return (
        value.lower()
        .replace(" ", "_")
        .replace(",", "")
        .replace("/", "_")
    )


def save_snapshot(snapshot: dict) -> Path:
    OUTPUT_DIR.mkdir(exist_ok=True)

    filename = "__".join(
        [
            safe_name(snapshot["keyword"]),
            safe_name(snapshot["location"]),
            snapshot["checked_at"][:10],
        ]
    )

    path = OUTPUT_DIR / f"{filename}.json"
    with path.open("w", encoding="utf-8") as file:
        json.dump(snapshot, file, indent=2, ensure_ascii=False)

    return path


if __name__ == "__main__":
    for config in QUERY_CONFIGS:
        raw = fetch_serp(config)
        snapshot = normalize_snapshot(config, raw)
        path = save_snapshot(snapshot)
        print(f"Saved {config['keyword']} for {config['location']} to {path}")

Compare locations carefully

Once you have snapshots for multiple locations, compare them with context.

Useful questions:

Which domains appear in every location?
Which domains are location-specific?
Which SERP features appear only in some markets?
Does the query show local intent in one city but informational intent in another?
Are competitors different across regions?

Do not assume every difference is meaningful. Use repeated checks before making a decision.

Where the SERP API fits

The SERP API provides structured search result data for each configured location.

Your code should make location visible in the config, file name, storage model, and report output.

For workflows that need repeatable location-aware search result collection, TalorData can provide the collection layer while your application owns the experiment design and comparison rules.

Final note

If you run local or regional SEO checks, do you compare cities, countries, languages, devices, or all of them separately?

Create a SERP Diff Table for Titles, URLs, and Positions

Elowen — Thu, 09 Jul 2026 02:08:53 +0000

A SERP diff is only useful if someone can read it.

It is easy to compare two JSON snapshots and produce a long list of changed objects. It is harder to turn those changes into a table that an SEO or growth teammate can review quickly.

This post shows a small Python pattern for creating a readable SERP diff table for URL, title, and position changes.

The search result snapshots can come from a SERP API such as TalorData SERP API. The diff table is the layer that turns raw changes into reviewable output.

The input shape

Assume each snapshot has normalized organic results:

{
  "keyword": "serp api for seo monitoring",
  "checked_at": "2026-07-09T09:00:00Z",
  "results": [
    {
      "position": 1,
      "title": "Example result title",
      "url": "https://example.com/page"
    }
  ]
}

The exact provider response does not matter for this step. Normalize first, then diff.

What the table should show

A first diff table can include:

keyword
change type
URL
previous position
current position
previous title
current title
short note

That gives reviewers enough context without asking them to open raw JSON.

Example output:

keyword,change_type,url,previous_position,current_position,note
serp api for seo monitoring,position_changed,example.com/page,3,7,URL moved down
serp api for seo monitoring,title_changed,example.com/page,3,3,Title changed
serp api for seo monitoring,new_url,newsite.com/post,,5,New URL entered tracked range

Build the diff rows

from dataclasses import dataclass
from typing import Any


@dataclass
class DiffRow:
    keyword: str
    change_type: str
    url: str
    previous_position: int | None
    current_position: int | None
    previous_title: str | None
    current_title: str | None
    note: str


def index_by_url(snapshot: dict[str, Any]) -> dict[str, dict[str, Any]]:
    return {
        result["url"]: result
        for result in snapshot.get("results", [])
        if result.get("url")
    }

Now compare two snapshots:

def build_diff_rows(before: dict[str, Any], after: dict[str, Any]) -> list[DiffRow]:
    keyword = after.get("keyword") or before.get("keyword") or "unknown"
    before_by_url = index_by_url(before)
    after_by_url = index_by_url(after)

    rows: list[DiffRow] = []

    for url, current in after_by_url.items():
        previous = before_by_url.get(url)

        if previous is None:
            rows.append(
                DiffRow(
                    keyword=keyword,
                    change_type="new_url",
                    url=url,
                    previous_position=None,
                    current_position=current.get("position"),
                    previous_title=None,
                    current_title=current.get("title"),
                    note="New URL entered tracked range",
                )
            )
            continue

        if previous.get("position") != current.get("position"):
            rows.append(
                DiffRow(
                    keyword=keyword,
                    change_type="position_changed",
                    url=url,
                    previous_position=previous.get("position"),
                    current_position=current.get("position"),
                    previous_title=previous.get("title"),
                    current_title=current.get("title"),
                    note="Position changed",
                )
            )

        if previous.get("title") != current.get("title"):
            rows.append(
                DiffRow(
                    keyword=keyword,
                    change_type="title_changed",
                    url=url,
                    previous_position=previous.get("position"),
                    current_position=current.get("position"),
                    previous_title=previous.get("title"),
                    current_title=current.get("title"),
                    note="Title changed",
                )
            )

    for url, previous in before_by_url.items():
        if url not in after_by_url:
            rows.append(
                DiffRow(
                    keyword=keyword,
                    change_type="removed_url",
                    url=url,
                    previous_position=previous.get("position"),
                    current_position=None,
                    previous_title=previous.get("title"),
                    current_title=None,
                    note="URL left tracked range",
                )
            )

    return rows

Write the table to CSV

import csv
from dataclasses import asdict
from pathlib import Path


def write_diff_csv(rows: list[DiffRow], output_path: str = "serp_diff.csv") -> None:
    fieldnames = [
        "keyword",
        "change_type",
        "url",
        "previous_position",
        "current_position",
        "previous_title",
        "current_title",
        "note",
    ]

    with Path(output_path).open("w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()
        for row in rows:
            writer.writerow(asdict(row))

The table is simple enough to attach to a weekly report or paste into a spreadsheet.

Add review labels before alerts

Do not send every diff as an alert.

Add review labels first:

def review_label(row: DiffRow) -> str:
    if row.change_type == "removed_url":
        return "review"

    if row.change_type == "new_url" and row.current_position and row.current_position <= 5:
        return "review"

    if row.change_type == "position_changed":
        if row.previous_position and row.current_position:
            movement = abs(row.previous_position - row.current_position)
            if movement >= 3:
                return "review"

    return "log"

This separates logging from alerting.

A diff table should help humans review changes. It should not become a notification machine by default.

Where the SERP API fits

The SERP API provides structured snapshots. The diff table makes those snapshots usable.

A tool like TalorData can provide the recurring search result data, while your code decides how to compare, label, and present changes.

Final note

If you were reviewing SERP diffs, what output would be easiest to use: CSV, Markdown table, Slack message, or a dashboard row?

Compare Competitor Domains Across SERP Snapshots with Python

Elowen — Wed, 08 Jul 2026 06:01:45 +0000

A competitor list is usually too static.

Search results are not.

If you track a set of keywords every week, you can learn which domains keep appearing in front of your audience. Some will be direct competitors. Some will be directories, review sites, forums, media pages, or content hubs that influence the buying journey.

This post shows a small Python pattern for comparing competitor domains across SERP snapshots.

The collection layer can be a SERP API such as TalorData SERP API. The analysis layer is a simple domain frequency report.

The goal

Given multiple SERP snapshots, we want to answer:

Which domains appear most often?
Which domains appear for the highest-priority keywords?
Which domains are new this week?
Which domains should be reviewed by the SEO or growth team?

This is not a full competitive intelligence system. It is a lightweight way to turn search results into a review list.

Example snapshot shape

Assume each snapshot contains one keyword and its top organic results:

{
  "keyword": "serp api for seo monitoring",
  "checked_at": "2026-07-08T09:00:00Z",
  "results": [
    {
      "position": 1,
      "title": "Example result title",
      "url": "https://example.com/page"
    }
  ]
}

The exact response depends on your SERP API provider. Normalize the data before building the competitor report.

Extract domains from URLs

Start with a small helper:

from urllib.parse import urlparse


def domain_from_url(url: str) -> str:
    parsed = urlparse(url)
    return parsed.netloc.lower().replace("www.", "")

This is intentionally simple. If you need production-level handling of subdomains and public suffixes, add a proper domain parsing library later.

Load snapshots and count domains

import json
from collections import Counter, defaultdict
from pathlib import Path
from urllib.parse import urlparse


SNAPSHOT_DIR = Path("serp_snapshots")


def domain_from_url(url: str) -> str:
    return urlparse(url).netloc.lower().replace("www.", "")


def load_snapshots() -> list[dict]:
    snapshots = []
    for path in SNAPSHOT_DIR.glob("*.json"):
        with path.open("r", encoding="utf-8") as file:
            snapshots.append(json.load(file))
    return snapshots


def count_domains(snapshots: list[dict]) -> Counter:
    counter = Counter()

    for snapshot in snapshots:
        for result in snapshot.get("results", []):
            url = result.get("url")
            if not url:
                continue
            counter[domain_from_url(url)] += 1

    return counter


if __name__ == "__main__":
    snapshots = load_snapshots()
    domain_counts = count_domains(snapshots)

    for domain, count in domain_counts.most_common(20):
        print(f"{domain}: {count}")

This gives you the most visible domains across your tracked keyword set.

Add keyword context

Domain frequency alone can be misleading.

A domain appearing once for a high-intent keyword may matter more than a domain appearing five times for low-priority research terms.

Add keyword-level context:

def domain_keyword_map(snapshots: list[dict]) -> dict[str, set[str]]:
    mapping = defaultdict(set)

    for snapshot in snapshots:
        keyword = snapshot.get("keyword")
        for result in snapshot.get("results", []):
            url = result.get("url")
            if keyword and url:
                mapping[domain_from_url(url)].add(keyword)

    return mapping

Now your report can show:

example.com: appeared for 8 keywords
  - serp api for seo monitoring
  - search data api
  - competitor serp tracking

That is easier to review than a raw count.

Compare this week with last week

To detect new competitor domains, compare two sets:

def domains_in_snapshots(snapshots: list[dict]) -> set[str]:
    domains = set()
    for snapshot in snapshots:
        for result in snapshot.get("results", []):
            url = result.get("url")
            if url:
                domains.add(domain_from_url(url))
    return domains


def new_domains(previous_week: list[dict], current_week: list[dict]) -> set[str]:
    previous_domains = domains_in_snapshots(previous_week)
    current_domains = domains_in_snapshots(current_week)
    return current_domains - previous_domains

A weekly report can then include:

most frequent domains
new domains this week
domains appearing for priority keywords
domains that need manual classification

Do not label everything as a competitor

This is important.

Not every visible domain is a direct competitor.

You may find:

direct product competitors
affiliate lists
review sites
marketplaces
documentation pages
forums
media articles
comparison pages
unrelated pages that need filtering

The first report should classify domains manually before the automation starts making conclusions.

Where the SERP API fits

The SERP API collects the search result snapshots. The domain analysis turns those snapshots into a competitor review list.

For workflows that need structured search result data, TalorData can provide the collection layer. Your code can own the domain grouping, classification rules, and weekly review format.

Final note

If you monitor competitors through SERPs, do you group by exact URL, domain, subdomain, or page type first?

Detect SERP Features in Search Results for Better SEO Context

Elowen — Tue, 07 Jul 2026 03:41:17 +0000

Organic ranking is useful, but it does not explain the whole search page.

A keyword can keep the same position while the SERP around it changes: a People Also Ask block appears, ads take more space, a local pack shows up, or video results start competing for attention.

If your monitoring script only stores organic positions, those changes disappear from the review.

This post shows a small pattern for detecting SERP features from structured search result data. A SERP API such as Talor Data SERP API can provide the search result data layer; your application can decide which features matter for your workflow.

What counts as a SERP feature?

The exact feature list depends on the search engine, query, location, and API response.

For a practical first version, you can track broad categories:

organic results
ads
People Also Ask
local pack
video results
image results
shopping results
top stories
related searches
AI or answer-style modules if your data source exposes them

Avoid overfitting the first version. The goal is to preserve context, not to model every possible page element on day one.

Normalize feature detection

Do not scatter feature checks throughout your reporting code.

Create one function that turns a raw or normalized SERP response into a stable internal shape:

{
  "keyword": "serp api for seo monitoring",
  "checked_at": "2026-07-07T09:00:00Z",
  "features": {
    "organic_results": true,
    "ads": false,
    "people_also_ask": true,
    "local_pack": false,
    "video_results": false
  }
}

This makes later reporting easier. You can compare feature presence day by day without rereading the full response every time.

A minimal Python detector

This example is generic. Replace the field names with the actual schema from your SERP API provider.

from typing import Any


FEATURE_PATHS = {
    "organic_results": ["organic_results"],
    "ads": ["ads", "paid_results"],
    "people_also_ask": ["people_also_ask", "related_questions"],
    "local_pack": ["local_pack", "local_results"],
    "video_results": ["videos", "video_results"],
    "image_results": ["images", "image_results"],
    "shopping_results": ["shopping_results"],
    "top_stories": ["top_stories", "news_results"],
    "related_searches": ["related_searches"],
}


def has_any_field(raw: dict[str, Any], possible_fields: list[str]) -> bool:
    for field in possible_fields:
        value = raw.get(field)
        if isinstance(value, list) and len(value) > 0:
            return True
        if isinstance(value, dict) and len(value) > 0:
            return True
        if value is True:
            return True
    return False


def detect_serp_features(raw: dict[str, Any]) -> dict[str, bool]:
    return {
        feature_name: has_any_field(raw, fields)
        for feature_name, fields in FEATURE_PATHS.items()
    }

This gives you a compact feature map that can be stored with each keyword snapshot.

Compare features over time

Once you store features per snapshot, you can compare two days:

def diff_features(before: dict[str, bool], after: dict[str, bool]) -> dict[str, list[str]]:
    added = []
    removed = []

    for feature, after_value in after.items():
        before_value = before.get(feature, False)

        if after_value and not before_value:
            added.append(feature)
        elif before_value and not after_value:
            removed.append(feature)

    return {
        "added": added,
        "removed": removed,
    }

A report row can then say:

keyword: serp api for seo monitoring
added: people_also_ask
removed: none

That is more useful than only saying the page stayed at position 4.

Store features next to ranking data

A practical snapshot can store both:

{
  "keyword": "serp api for seo monitoring",
  "checked_at": "2026-07-07T09:00:00Z",
  "top_organic_position": 4,
  "features": {
    "organic_results": true,
    "people_also_ask": true,
    "ads": false,
    "local_pack": false
  }
}

This lets you ask better questions later:

Did clicks drop because position changed, or because the SERP added more competing elements?
Did a keyword become more local, visual, or question-driven?
Are some content formats appearing more often than before?
Should a change be an alert, a weekly report item, or just a log entry?

Where the SERP API fits

The SERP API gives you structured search result data. The feature detector turns that data into a stable monitoring signal.

For teams building this kind of workflow, Talor Data can be used as the data collection layer, while your code owns feature naming, storage, and reporting rules.

Final note

If you monitor SERPs, which features do you track first: ads, People Also Ask, local results, videos, shopping, or something else?

Build a Weekly SERP Trend Report with a Simple CSV Output

Elowen — Mon, 06 Jul 2026 02:31:06 +0000

A weekly SERP report does not need to start as a dashboard.

For many teams, a CSV file is enough for the first version.

The goal is simple: take daily SERP snapshots, compare them over a week, and summarize the changes that someone should review.

This post shows a lightweight pattern for building that report from normalized SERP data. The collection layer can be a SERP API such as Talor Data SERP API, while the report logic stays in your own code.

The input shape

Assume you already save one normalized SERP snapshot per keyword per day.

A simplified record can look like this:

{
  "keyword": "serp api for seo monitoring",
  "checked_at": "2026-07-06T09:00:00Z",
  "results": [
    {
      "position": 1,
      "title": "Example result title",
      "url": "https://example.com/page",
      "domain": "example.com"
    }
  ]
}

The exact fields depend on your provider and normalization layer. Do not build the report directly against a provider-specific response if you can avoid it. Normalize first, report second.

What the weekly report should answer

For a first version, avoid too many metrics.

I would start with five questions:

Which domains appeared most often this week?
Which tracked URLs moved up or down?
Which new URLs entered the top results?
Which URLs disappeared from the tracked range?
Which titles changed for important pages?

Those questions are easier to review than a raw dump of every daily result.

Minimal Python example

This example assumes you have a folder of JSON snapshots.

import csv
import json
from pathlib import Path
from urllib.parse import urlparse


SNAPSHOT_DIR = Path("serp_snapshots")
REPORT_PATH = Path("weekly_serp_report.csv")


def domain_from_url(url: str) -> str:
    return urlparse(url).netloc.replace("www.", "")


def load_snapshots() -> list[dict]:
    snapshots = []
    for path in SNAPSHOT_DIR.glob("*.json"):
        with path.open("r", encoding="utf-8") as file:
            snapshots.append(json.load(file))
    return snapshots


def flatten_results(snapshots: list[dict]) -> list[dict]:
    rows = []
    for snapshot in snapshots:
        keyword = snapshot["keyword"]
        checked_at = snapshot["checked_at"]

        for result in snapshot.get("results", []):
            url = result.get("url")
            if not url:
                continue

            rows.append(
                {
                    "keyword": keyword,
                    "checked_at": checked_at,
                    "position": result.get("position"),
                    "title": result.get("title"),
                    "url": url,
                    "domain": result.get("domain") or domain_from_url(url),
                }
            )
    return rows


def write_csv(rows: list[dict]) -> None:
    fieldnames = ["keyword", "checked_at", "position", "domain", "title", "url"]
    with REPORT_PATH.open("w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)


if __name__ == "__main__":
    snapshots = load_snapshots()
    rows = flatten_results(snapshots)
    write_csv(rows)
    print(f"Wrote {len(rows)} rows to {REPORT_PATH}")

This is not a finished analytics system. It is a clean export that lets you inspect the week before deciding what to automate next.

Add a small summary layer

A raw CSV is useful, but a weekly review needs a summary.

You can start with domain frequency:

from collections import Counter


def summarize_domains(rows: list[dict]) -> list[tuple[str, int]]:
    counts = Counter(row["domain"] for row in rows if row.get("domain"))
    return counts.most_common(20)

Then write a second CSV:

def write_domain_summary(domain_counts: list[tuple[str, int]]) -> None:
    with Path("weekly_domain_summary.csv").open("w", newline="", encoding="utf-8") as file:
        writer = csv.writer(file)
        writer.writerow(["domain", "appearances"])
        writer.writerows(domain_counts)

Now the team has two useful files:

one detailed result export
one quick domain-level summary

Keep the first report boring

The first weekly report should not try to explain everything.

It should help someone ask better questions:

Why did this competitor domain appear so often?
Which keywords changed the most?
Are the same directories or review pages showing up repeatedly?
Did the search results become more commercial, educational, or comparison-driven?

A SERP API like Talor Data can provide the structured search result data. The report layer should translate that data into a review habit your team can maintain.

What I would add later

After the CSV version is useful, then consider adding:

position change calculations
new and removed URL detection
SERP feature summaries
keyword priority grouping
Slack or email delivery
a dashboard only if people keep asking for one

That order matters. A report that people read is better than a dashboard nobody reviews.

Final note

If you build weekly SERP reports, what do you summarize first: domains, URLs, ranking movement, SERP features, or content formats?