DEV Community: TalorData

How to Compare Two SERP Snapshots and Detect Ranking Changes

TalorData — Mon, 18 May 2026 08:27:12 +0000

If you are building a keyword rank tracker, collecting SERP data is only the first step.

The useful part is detecting what changed between two search result snapshots.

For example:

Did a page move from position 8 to position 3?
Did a competitor enter the top 10?
Did a URL disappear from page one?
Did the same domain rank with a different page?

What to store in a snapshot

A SERP snapshot should include both the results and the search context.

{
  "query": "search api",
  "engine": "google",
  "device": "desktop",
  "country": "us",
  "checked_at": "2026-05-18T08:00:00Z",
  "results": [
    {
      "position": 1,
      "title": "Example result",
      "url": "https://example.com/page",
      "domain": "example.com"
    }
  ]
}

Do not compare two snapshots unless the context matches:

query
engine
device
country or location

Otherwise, the ranking change may be caused by a different search context, not by SEO movement.

Compare by URL and domain

You usually need two views.

URL-level comparison answers:

Did this exact page move?

Domain-level comparison answers:

Is this website still visible, even if a different page ranks?

Example:

Yesterday: example.com/old-page at position 5
Today: example.com/new-page at position 6

At URL level:

old-page dropped
new-page is new

At domain level:

example.com moved from 5 to 6

Both are useful.

Basic change types

For each URL or domain, classify the change as:

new
dropped
moved_up
moved_down
unchanged

Use this delta formula:

delta = previous_position - current_position

So:

8 -> 3 = +5
3 -> 8 = -5

Positive means the result improved.

Example change report

{
  "query": "search api",
  "engine": "google",
  "device": "desktop",
  "country": "us",
  "changes": [
    {
      "type": "moved_up",
      "url": "https://example.com/page",
      "domain": "example.com",
      "previous_position": 8,
      "current_position": 3,
      "delta": 5
    },
    {
      "type": "new",
      "url": "https://competitor.com/page",
      "domain": "competitor.com",
      "previous_position": null,
      "current_position": 7,
      "delta": null
    }
  ]
}

Store snapshots, not only diffs

Do not store only the ranking changes.

Store the full snapshots first, then calculate diffs from them.

That makes it easier to:

reprocess old data
fix comparison bugs
change URL normalization rules
audit strange ranking changes
generate new reports later

A simple data model is:

serp_snapshots
serp_results
serp_changes

The serp_changes table can always be regenerated.

Common mistakes

The most common mistakes are:

comparing different devices
comparing different countries
not normalizing URLs
tracking only one domain
mixing URL-level and domain-level changes
using confusing delta direction
storing only diffs

Takeaway

A reliable SERP change detector needs more than yesterday’s position minus today’s position.

You need consistent search context, URL normalization, URL-level comparison, domain-level comparison, and full snapshot storage.

Once those are in place, ranking changes become much easier to explain.

What Role Does a Search API Play in AI/RAG Workflows?

TalorData — Tue, 12 May 2026 08:37:41 +0000

RAG systems need good retrieval before they can generate useful answers.

Most examples focus on vector databases, embeddings, and chunking. Those are important, but there is another layer that often gets overlooked:

How do you find fresh, relevant web sources in the first place?

That is where a Search API can help.

In an AI or RAG workflow, a Search API is usually used before crawling, embedding, or summarizing. It can provide structured search results for a user query, topic, company, product, or keyword.

A typical flow looks like this:
User query

Search API
Relevant URLs
Fetch page content
Clean and chunk text
Embed or summarize
Generate answer

This is useful when the system needs information that changes often, such as:

market research
competitor monitoring
product comparisons
news discovery
SEO research
brand monitoring
lead research
topic discovery

A Search API does not replace a vector database. It solves a different problem.

The vector database helps retrieve from content you already collected.

The Search API helps discover external content you may not have yet.

For many AI agents, this matters because the web is constantly changing. If the retrieval layer only uses old internal documents, the output can become stale.

A good search layer should return enough structured data to decide what to fetch next:

title
URL
snippet
position
search engine
location
language
timestamp

For some workflows, raw HTML or screenshots can also help with verification.

The main benefit is that the AI system can start from structured search results instead of relying on fragile scraping logic or manual URL lists.

In short:
Search API = discovery layer
Crawler = content collection layer
Vector database = memory layer
LLM = reasoning and generation layer

If your RAG system needs fresh web information, the search layer is worth designing carefully.

How to Get Google, Bing, and Yandex Search Results as JSON

TalorData — Sat, 09 May 2026 07:56:40 +0000

Search result data is useful for many developer and SEO workflows:

rank tracking
keyword research
competitor monitoring
market research
SEO tools
internal data pipelines
automation workflows

You can build your own scraper for this, but maintaining it usually means dealing with proxies, retries, browser rendering, parser updates, rate limits, anti-bot systems, and monitoring.

For many projects, it is simpler to use a Search API and get structured results back directly.

In this post, I’ll show a basic example using TalorData, a Search API that supports Google, Bing, and Yandex.

It can return:

JSON
raw HTML
screenshots

There are 1,000 free requests, and no credit card is required.

Website
Playground

Example: Get Google SERP Data as JSON

Here is a basic curl request:

curl -X POST 'https://serpapi.talordata.net/serp/v1/request' \
  -H 'Authorization: Bearer YOUR_API_TOKEN' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'engine=google' \
  -d 'q=search api' \
  -d 'json=2'

The request includes:

engine=google: the search engine
q=search api: the search query
json=2: return structured JSON output

TalorData also supports Bing and Yandex, so you can change the engine parameter depending on the search source you need.

Example Response Shape

The API returns structured data that you can use in an app, dashboard, report, or data pipeline.

A simplified response shape might look like this:

{
  "search_metadata": {
    "engine": "google",
    "query": "search api"
  },
  "organic_results": [
    {
      "position": 1,
      "title": "Example Result",
      "link": "https://example.com",
      "snippet": "Example search result snippet."
    }
  ]
}

Python Example

import requests

url = "https://serpapi.talordata.net/serp/v1/request"

headers = {
    "Authorization": "Bearer YOUR_API_TOKEN",
    "Content-Type": "application/x-www-form-urlencoded"
}

data = {
    "engine": "google",
    "q": "search api",
    "json": "2"
}

response = requests.post(url, headers=headers, data=data)
response.raise_for_status()

result = response.json()

print(result)

JavaScript Example

const response = await fetch("https://serpapi.talordata.net/serp/v1/request", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_TOKEN",
    "Content-Type": "application/x-www-form-urlencoded"
  },
  body: new URLSearchParams({
    engine: "google",
    q: "search api",
    json: "2"
  })
});

if (!response.ok) {
  throw new Error(`Request failed: ${response.status}`);
}

const result = await response.json();

console.log(result);

When to Use a Search API Instead of Building a Scraper

A Search API is usually a better fit when:

you need structured SERP data quickly
you do not want to maintain proxies
you need location or country-specific search results
you need JSON output for an app or workflow
scraping infrastructure is not your core product

Building your own scraper can still make sense if you need complete control, very custom behavior, or already have scraping infrastructure in place.

Common Use Cases

Search result APIs are often used for:

SEO rank tracking
keyword research
competitor analysis
brand monitoring
market research
AI and data enrichment workflows
internal dashboards

Final Thoughts

Search result scraping looks simple at first, but production systems usually require much more than one HTTP request.

Using a Search API can reduce the amount of infrastructure you need to maintain and let you focus on the data and product experience.

If you try TalorData, I’d be interested in feedback on:

response fields
output formats
pricing
playground usability
missing parameters