Google SERP Scraper API: How to Track Keyword Rankings with Python

If you manage SEO for your own site or client portfolios, you already know that checking keyword positions by hand is unreliable. Your location, search history, and device change what Google shows, and after the first few keywords you stop doing it consistently. A Google SERP scraper API solves this by returning structured search results for a list of keywords, so you can track rankings, compare competitors, and store history in a database or spreadsheet.

This guide is for SEOs, developers, and agency operators who want a repeatable Python workflow for keyword rank tracking without building their own proxy pool or parser. We will cover how a SERP scraper API works, what fields you can expect, a runnable Python script, and when a managed scraper makes more sense than self-hosting.

TL;DR: The Practical Path

Use a managed Google SERP scraper API to send a list of keywords and receive structured results—position, title, URL, snippet, and page features—then find where your domain ranks for each keyword. Store the results in SQLite or a CSV, run the script daily or weekly, and use the trend data to prioritize content updates. If you want to skip proxy rotation, parsing, and rate-limit negotiation, a managed service like the CoreClaw Google Search scraper handles the infrastructure layer and returns JSON you can plug directly into your tracker.

Why Manual Rank Checking Fails

Manual checks introduce three problems that make trend data almost useless:

1. Personalization and geo-variance. Google results change based on location, language, device, and signed-in state. A ranking you see in Hangzhou is not the same ranking a user sees in London.
2. Scale. A modest content site targets 50–200 keywords. An agency may track thousands. Opening an incognito browser tab for each keyword is not a workflow.
3. History. Even if you capture a position today, you need dated records to detect momentum. Spreadsheets filled manually decay within weeks.

The official alternative is the Google Search Console API, which shows average position, impressions, and clicks for queries your site already appears for. It is useful, but it does not show you results for competitors, exact SERP features, or keywords you are not yet ranking for. That is where a SERP scraper API fits.

What a Google SERP Scraper API Returns

A SERP scraper API sends a query to Google on your behalf and returns the page in a structured format. Depending on the provider and parameters, the output typically includes:

• keyword: the query that was searched
• position: the 1-based rank of the result
• title: the page title from the search result
• url: the destination URL
• snippet: the visible meta description or text fragment
• domain: the root domain extracted from the URL
• device: desktop, mobile, or tablet
• location / country: the geo parameters used for the search
• checked_at: ISO timestamp of when the check ran
• is_featured_snippet, is_ad, people_also_ask, related_searches: optional SERP feature flags

The API abstracts away the browser, proxy rotation, parsing, and retry logic. You send JSON, get JSON, and focus on analysis.

A Runnable Python Rank Tracker

The script below reads your API credentials from environment variables, sends a list of keywords to a SERP scraper endpoint, finds your domain's position in each result set, and stores the rows in SQLite. The endpoint and key are not hardcoded, so you can point it at any provider you verify.

import os
import json
import sqlite3
import time
from datetime import datetime, timezone
from urllib.parse import urlparse

import requests

# Read from environment variables. Do not commit keys to source control.
API_KEY = os.environ["CORECLAW_API_KEY"]
SERP_ENDPOINT = os.environ["CORECLAW_SERP_ENDPOINT"]
TARGET_DOMAIN = os.environ.get("TARGET_DOMAIN", "example.com")

KEYWORDS = [
    "best project management software",
    "asana alternative",
    "trello vs monday",
]

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

PAYLOAD = {
    "keywords": KEYWORDS,
    "location": "United States",
    "language": "en",
    "device": "desktop",
    "num_results": 10,
}


def store_rankings(rows: list[dict]) -> None:
    conn = sqlite3.connect("rankings.db")
    conn.execute(
        """
        CREATE TABLE IF NOT EXISTS serp_rankings (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            checked_at TEXT,
            keyword TEXT,
            position INTEGER,
            title TEXT,
            url TEXT,
            snippet TEXT,
            is_target_domain INTEGER
        )
        """
    )
    conn.executemany(
        """
        INSERT INTO serp_rankings
        (checked_at, keyword, position, title, url, snippet, is_target_domain)
        VALUES (:checked_at, :keyword, :position, :title, :url, :snippet, :is_target_domain)
        """,
        rows,
    )
    conn.commit()
    conn.close()


def fetch_serp() -> dict:
    response = requests.post(
        SERP_ENDPOINT,
        headers=HEADERS,
        json=PAYLOAD,
        timeout=120,
    )
    response.raise_for_status()
    return response.json()


def extract_rows(result: dict) -> list[dict]:
    """Adapt this to the exact schema returned by your provider."""
    keyword = result.get("keyword", "")
    checked_at = datetime.now(timezone.utc).isoformat()
    rows = []

    for item in result.get("organic_results", []):
        url = item.get("url", "")
        domain = urlparse(url).netloc.replace("www.", "")
        rows.append({
            "checked_at": checked_at,
            "keyword": keyword,
            "position": item.get("position"),
            "title": item.get("title", ""),
            "url": url,
            "snippet": item.get("snippet", ""),
            "is_target_domain": 1 if TARGET_DOMAIN in domain else 0,
        })
    return rows


def main() -> None:
    data = fetch_serp()
    all_rows = []

    # Some APIs return one object per keyword; others return a list.
    per_keyword = data if isinstance(data, list) else [data]

    for keyword_result in per_keyword:
        all_rows.extend(extract_rows(keyword_result))
        time.sleep(1)  # Be polite between result sets if you loop manually.

    store_rankings(all_rows)

    target_rows = [r for r in all_rows if r["is_target_domain"]]
    print(f"Stored {len(all_rows)} rows.")
    for row in target_rows:
        print(f"{row['keyword']} -> position {row['position']} ({row['url']})")


if __name__ == "__main__":
    main()

Set the environment variables before running:

export CORECLAW_API_KEY="your_api_key"
export CORECLAW_SERP_ENDPOINT="https://api.coreclaw.com/v1/search"  # copy from console
export TARGET_DOMAIN="yourdomain.com"
python rank_tracker.py

The endpoint value is intentionally an example. Copy the current endpoint from your provider's console or documentation. If you use CoreClaw, the CoreClaw console lists the active endpoint for the Google Search scraper worker.

Scheduling the Tracker

Once the script runs locally, the next step is automation. The simplest option is a cron job or a scheduled task that runs once per day:

# Run every weekday at 9:00 AM UTC
0 9 * * 1-5 cd /path/to/project && python rank_tracker.py >> serp.log 2>&1

For teams that prefer no-code scheduling, wrap the script in a GitHub Actions workflow, an n8n node, or a cloud function. The key constraint is pacing: if you track 100 keywords and each API call handles one keyword, spread the calls across a few minutes rather than hammering the endpoint in a tight loop. If your provider supports batch queries, prefer batching to reduce request count and improve cost efficiency.

Add a small check that skips storage when the API returns an empty result or a non-200 status. Rank tracking data is only useful when it is consistent, so a failed run should alert you instead of silently polluting your history table.

Output Schema Example

After running the script, your rankings.db table might contain rows like this:

checked_at	keyword	position	title	url	is_target_domain
2026-08-10T09:15:00+00:00	best project management software	1	Best Project Management Software of 2026 — Tested	https:// competitor-a.com/...	0
2026-08-10T09:15:00+00:00	best project management software	4	The 10 Best Project Management Software	https://yourdomain.com/...	1
2026-08-10T09:15:00+00:00	asana alternative	2	Top Asana Alternatives for Remote Teams	https://yourdomain.com/...	1

The is_target_domain flag lets you filter for your own rankings instantly. The position field tracks the headline organic result position, which is what most SEO reporting uses.

Business Use Cases

A rank tracker built on a SERP scraper API is useful in a few concrete scenarios:

• Content teams. Track whether new articles break into the top 10 within the first 30–90 days after publication.
• SEO agencies. Run weekly bulk checks across dozens of client domains and export structured data into reporting dashboards.
• E-commerce operators. Monitor product-category keywords such as "best wireless headphones" or "organic dog food" and catch ranking drops before they hurt traffic.
• Competitive analysts. Compare who holds top positions for shared keywords across geographies and devices.

The common thread is that you need structured, repeatable SERP data—not a one-time export.

Build vs. Buy: Three Approaches

Approach	Best for	Setup burden	Maintenance burden	Data freshness
Official APIs (Search Console, Custom Search JSON)	Sites you own; basic query data	Low	Low	Delayed by 1–3 days
Self-hosted scraper with proxies	Teams with infra budget and compliance expertise	High	High	Configurable
Managed SERP scraper API	Teams that want JSON results without proxy work	Low	Low	Near real-time

Official APIs are the safest first step if you only need your own data. Self-hosting gives full control but requires proxy pools, parser maintenance, and rate-limit handling. A managed Google SERP scraper API sits in the middle: you keep control of the query logic and storage while the provider handles execution. If you are comparing options, the CoreClaw pricing page explains the pay-per-result model, which avoids the fixed monthly commitment common to traditional rank-tracking tools.

Limitations, Freshness, and Compliance

No scraper API is a perfect replacement for every SEO workflow. Keep these constraints in mind:

• SERP variance. Google results change by location, device, language, and time of day. Pin those parameters and document them so your history is comparable.
• Layout changes. Google updates result pages constantly. A parser that works today may need adjustment tomorrow. Managed APIs absorb most of this, but output fields can still shift.
• Quota and pacing. Respect provider rate limits and plan your cron schedule around them. Daily checks are common; hourly checks may require a higher throughput plan.
• Regional coverage. Verify that the provider supports the countries and languages you care about. Coverage is usually broad but not universal.
• Legal and terms. Only collect public data, respect robots directives where relevant, and comply with local privacy laws and the provider's terms. Do not use SERP data to impersonate users or evade access controls.

FAQ

Is there an official Google API for competitor rank tracking?
No. Google Search Console shows data for queries your own site already appears for. It does not return full SERP listings for arbitrary keywords or competitor domains. For that, you need a SERP scraper API.

What data fields are returned by a SERP scraper API?
Most providers return organic results with position, title, URL, snippet, and domain. Some also include ads, featured snippets, people-also-ask questions, related searches, knowledge panels, and local packs. Check the provider's schema before building reports.

How often should I run rank checks?
For most sites, daily or weekly is enough. More frequent checks add cost without adding insight unless you are monitoring a volatile keyword or a product launch.

What happens when Google changes its page layout?
A managed provider updates its parser to match new layouts. Self-hosted scrapers break until you update selectors. Either way, validate output fields whenever you see missing data.

Can I connect this to a CRM, n8n, or an AI agent?
Yes. Because the API returns JSON, you can route it into any workflow. Send results to Google Sheets, Airtable, a Postgres database, an n8n flow, or an LLM that summarizes ranking movements.

What should I verify before production use?
Confirm the endpoint, supported locations and devices, rate limits, pricing model, and output schema with your provider. Test with a small keyword set before scaling.

Do I need to manage proxies?
Only if you self-host. Managed SERP scraper APIs typically include proxy rotation, retries, and parsing in the price per request.

Start Tracking Rankings with Less Overhead

A Python rank tracker is simpler than most SEO SaaS tools once you have a reliable data source. The hard part is not the script; it is getting clean, consistent SERP data at scale. If you want to skip proxy setup and focus on analysis, the CoreClaw Google Search scraper returns structured search results through an API, and the CoreClaw Workers store has ready-made scrapers you can deploy without writing infrastructure code.

Related links:

• CoreClaw Google Search scraper
• CoreClaw pricing
• CoreClaw Workers store