DEV Community: Prithwish Nath

How To Validate Any API Response with Great Expectations (GX)

Prithwish Nath — Wed, 15 Apr 2026 05:59:38 +0000

A lot of my data analysis work is forensic (example here). I’ll pull repeated snapshots — content, traffic, SERPs, metadata — and look at patterns across hundreds of these ingestions. So naturally, bad batches will skew results in ways that are just convincing enough so I don’t catch it. Disastrous.

You can add try-catches, but a 200 OK from your API with empty titles and duplicate rows is still considered a success on the wire unless you model validation as its own failure path. So is there a better way?

This is exactly the gap Great Expectations (GX Core) fills. It’s an open-source Python library for defining declarative quality rules on your data — “quality gates”, explicit rules your data must pass before it’s trusted — things like “this field must never be null”, “these IDs must be unique”, or “this value must fall within a known range”. You basically codify what “good” looks like, run it as a validation step in your pipeline, and get a clear pass/fail on every batch — before bad data ever touches your analysis.

Let’s turn that idea into code — we’ll get data at scale via Bright Data (a SERP API), run a GX suite over each batch, then store it in DuckDB (a file-backed analytical SQL database that’s just a .duckdb file on disk) clean rows to the main table, failed batches to quarantine with enough context to debug.

The Setup

Here’s what you need:

requests>=2.28.0  
python-dotenv>=1.0.0  
duckdb>=1.0.0  
pandas>=2.0.0  
psutil>=5.9.0  
great-expectations>=1.0.0

Note that Python 3.10+ is the floor for Great Expectations 1.x with current pandas / duckdb wheels.

The important ones:

great-expectations — the quality gates: expectations on each batch, clear pass/fail and reports. All we need is GX Core.
duckdb — Doesn't need a server to run.
requests —For HTTP to Bright Data’s POST /request endpoint (your zone must be a SERP zone).

Before running pip install -r requirements.txt, create a .env file with:

BRIGHT_DATA_API_KEY=your_api_key  
BRIGHT_DATA_ZONE=serp # or your SERP zone name from the Bright Data dashboard  
BRIGHT_DATA_COUNTRY=us # optional

The Bright Data client reads these when you run the pipeline. You need a SERP-capable zone — without one, POST /request will not match what this code expects. Proxy rotation and unblocking stay on Bright Data’s side.

How Does the Pipeline Actually Work?

This diagram should make it clear.

bright_data.py # Bright Data API client  
serp_expectations.py # Great Expectations validation suite  
duckdb_store.py # DuckDB schema + insert/quarantine logic  
ingest.py # Pipeline: fetch → validate → store or quarantine

The flow is straightforward: one query equals one batch. For each batch, we check things in a strict order before anything touches the database.

Step 1: Fetching Structured SERP Data with Bright Data

Our API client will just be a thin wrapper around Bright Data’s POST /request endpoint.

bright_data.py

import json  
import os  
from dataclasses import dataclass  
from pathlib import Path  

import requests  
from dotenv import load_dotenv  

_GX_ROOT = Path(__file__).resolve().parent  
load_dotenv(_GX_ROOT / ".env")  


def normalize_serp_payload(raw):  
    """Bright Data may wrap JSON in a string body."""  
    if isinstance(raw, dict) and "body" in raw:  
        body = raw["body"]  
        if isinstance(body, str):  
            return json.loads(body)  
        if isinstance(body, dict):  
            return body  
    return raw  


@dataclass  
class SerpApiResponse:  
    """Bright Data POST /request: HTTP status plus parsed JSON body (if any)."""  

    status_code: int  
    data: dict  


class BrightDataClient:  
    """SERP API zone client; uses https://api.brightdata.com/request"""  

    def __init__(self, api_key=None, zone=None, country=None):  
        self.api_key = api_key or os.getenv("BRIGHT_DATA_API_KEY")  
        self.zone = zone or os.getenv("BRIGHT_DATA_ZONE")  
        self.country = country or os.getenv("BRIGHT_DATA_COUNTRY")  
        self.api_endpoint = "https://api.brightdata.com/request"  

        if not self.api_key:  
            raise ValueError("BRIGHT_DATA_API_KEY must be set in the environment or constructor.")  
        if not self.zone:  
            raise ValueError("BRIGHT_DATA_ZONE must be set in the environment or constructor.")  

        self.session = requests.Session()  
        self.session.headers.update(  
            {  
                "Content-Type": "application/json",  
                "Authorization": f"Bearer {self.api_key}",  
            }  
        )  

    def search(self, query, num_results=10, language=None, country=None):  
        """Raises on non-200 or network error."""  
        serp = self.search_with_status(query, num_results, language, country)  
        if serp.status_code != 200:  
            raise RuntimeError(  
                f"Search request failed with HTTP {serp.status_code}: {serp.data!r}"[:500]  
            )  
        return normalize_serp_payload(serp.data)  

    def search_with_status(self, query, num_results=10, language=None, country=None):  
        """Does not raise on non-200 — for ingest + quarantine routing."""  
        search_url = (  
            f"https://www.google.com/search"  
            f"?q={requests.utils.quote(query)}"  
            f"&num={num_results}"  
            f"&brd_json=1"  
        )  
        if language:  
            search_url += f"&hl={language}&lr=lang_{language}"  

        target_country = country or self.country  
        payload = {"zone": self.zone, "url": search_url, "format": "json"}  
        if target_country:  
            payload["country"] = target_country  

        try:  
            response = self.session.post(self.api_endpoint, json=payload, timeout=30)  
        except requests.exceptions.RequestException as e:  
            return SerpApiResponse(status_code=0, data={"_error": str(e)})  

        data = _parse_response_body(response)  
        return SerpApiResponse(status_code=response.status_code, data=data)  


def _parse_response_body(response):  
    if not response.text:  
        return {}  
    try:  
        return response.json()  
    except ValueError:  
        return {"_non_json_body": response.text[:8000]}

Some things to note here:

The search_with_status does not raise on non-200 so ingest can quarantine API failures
Transport errors are caught so you get status_code=0 and the same api_error path as HTTP failures.
Also, search() raises on failure for callers that want exceptions.
And finally,normalize_serp_payload unwraps responses just in case your API puts JSON under a string body.

Step 2: What Happens to Data That Doesn’t Pass?

Every batch in this pipeline has two possible destinations in DuckDB:

If it clears all our defined quality gates, it goes to serp_results.
If it doesn't — wrong HTTP status, empty organic list, or a failed GX expectation — it lands in serp_quarantine instead, with enough context to understand why.

Let's build that store before we wire up the gates.

duckdb_store.py

import hashlib  
import json  
import os  
from datetime import datetime  

import duckdb  
import pandas as pd  
import psutil  


class SerpStore:  
    """serp_results + serp_quarantine"""  

    def __init__(self, db_path, memory_limit=None):  
        parent = os.path.dirname(db_path)  
        if parent:  
            os.makedirs(parent, exist_ok=True)  

        self.db_path = db_path  
        self.conn = duckdb.connect(db_path)  

        if memory_limit:  
            self.conn.execute(f"SET memory_limit='{memory_limit}'")  
        else:  
            available_memory = psutil.virtual_memory().available  
            memory_gb = int(available_memory / (1024**3) * 0.8)  
            self.conn.execute(f"SET memory_limit='{memory_gb}GB'")  

        self._create_schema()  

    def _create_schema(self):  
        self.conn.execute("""  
            CREATE TABLE IF NOT EXISTS serp_results (  
                id BIGINT PRIMARY KEY,  
                query TEXT NOT NULL,  
                timestamp TIMESTAMP NOT NULL,  
                result_position INTEGER NOT NULL,  
                title TEXT,  
                url TEXT,  
                snippet TEXT,  
                domain TEXT,  
                rank INTEGER,  
                previous_rank INTEGER,  
                rank_delta INTEGER  
            )  
        """)  
        self.conn.execute("CREATE INDEX IF NOT EXISTS idx_query ON serp_results(query)")  
        self.conn.execute("CREATE INDEX IF NOT EXISTS idx_domain ON serp_results(domain)")  

        self.conn.execute("""  
            CREATE TABLE IF NOT EXISTS serp_quarantine (  
                query TEXT,  
                timestamp TIMESTAMP,  
                reason TEXT,  
                organic_count INTEGER,  
                payload_hash TEXT,  
                http_status INTEGER,  
                raw_json JSON  
            )  
        """)  

    @staticmethod  
    def _payload_hash(payload):  
        canonical = json.dumps(payload, sort_keys=True, default=str)  
        return hashlib.sha256(canonical.encode()).hexdigest()  

    def insert_quarantine(        self,  
        query,  
        reason,  
        organic_count,  
        raw_json,  
        http_status,  
        timestamp=None,    ):  
        if timestamp is None:  
            timestamp = datetime.now()  
        payload_hash = self._payload_hash(raw_json)  
        self.conn.execute(  
            """  
            INSERT INTO serp_quarantine  
                (query, timestamp, reason, organic_count, payload_hash, http_status, raw_json)  
            VALUES (?, ?, ?, ?, ?, ?, ?)  
            """,  
            [ 
                query,  
                timestamp,  
                reason,  
                organic_count,  
                payload_hash,  
                http_status,  
                json.dumps(raw_json, default=str),  
            ],  
        )  

    def insert_batch(self, vdf, query, timestamp=None):  
        """Insert from the GX validation frame (output of organic_to_validation_df).  

        Store what we validated: normalized url, derived domain, snippet, and positional rank —  
        not a second parse from raw organic dicts (avoids drift vs Great Expectations).  
        """  
        if timestamp is None:  
            timestamp = datetime.now()  
        if vdf.empty:  
            return  

        max_id_result = self.conn.execute("SELECT COALESCE(MAX(id), 0) FROM serp_results").fetchone()  
        next_id = (max_id_result[0] if max_id_result else 0) + 1  

        rows = []  
        for idx, (_, row) in enumerate(vdf.iterrows()):  
            title = "" if pd.isna(row["title"]) else str(row["title"])  
            url = "" if pd.isna(row["url"]) else str(row["url"])  
            snippet = "" if pd.isna(row["snippet"]) else str(row["snippet"])  
            domain = "" if pd.isna(row["domain"]) else str(row["domain"])  
            rank_val = int(row["rank"])  
            rows.append(  
                {  
                    "id": next_id + idx,  
                    "query": query,  
                    "timestamp": timestamp,  
                    "result_position": idx + 1,  
                    "title": title,  
                    "url": url,  
                    "snippet": snippet,  
                    "domain": domain,  
                    "rank": rank_val,  
                    "previous_rank": None,  
                    "rank_delta": None,  
                }  
            )  

        df = pd.DataFrame(rows)  
        self.conn.execute("""  
            INSERT INTO serp_results (id, query, timestamp, result_position, title, url, snippet, domain, rank, previous_rank, rank_delta)  
            SELECT id, query, timestamp, result_position, title, url, snippet, domain, rank, previous_rank, rank_delta FROM df  
        """)  

    def get_row_count(self):  
        result = self.conn.execute("SELECT COUNT(*) FROM serp_results").fetchone()  
        return result[0] if result else 0  

    def close(self):  
        self.conn.close()  

    def __enter__(self):  
        return self  

    def __exit__(self, *args):  
        self.close()

What’s happening here?

The reason field is an enum-like string: api_error, organic_empty, or validation_failed.
The payload_hash is a SHA-256 of the canonical JSON — so you can tell if the same bad payload keeps coming back (very useful!)
Each row stores a JSON blob in raw_json: for api_error and organic_empty it is the normalized SERP dict; for validation_failed it is {"serp":[normalized serp here], "gx_validation":[GX result dict here]} so you still have both the SERP payload and the failing expectation details. Every bad batch lands in serp_quarantine with a paper trail instead of silently disappearing or, worse, silently making it into serp_results. When a batch passes all gates, insert_batch writes one row per organic result from the validated DataFrame — same normalized fields GX validated. previous_rank / rank_delta are reserved for later rank-tracking; they stay null on first ingest.

Step 3: The Gate Order — Why Sequence Matters

Bright Data's parsed SERP JSON path (request URL includes brd_json=1 or, in their docs, brd_json=json) returns the same kind of structured object you see in their examples and in real dumps — organic, general, input, and so on — not a page of HTML you parse yourself.

So when Google can't be reached or nothing usable is extracted, that tends to show up as a non-200 from POST /request, a network error (we treat as api_error), or an empty organic array —and not as a captcha HTML document mistaken for a normal organic list. So the gates to implement are exactly three:

non-200,
empty organic,
then GX on whatever is left.

ingest.py

import json  
from datetime import datetime, timezone  
from pathlib import Path  

_ROOT = Path(__file__).resolve().parent  

from bright_data import BrightDataClient, normalize_serp_payload  
from duckdb_store import SerpStore  
from serp_expectations import organic_to_validation_df, validate_organic_batch

def _write_validation_report(report_entries):  
    reports_dir = _ROOT / "data" / "reports"  
    reports_dir.mkdir(parents=True, exist_ok=True)  
    ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")  
    path = reports_dir / f"validation_{ts}.json"  
    doc = {  
        "generated_at_utc": ts,  
        "batches": report_entries,  
    }  
    path.write_text(json.dumps(doc, indent=2, default=str), encoding="utf-8")  
    return path  


def ingest_live(    queries,  
    *,  
    db_path=None,  
    num_results=10,  
    memory_limit="2GB",  
    serp_json_out=None,):  
    """  
    One batch = one query → one SERP response.  
    Order: api_error → organic_empty → GX validate → insert or validation_failed quarantine.  
    """  
    path = db_path or str(_ROOT / "data" / "serp_gx.duckdb")  
    client = BrightDataClient()  
    total = 0  
    batches_total = 0  
    empty_batches = 0  
    api_error_batches = 0  
    validation_failed_batches = 0  
    report_entries = []  
    serp_dump_batches = []  

    with SerpStore(path, memory_limit=memory_limit) as store:  
        for q in queries:  
            batches_total += 1  
            serp = client.search_with_status(q, num_results=num_results)  
            raw = normalize_serp_payload(serp.data)  
            if serp_json_out:  
                serp_dump_batches.append(  
                    {  
                        "query": q,  
                        "http_status": serp.status_code,  
                        "normalized_serp": raw,  
                    }  
                )  
            organic = raw.get("organic") or []  
            organic_count = len(organic)  

            # Optional (not implemented): best-effort catch-all for an explicit blocked API response  
            # (e.g. substring-match json.dumps(raw) for "captcha" / "unusual traffic") — just in case.  
            # Not needed here, only include if the API you're using can error out like that  

            if serp.status_code != 200:  
                api_error_batches += 1  
                store.insert_quarantine(q, "api_error", organic_count, raw, serp.status_code)  
                report_entries.append(  
                    {  
                        "query": q,  
                        "outcome": "api_error",  
                        "http_status": serp.status_code,  
                        "organic_count": organic_count,  
                        "gx": None,  
                    }  
                )  
                continue  
            if not organic:  
                empty_batches += 1  
                store.insert_quarantine(q, "organic_empty", 0, raw, serp.status_code)  
                report_entries.append(  
                    {  
                        "query": q,  
                        "outcome": "organic_empty",  
                        "http_status": serp.status_code,  
                        "organic_count": 0,  
                        "gx": None,  
                    }  
                )  
                continue  

            vdf = organic_to_validation_df(organic)  
            gx_ok, gx_payload = validate_organic_batch(vdf, num_results=num_results)  
            if not gx_ok:  
                validation_failed_batches += 1  
                store.insert_quarantine(  
                    q,  
                    "validation_failed",  
                    organic_count,  
                    {"serp": raw, "gx_validation": gx_payload},  
                    serp.status_code,  
                )  
                report_entries.append(  
                    {  
                        "query": q,  
                        "outcome": "validation_failed",  
                        "http_status": serp.status_code,  
                        "organic_count": organic_count,  
                        "gx": gx_payload,  
                    }  
                )  
                continue  

            # Persist the validated DataFrame so DuckDB gets the same normalized url/domain/rank GX checked.  
            store.insert_batch(vdf, q)  
            total += len(vdf)  
            report_entries.append(  
                {  
                    "query": q,  
                    "outcome": "inserted",  
                    "http_status": serp.status_code,  
                    "organic_count": organic_count,  
                    "gx": gx_payload,  
                }  
            )  

        row_count = store.get_row_count()  

    report_path = _write_validation_report(report_entries)  

    stats = {  
        "batches_total": batches_total,  
        "empty_batches": empty_batches,  
        "api_error_batches": api_error_batches,  
        "validation_failed_batches": validation_failed_batches,  
        "empty_rate": (empty_batches / batches_total) if batches_total else 0.0,  
        "api_error_rate": (api_error_batches / batches_total) if batches_total else 0.0,  
        "validation_failed_rate": (validation_failed_batches / batches_total)  
        if batches_total  
        else 0.0,  
        "rows_ingested": total,  
        "serp_results_row_count": row_count,  
        "db_path": path,  
        "validation_report_path": str(report_path),  
    }  
    if serp_json_out:  
        out_path = Path(serp_json_out)  
        out_path.parent.mkdir(parents=True, exist_ok=True)  
        ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")  
        doc = {  
            "generated_at_utc": ts,  
            "num_results": num_results,  
            "batches": serp_dump_batches,  
        }  
        out_path.write_text(json.dumps(doc, indent=2, default=str), encoding="utf-8")  
        stats["serp_json_path"] = str(out_path.resolve())  

    return stats  


def main():  
    import argparse  

    p = argparse.ArgumentParser(  
        description="gx: Bright Data SERP → Great Expectations → DuckDB (fail fast on GX failure)",  
    )  
    p.add_argument(  
        "queries",  
        nargs="*",  
        default=[ 
            "inference engineering",  
            "python duckdb",  
            "Great Expectations data validation",  
            "machine learning",  
            "opentelemetry how to",  
        ],  
        help\="Search queries (default: five example queries if none given)",  
    )  
    p.add_argument("--num-results", type\=int, default=10)  
    p.add_argument("--db", type\=str, default=None, help\="DuckDB file path")  
    p.add_argument(  
        "--save-serp-json",  
        type\=str,  
        default=None,  
        metavar="PATH",  
        help\="Write normalized SERP (+ query, http_status) per batch to this JSON file.",  
    )  
    args = p.parse_args()  
    out = ingest_live(  
        args.queries,  
        db_path=args.db,  
        num_results=args.num_results,  
        serp_json_out=args.save_serp_json,  
    )  
    print(json.dumps(out, indent=2))  


if __name__ == "__main__":  
    main()

The main loop enforces that order we talked about: api_error and organic_empty are cheap pre-checks that catch the most common failure modes without spinning up GX at all.

In fact, GX only runs when there's actual data to validate. store.insert_batch(vdf, q) passes the validated DataFrame, not the original raw organic list — what goes into DuckDB is exactly what GX checked, with the same URL normalization applied.

Step 4: What Quality Gates Should You Put on SERP Data?

To reiterate, when we say “Quality gates”, we mean explicit Great Expectations rules on the DataFrame we validate: each “expectation” is simply one condition the batch must satisfy before you trust it for analytics.

For this pipeline we normalize the organiclist in our API response into a small schema — url, title, snippet, domain, and rank — and run the suite on that frame only.

serp_expectations.py

import uuid  
from urllib.parse import urlparse  

import great_expectations as gx  
import pandas as pd  
from great_expectations.data_context.types.base import ProgressBarsConfig  


# helpers   
def _extract_domain(url):  
    if not url:  
        return ""  
    return urlparse(url).netloc.replace("www.", "")  


def _coerce_rank(raw_rank, fallback):  
    if raw_rank is None:  
        return fallback  
    try:  
        return int(float(raw_rank))  
    except (TypeError, ValueError):  
        return fallback  


def _normalize_url(url):  
    if not url:  
        return url  
    try:  
        parsed = urlparse(url)  
        normalised = parsed._replace(  
            scheme=parsed.scheme.lower(),  
            fragment="",  
        )  
        return normalised.geturl()  
    except Exception:  
        return url  

# before anything, normalize the data  
def organic_to_validation_df(organic):  
    rows = []  
    for i, r in enumerate(organic):  
        raw_url = (r.get("url") or r.get("link") or "").strip()  
        url = _normalize_url(raw_url)  
        title = (r.get("title") or "").strip()  
        snippet = (r.get("snippet") or r.get("description") or "").strip()  
        domain = _extract_domain(url)  
        raw_rank = r.get("rank")  
        rank = _coerce_rank(raw_rank, fallback=i + 1)  
        rows.append(  
            {  
                "url": url,  
                "title": title,  
                "snippet": snippet,  
                "domain": domain,  
                "rank": rank,  
            }  
        )  
    return pd.DataFrame(rows)  

# now add the actual expectations  
def build_serp_expectation_suite(num_results):  
    _meta = {"notes": "SERP organic batch gates for Bright Data brd_json=1 payloads."}  

    return gx.ExpectationSuite(  
        name="serp_organic_batch",  
        expectations=[ 
            gx.expectations.ExpectColumnValuesToNotBeNull(  
                column="url",  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValuesToNotBeNull(  
                column="title",  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValuesToMatchRegex(  
                column="url",  
                regex=r"^https?://\\S+",  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValuesToBeUnique(  
                column="url",  
                meta=_meta,  
            ),  
            gx.expectations.ExpectTableRowCountToBeBetween(  
                min_value=1,  
                max_value=num_results,  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValueLengthsToBeBetween(  
                column="title",  
                min_value=1,  
                max_value=500,  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValuesToMatchRegex(  
                column="title",  
                regex=r"\\S",  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValueLengthsToBeBetween(  
                column="domain",  
                min_value=1,  
                max_value=253,  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValuesToBeBetween(  
                column="rank",  
                min_value=1,  
                max_value=float(num_results),  
                meta=_meta,  
            ),  
            gx.expectations.ExpectColumnValuesToNotBeNull(  
                column="snippet",  
                mostly=0.8,  
                meta=_meta,  
            ),  
        ],  
    )  

# gx in ephemeral mode  
def validate_organic_batch(df, *, num_results):  
    if df.empty:  
        return False, {  
            "success": False,  
            "statistics": {  
                "evaluated_expectations": 0,  
                "successful_expectations": 0,  
                "unsuccessful_expectations": 0,  
            },  
            "results": [],  
            "exception_message": "validation_frame_empty",  
        }  

    context = gx.get_context(mode="ephemeral")  
    context.variables.progress_bars = ProgressBarsConfig(  
        globally=False,  
        metric_calculations=False,  
    )  

    data_source = context.data_sources.add_pandas(f"serp_{uuid.uuid4().hex[:12]}")  
    asset = data_source.add_dataframe_asset(name="organic_batch")  
    batch_definition = asset.add_batch_definition_whole_dataframe("whole")  
    batch = batch_definition.get_batch(batch_parameters={"dataframe": df})  
    suite = build_serp_expectation_suite(num_results=num_results)  
    result = batch.validate(suite)  
    payload = result.to_json_dict()  
    return bool(result.success), payload

Let’s walk through the decisions we made here, because they’re not arbitrary.

Normalize Before You Validate — Never Inside the Validation Layer

Before anything in GX runs, organic_to_validation_df normalizes the data:

def _normalize_url(url: str) -> str:  
    """Lowercase scheme, strip fragment."""  
    parsed = urlparse(url)  
    return parsed._replace(scheme=parsed.scheme.lower(), fragment="").geturl()  
def organic_to_validation_df(organic: List[Dict[str, Any]]) -> pd.DataFrame:  
    rows = []  
    for i, r in enumerate(organic):  
        raw_url = (r.get("url") or r.get("link") or "").strip()  
        url = _normalize_url(raw_url)  
        title = (r.get("title") or "").strip()  
        snippet = (r.get("snippet") or r.get("description") or "").strip()  
        domain = _extract_domain(url)  
        rank = _coerce_rank(r.get("rank"), fallback=i + 1)  
        rows.append({"url": url, "title": title, "snippet": snippet, "domain": domain, "rank": rank})  
    return pd.DataFrame(rows)

A few things happening here:

Bright Data’s brd_json=1 responses use link as the key, not url. We normalize this before GX sees it. You’ve probably used such remappings plenty of times, for most API responses.
Fragments are stripped and scheme is lowercased before the uniqueness check — so https://example.com/page#section and https://example.com/page don't pass as different URLs.
_coerce_rank handles the fact that rank values from APIs can come back as int, float, numpy scalars, or even strings. Coerce to a consistent type before GX, not inside an expectation.

Ephemeral mode

The [mode="ephemeral"] simply means no config files, no persisted context directory, and no great_expectations.yml — a pure in-process validator you spin up per batch. Progress bars are turned off in code when batching many runs. We do this to make GX lightweight and easier to get started with.

Null Checks Must Come Before Regex Gate

ExpectColumnValuesToNotBeNull on url and title must come before the regex and length gates. That's because GX's ExpectColumnValuesToMatchRegex silently skips null values by default — it only evaluates non-null rows. Without an explicit null check, a null URL would slip through the regex gate undetected.

Why the URL Regex Is Intentionally Loose

^https?://\S+ is deliberately not a strict RFC 3986 URL validator. Overly strict URL validation generates false positives on legitimately unusual URLs — CDN URLs, tracking URLs, URLs with encoded characters. The goal here is to catch the two actual failure modes: an empty string and a non-URL value like "N/A" or a relative path.

URL Uniqueness Catches Parse Artifacts

ExpectColumnValuesToBeUnique on url catches cases where the same URL appears twice in a single batch. That's a real thing that can happen with pagination artifacts or certain response formats. In a ranking pipeline, a duplicate URL means your rank counts are wrong.

Use a Fixed Ceiling for Rank and Row Count, Not a Self-Adjusting One

Notice that max_value=float(num_results) for the rank gate, and max_value=num_results for the row count, both use the same value — the same number we used for our SERP API. This is intentional and it matters.

A common mistake is to derive the ceiling from the batch itself:

# Don't do this  
max_rank = max(num_results, int(float(vdf["rank"].max())))

Why? Well, if the API returns a result with rank=47 for a 10-result query, max_rank becomes 47 and the gate passes. You've made the gate self-adjust to whatever the data says, which means the gate can never actually fail on an out-of-range rank. Using a fixed num_results means an unexpected rank value will actually trigger a quarantine.

Optional Fields Need Soft Gates, Not Hard Ones

snippet uses mostly=0.8 rather than a hard non-null gate. That's because Google legitimately suppresses snippets for some result types — videos, certain knowledge panel entries, sitelinks. A hard non-null gate on snippet would quarantine perfectly valid batches. The mostly parameter lets you say "80% of rows must pass this check" — if more than 20% of rows have no snippet, that's a signal the response structure has changed, not normal Google behaviour.

What Does a Failed GX Validation Actually Look Like?

When a batch fails validation, the gx block in the report tells you exactly which expectation failed and what the unexpected values were. Here's a representative example — two organic rows end up with the same normalized URL (duplicate rows in the response, or two URLs that collapse to one after fragment stripping), so the uniqueness gate fires:

{  
  "query": "inference engineering",  
  "outcome": "validation_failed",  
  "http_status": 200,  
  "organic_count": 2,  
  "gx": {  
    "success": false,  
    "statistics": {  
      "evaluated_expectations": 10,  
      "successful_expectations": 9,  
      "unsuccessful_expectations": 1  
    },  
    "results": [ 
      {  
        "success": false,  
        "expectation_config": {  
          "type": "expect_column_values_to_be_unique",  
          "kwargs": { "column": "url" }  
        },  
        "result": {  
          "element_count": 2,  
          "unexpected_count": 2,  
          "unexpected_percent": 100.0,  
          "partial_unexpected_list": [ 
            "https://example.com/page?q=1",  
            "https://example.com/page?q=1"  
          ]  
        }  
      }  
    ]  
  }  
}

The http_status is 200. Without the quality gate, this batch would have been inserted. The downstream join on URL would have silently doubled a result's apparent frequency in your analytics.

The Validation Report

After every run, we emit a timestamped JSON report with one entry per query:

{  
  "generated_at_utc": "20260326T102005Z",  
  "batches": [ 
    {  
      "query": "inference engineering",  
      "outcome": "inserted",  
      "http_status": 200,  
      "organic_count": 8,  
      "gx": { "success": true, "statistics": { "evaluated_expectations": 10, ... } }  
    },  
    {  
      "query": "some broken query",  
      "outcome": "organic_empty",  
      "http_status": 200,  
      "organic_count": 0,  
      "gx": null  
    }  
  ]  
}

The possible outcome values are inserted, api_error, organic_empty, and validation_failed. Over time, watching the rate of each outcome per query is how you'd catch gradual degradation — a rising organic_empty rate is a signal before it becomes a crisis.

How to Run the Pipeline

Put the four modules in one package or folder on your PYTHONPATH, install dependencies (pip install -r requirements.txt), and set Bright Data credentials (BRIGHT_DATA_API_KEY, BRIGHT_DATA_ZONE, plus optional BRIGHT_DATA_COUNTRY) in a .env file next to the code or in the environment.

Our orchestrator uses argparse. So here’s some useful flags worth including, mostly for quality-of-life:

--num-results (default 10),
--db (override the DuckDB file path),
and optionally, I’ve found using a --save-serp-json PATH to dump normalized SERP payloads while tuning your rules really helps.

# One query  
python ingest.py "inference engineering"  

# Multiple queries (defaults to five example queries if you pass none)  
python ingest.py  

# Example: 20 results, custom DB path  
python ingest.py "python asyncio" --num-results 20 --db ./data/my_run.duckdb

On success, the script prints a small JSON summary — batch counts, rows written, paths to the DuckDB file and the validation report:

{  
  "batches_total": 1,  
  "empty_batches": 0,  
  "api_error_batches": 0,  
  "validation_failed_batches": 0,  
  "empty_rate": 0.0,  
  "api_error_rate": 0.0,  
  "validation_failed_rate": 0.0,  
  "rows_ingested": 8,  
  "serp_results_row_count": 8,  
  "db_path": "./data/serp_gx.duckdb",  
  "validation_report_path": "./data/reports/validation_20260326T102005Z.json"  
}

That’s pretty much everything. I’ve walked you through each file. Just know ingest.py is the entry point.

Frequently Asked Questions (FAQ)

Q: Why not just write the validation logic myself?

A: You can, and for one or two checks, you probably should! The case for GX is not that something likeif not url or not url.startswith("http") is hard to write — it's that twenty of those checks scattered across your pipeline become hard to read, hard to audit, and easy to accidentally skip when you're in a hurry. GX gives you a single place where all your rules live, a consistent result structure across every check, and a report that tells you exactly which rule failed and on which values. It saves a ton of trouble in the long run.

Q: Does running GX on every batch slow the pipeline down?

A: In practice, no — not at the batch sizes typical of API calls (let’s say 10–50 rows per query). To make double sure, this is why I run GX in Ephemeral context mode (mode=”ephemeral”) — that avoids any file I/O or context persistence overhead. The validation itself is pandas operations under the hood anyway.

Q: What happens to bad data in a pipeline without validation?

A: It gets inserted and you don’t find out until something downstream breaks — or worse, until it produces wrong answers that are just plausible enough to go unnoticed. A 200 OK with duplicate rows, empty titles, or out-of-range values looks like a success on the wire. Without an explicit quality gate, that batch goes straight into your database and silently skews every query that touches it.

Q: How do I know what rules to write for my own data?

A: Sample first, then write rules. Pull a handful of real responses from your source API, inspect the fields and edge cases, then encode what “good” looks like. Rules written speculatively against an imagined schema generate false positives; rules written against real samples catch actual failure modes. Start with the fields you’d join on or aggregate in your analytics, and only add gates for everything else if you’ve seen it break.

What GX + Bright Data Gives You

So why does this pattern work reliably?

Bright Data handles the upstream layer. Proxy rotation, bot detection, structured extraction — all abstracted behind a single API call that returns structured JSON.
Great Expectations handles downstream trust. It doesn’t fix bad data. It measures whether incoming data meets your rules before you trust it with your analytics.
The quarantine table (in DuckDB) gives you an audit trail. Not just “something went wrong,” but what the payload was, which expectations failed, and when. You can query the quarantine table to understand your pipeline’s health over time.

Web pipelines that don’t have explicit quality gates just gradually produce wrong answers. Catching that before it reaches your database is the whole point.

Why You Should Add Observability to Your Data Extraction with OpenTelemetry

Prithwish Nath — Mon, 06 Apr 2026 03:54:52 +0000

TL;DR: This is a step-by-step tutorial on the quickest way to add observability to any data ingestion pipeline — whether you’re scraping or using an API.

Anything that fetches data at scale has a class of failure that error handling won’t catch. Not because your error handling code is bad (it probably isn’t) but because retries that eventually succeed, queries that take 10x longer than average, and domains that silently time out — don’t throw exceptions because they’re not technically errors. And you’ll never know. The solution is actually adding proper observability.

Overkill? Not at all. Because a data pipeline — any data pipeline — with network calls, retries, timeouts, and wildly variable latency across different queries and domains is a textbook distributed system. It has all the same failure modes, and so it deserves the same tooling.

In this post, we’ll build a SERP pipeline on top of Bright Data’s API and instrument it with OpenTelemetry (See: Python docs), the open-source standard for distributed tracing. Bright Data reduces blocks and proxy headaches out of the box — but proper Otel tracing shows you exactly where risk remains.

By the end, you’ll be able to see what each call costs you in time, where retries are hiding, and which queries are slow.

What This Actually Gets You

What I’m trying to do is surface problems you’ll probably run into, and otherwise just silently pay for. These patterns map nearly 1:1 to wherever your data ingest pipelines look like in production.

Retry storm detection. If a domain starts blocking aggressively, you won’t see it as hard errors anymore, but a creeping spike in scraper.retries > 0 spans. That’s your early warning before you trigger a full ban or blow past your proxy quota for the month.
Actual cost visibility. Every retry is another proxy request. If you’re paying per request or per GB, scraper.retries on your spans maps directly to a line item on your invoice. You can aggregate this and alert on it — I haven’t been doing this before adding OTel, and most likely, neither have you. 😅
Per-query latency profiling. Some queries are just structurally slower — more competitive terms, heavier result pages, more contention in the proxy pool. Traces let you see this per-query instead of as a blended average that makes everything look fine. Once you can see the outliers, you can do something about them.

Basically, if you take ONE thing away from this read, let it be this: data pipelines have exactly the same failure modes as any distributed system — timeouts, partial failures, retry amplification, silent degradation — whether data is obtained via an API call or just scraping.

So let’s build a data collection stack you can reason about. And, as it turns out, the tooling you’d use for microservices works perfectly well here too.

The Setup

Here’s what you need:

opentelemetry-api>=1.20.0  
opentelemetry-sdk>=1.20.0  
opentelemetry-instrumentation-requests>=0.41b0  
opentelemetry-exporter-otlp-proto-http>=1.20.0  
requests>=2.28.0  
python-dotenv>=1.0.0

The important ones:

opentelemetry-instrumentation-requests — this gives us automatic HTTP tracing. Zero manual work.
opentelemetry-exporter-otlp-proto-http — for when we want to send traces somewhere real, like Jaeger.

Before running pip install -r requirements.txt, create a .env file with:

BRIGHT_DATA_API_KEY=your_api_key  
BRIGHT_DATA_ZONE=serp # or your SERP zone name from Bright Data dashboard  
BRIGHT_DATA_COUNTRY=us # optional  
OTEL_EXPORTER=console   # set to "jaeger" to send traces to Jaeger (must be running)

The client reads these on instantiation. Replace with your own API credentials if you need them, but don’t forget OTEL_EXPORTER — it controls where traces go.

Initializing OpenTelemetry

We want two modes: a console exporter for development where traces print right in the terminal, and an OTLP exporter for production. A single env var switches between them:

import os  
from opentelemetry import trace  
from opentelemetry.instrumentation.requests import RequestsInstrumentor  
from opentelemetry.sdk.trace import TracerProvider  
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter  
from opentelemetry.sdk.resources import SERVICE_NAME, Resource  


def init_otel(exporter: str = "console", service_name: str = "bd-scraper"):  
    resource = Resource.create(attributes={SERVICE_NAME: service_name})  
    provider = TracerProvider(resource=resource)  

    if exporter == "jaeger":  
        from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter  
        endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318")  
        processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces"))  
    else:  
        processor = BatchSpanProcessor(ConsoleSpanExporter())  

    provider.add_span_processor(processor)  
    trace.set_tracer_provider(provider)  
    RequestsInstrumentor().instrument()  # this is where the magic happens!  
    return trace.get_tracer(service_name, "1.0.0")

That one line — RequestsInstrumentor().instrument() — hooks into the requests library globally. Every HTTP call your code makes from this point forward gets a trace span, including the ones in third-party code you didn’t write. You get that for free.

One thing that’ll catch you out if you’re not careful: init_otel must run before any requests.Session is created. That means calling it before importing BrightDataClient in your entrypoint. Yes, the import order matters here.

The Client with Custom Spans

Automatic HTTP tracing is great, but it only tells you about the transport layer. It has no idea this call was for the query “machine learning”, or that it targetedgoogle.com, or that it had to retry once before it worked. That context is what custom spans are for.

import json  
import os  
import time  
import requests  
from typing import Optional  

from dotenv import load_dotenv  
load_dotenv()  


class BrightDataClient:  
    def __init__(        self,  
        api_key: Optional[str] = None,  
        zone: Optional[str] = None,  
        country: Optional[str] = None,    ):  
        self.api_key = api_key or os.getenv("BRIGHT_DATA_API_KEY")  
        self.zone = zone or os.getenv("BRIGHT_DATA_ZONE")  
        self.country = country or os.getenv("BRIGHT_DATA_COUNTRY")  
        self.api_endpoint = "https://api.brightdata.com/request"  

        if not self.api_key or not self.zone:  
            raise ValueError("BRIGHT_DATA_API_KEY and BRIGHT_DATA_ZONE required (env or constructor)")  

        self.session = requests.Session()  
        self.session.headers.update({  
            "Content-Type": "application/json",  
            "Authorization": f"Bearer {self.api_key}",  
        })  

    def search(        self,  
        query: str,  
        num_results: int = 10,  
        language: Optional[str] = None,  
        country: Optional[str] = None,  
        max_retries: int = 2,    ) -> dict:  
        from opentelemetry import trace  
        from opentelemetry.trace import StatusCode  

        tracer = trace.get_tracer(__name__, "1.0.0")  
        target_domain = "google.com"  

        with tracer.start_as_current_span("bright_data.search") as span:  
            span.set_attribute("scraper.query", query)  
            span.set_attribute("scraper.target_domain", target_domain)  
            span.set_attribute("scraper.num_results", num_results)  

            start = time.perf_counter()  
            last_err = None  

            for attempt in range(max_retries + 1):  
                try:  
                    result = self._do_search(query, num_results, language, country)  
                    latency_ms = (time.perf_counter() - start) * 1000  
                    span.set_attribute("scraper.latency_ms", round(latency_ms, 2))  
                    span.set_attribute("scraper.retries", attempt)  
                    # clean success — no retries needed  
                    if attempt == 0:  
                        span.set_status(StatusCode.OK)  
                    else:  
                        # recovered, but we want this surfaced in Jaeger  
                        span.set_status(StatusCode.ERROR, "Recovered after retry")  
                    return result  
                except Exception as e:  
                    last_err = e  
                    span.set_attribute("scraper.retries", attempt + 1)  
                    if attempt `< max_retries:  
                        time.sleep(0.5 * (attempt + 1))  

            # all retries exhausted  
            span.set_attribute("scraper.error", str(last_err))  
            span.set_status(StatusCode.ERROR, str(last_err))  
            span.record_exception(last_err)  
            raise last_err  

    def _do_search(        self,  
        query: str,  
        num_results: int,  
        language: Optional[str],  
        country: Optional[str],    ) ->` dict:  
        search_url = (  
            f"https://www.google.com/search"  
            f"?q={requests.utils.quote(query)}&num={num_results}&brd_json=1"  
        )  
        if language:  
            search_url += f"&hl={language}&lr=lang_{language}"  
        target_country = country or self.country  
        payload = {"zone": self.zone, "url": search_url, "format": "json"}  
        if target_country:  
            payload["country"] = target_country  

        response = self.session.post(self.api_endpoint, json=payload, timeout=30)  
        response.raise_for_status()  
        result = response.json()  
        # Bright Data may return body as JSON string — unpack it  
        if isinstance(result, dict) and "body" in result:  
            body = result["body"]  
            result = json.loads(body) if isinstance(body, str) else body  
        return result

I’m using a SERP API for data, but swap it out with whatever you’re using. The concepts apply to anything. Also, a couple of things worth understanding here:

The Parent-Child Relationship

The _do_search helper builds the target URL and POSTs it to our API endpoint (api.brightdata.com/request). When that call runs, the RequestsInstrumentorauto-creates a child POST span inside our bright_data.search parent span. They share the same trace_id.

In Jaeger, you’ll get a proper timeline: the outer business operation wrapping the inner HTTP call. That nesting is what makes traces actually useful — you see the whole story, not just individual events.

You Have to Set Span Status Yourself

This one surprised me. OTel records all the data you throw at it, but it won’t decide what matters on your behalf. If you don’t explicitly call [span.set_status(…)](https://opentelemetry.io/docs/languages/python/instrumentation/#set-span-status), every span stays UNSET— even when a retry happened underneath. A query that timed out, retried, and recovered would be completely invisible to a Jaeger filter like status=ERROR. You’d never find it.

So there’s a deliberate tradeoff we’re making in the code above: recovered retries are marked ERRORso they show up in dashboards. Some teams prefer to use OKand add a scraper.recovered = true attribute instead, keeping error rate metrics clean.

Honestly, both are fine 🤷‍♂️ It just depends on whether you want alerting to treat “degraded success” as a failure. The important thing is to choose consciously, and not fall through to UNSETby accident.

Putting It All Together

Let’s do this in a file, call it something like scraper.py

import argparse  
import os  
import time  

from dotenv import load_dotenv  
load_dotenv()  
_exporter = os.getenv("OTEL_EXPORTER", "console") # console fallback as a default  

from otel_config import init_otel  
init_otel(exporter=_exporter)  # must come before BrightDataClient import  

from bright_data_otel import BrightDataClient  


def run(calls: int = 10, delay: float = 0.5):  
    queries = [  
        "python programming", "machine learning", "web development",  
        "data science", "cloud computing",  
    ]  
    client = BrightDataClient()  
    start = time.time()  
    for i in range(calls):  
        q = queries[i % len(queries)]  
        try:  
            data = client.search(q, num_results=5)  
            n = len(data.get("organic", [])) if isinstance(data, dict) else 0  
            print(f"  [{i+1}/{calls}] {q}: {n} results")  
        except Exception as e:  
            print(f"  [{i+1}/{calls}] {q}: error — {e}")  
        if i `< calls - 1:  
            time.sleep(delay)  
    print(f"Done in {time.time() - start:.1f}s")  


if __name__ == "__main__":  
    p = argparse.ArgumentParser()  
    p.add_argument("--count", type=int, default=10)  
    p.add_argument("--delay", type=float, default=0.5)  
    args = p.parse_args()  
    run(calls=args.count, delay=args.delay)

Run it in console mode:

> python scraper.py --count 5

Or point it at Jaeger:

# to be honest, just set OTEL_EXPORTER in .env  
> OTEL_EXPORTER=jaeger python scraper.py --count 5

What the Traces Actually Show

Here’s what the terminal printed for my five-query run:

[1/5] python programming: 6 results  
[2/5] machine learning: 8 results  
[3/5] web development: 9 results  
[4/5] data science: 9 results  
[5/5] cloud computing: 9 results  
Done in 75.1s

Five queries, all with results, no errors printed. Looks perfectly healthy….right? Let’s see what the traces say.

The clean calls…

For python programming, the bright_data.searchspan looks exactly as expected:

{  
  "name": "bright_data.search",  
  "attributes": {  
    "scraper.query": "python programming",  
    "scraper.target_domain": "google.com",  
    "scraper.latency_ms": 3686.01,  
    "scraper.retries": 0  
  }

3.7 seconds, zero retries, one nested POST span confirming the HTTP round-trip happened exactly once. Looks good! Moving on.

…and the ones that weren’t so clean.

The query data science printed 9 results. Except the traces show three spans for that single call:

{  
  "name": "POST",  
  "status": {  
    "status_code": "ERROR",  
    "description": "ReadTimeout: ...Read timed out. (read timeout=30)"  
  },  
  "start_time": "2026-03-20T21:18:24.986273Z",  
  "end_time": "2026-03-20T21:18:54.999505Z",  
  "events": [{  
    "name": "exception",  
    "attributes": {  
      "exception.type": "requests.exceptions.ReadTimeout",  
      "exception.stacktrace": "..."  
    }  
  }]  
}  
{  
  "name": "POST",  
  "status": { "status_code": "UNSET" },  
  "start_time": "2026-03-20T21:18:55.505186Z",  
  "end_time": "2026-03-20T21:19:20.097874Z"  
}  
{  
  "name": "bright_data.search",  
  "attributes": {  
    "scraper.query": "data science",  
    "scraper.latency_ms": 55113.46,  
    "scraper.retries": 1  
  }  
}

Turns out, this query hit the 30-second read timeout, waited for the retry backoff, tried again, and finally came back with data — costing you two proxy requests and 55 seconds instead of one request and ~4 seconds.

You‘d have absolutely no idea from the terminal output itself.

This happens more often than you think — failures that silently blend in with the clean calls around it, that your pipeline still declare a success. That’s the whole argument for adding observability to your data ingest pipeline, right there.

This failure was invisible. No exception or warning, nothing in your logs. OTel may not have prevented this failure — it has nothing to do with networking or data ingestion — but it definitely made it impossible to miss.

The Latency Breakdown Across All Five Queries

Query	Latency	Retries
python programming	3,686ms	0
machine learning	6,558ms	0
web development	3,079ms	0
data science	55,113ms	1
cloud computing	4,600ms	0

The scraper.target_domain attribute lets you aggregate this same breakdown per domain when you scrape multiple targets (e.g. google.com vs bing.com).

Four of five calls were clean. One was 15x slower than average, and you only know that because you were looking at traces.

If you’re running this at scale across hundreds of queries, that pattern — most queries fast, some consistently slow or timeout-prone — is exactly the info you need to tune retry budgets, adjust per-query timeouts, or start asking why that“data science” query keeps choking.

You can’t turn knobs on what you can’t see, after all. Adding observability gives you visibility into things you may not even have thought of.

Going to Production with Jaeger

The console exporter is great for development, but for anything actually running in production you want traces going somewhere persistent. The easiest starting point is Jaeger’s all-in-one Docker image:

docker run -d --name jaeger \  
  -p 16686:16686 \ 
  -p 4318:4318 \  
  jaegertracing/all-in-one:latest

Then, as before:

> python scraper.py --count 10

After this run, open http://localhost:16686, search for bd-scraper(or whatever you called yours) and you’ll see each bright_data.search span as a row in the trace timeline with the nested POST spans inside.

The data science query was slow again, but hey, at least it didn’t fail? Small victories. 😅 It stands out immediately in the Jaeger UI — one that’s wider than everything else on the screen (~17 seconds.)

This is what http://localhost:16686/search will look like after a run.

Click through it for more info.

And you can expand through traces here to as fine grained a detail as you need.

For a real setup, swap Jaeger out for whatever backend you already run. Grafana Tempo, Honeycomb, Datadog — the OTLP exporter speaks the same protocol to all of them.

That’s everything! Feel free to reach out on LinkedIn if you have questions, or leave a comment below.👋

Building a Local Data Analytics Pipeline with dbt Core and DuckDB

Prithwish Nath — Wed, 18 Mar 2026 16:49:14 +0000

TL;DR: This pipeline uses dbt Core + DuckDB locally — no infrastructure — to normalize domains, deduplicate URLs, enforce data contracts via tests, and materialize four analyst-ready mart tables from raw SERP API output.

Press enter or click to view image in full size

After web ingestion, you’ll have inconsistent domains, duplicate URLs across collection runs, null titles, and more. This is not wrong data, per se, just unprocessed data. The gap between “data in a table” and “data you can trust in a query” is bigger than you think.

dbt (data build tool) is an open-source transformation framework that can help us with exactly that problem: you write SQL models, it materializes them in dependency order, and it tracks lineage from raw source to final output. Paired with DuckDB via the community dbt-duckdb adapter — no infrastructure needed, it’s all.duckdb files — it's a surprisingly capable local setup for closing that gap.

I’ll walk you through the Python-based pipeline I use — one that takes SERP data and produces analytics ready tables.

Requirements

What you need: Python 3.x, first of all. Then we can install our requirements like so:

pip install dbt-core dbt-duckdb duckdb requests python-dotenv pandas

For ingestion, we’ll be using a SERP API — I’m using the one I have access to, Bright Data. For this, you’ll need an account with a SERP zone (get API key and zone from its dashboard).

Create a project directory with this layout:

ingest/ for the Python scripts (bright_data.py, duckdb_manager.py, scraper.py),
models/ for dbt (with subfolders staging/, intermediate/, marts/),
data/ for the .duckdb files, and
profiles.yml, and dbt_project.yml at the root.

So there are two main phases: a Python ingest layer that collects and streams results into DuckDB, and a dbt transformation layer with three tiers — staging, intermediate, and marts.

Phase 1: Ingesting SERP Data into DuckDB

Before dbt has anything to work with, we need data in the database. The ingest layer is three Python files: bright_data.py wraps the Bright Data SERP API, duckdb_manager.py handles the DuckDB connection and schema, and scraper.py orchestrates the collection loop. Place them in an ingest/ subdirectory.

The Bright Data Client

Bright Data’s SERP API works differently from a proxy setup. Rather than routing requests through a proxy, you POST a target URL to their API endpoint and get back structured JSON.

ingest/bright_data.py:

"""  
Bright Data SERP API client for fetching search results  
"""  

import os  
import requests  
from typing import Dict, Any, Optional  
from dotenv import load_dotenv  

load_dotenv()  


class BrightDataClient:  
    """  
    Client for Bright Data SERP API  
    Uses the SERP API endpoint (not proxy) for Google search access  
    """  

    def __init__(  
        self,  
        api_key: Optional[str] = None,  
        zone: Optional[str] = None,  
        country: Optional[str] = None  
    ):  
        env_api_key = os.getenv("BRIGHT_DATA_API_KEY")  
        env_zone = os.getenv("BRIGHT_DATA_ZONE")  
        env_country = os.getenv("BRIGHT_DATA_COUNTRY")  

        self.api_key = api_key or env_api_key  
        self.zone = zone or env_zone  
        self.country = country or env_country  
        self.api_endpoint = "https://api.brightdata.com/request"  

        if not self.api_key:  
            raise ValueError(  
                "BRIGHT_DATA_API_KEY must be provided via constructor or environment variable. "  
                "Get your API key from: https://brightdata.com/cp/setting/users"  
            )  

        if not self.zone:  
            raise ValueError(  
                "BRIGHT_DATA_ZONE must be provided via constructor or environment variable. "  
                "Manage zones at: https://brightdata.com/cp/zones"  
            )  

        self.session = requests.Session()  
        self.session.headers.update({  
            'Content-Type': 'application/json',  
            'Authorization': f'Bearer {self.api_key}'  
        })  

    def search(  
        self,  
        query: str,  
        num_results: int = 10,  
        language: Optional[str] = None,  
        country: Optional[str] = None  
    ) -> Dict[str, Any]:  
        """  
        Execute a Google search via Bright Data SERP API  

        Args:  
            query: Search query string  
            num_results: Number of results to return (default: 10)  
            language: Language code (e.g., 'en', 'es', 'fr')  
            country: Country code (e.g., 'us', 'uk', 'ca')  

        Returns:  
            Dictionary containing search results in JSON format  
        """  
        search_url = (  
            f"https://www.google.com/search"  
            f"?q={requests.utils.quote(query)}"  
            f"&num={num_results}"  
            f"&brd_json=1"  
        )  

        if language:  
            search_url += f"&hl={language}&lr=lang_{language}"  

        target_country = country or self.country  

        payload = {  
            'zone': self.zone,  
            'url': search_url,  
            'format': 'json'  
        }  

        if target_country:  
            payload['country'] = target_country  

        try:  
            response = self.session.post(  
                self.api_endpoint,  
                json=payload,  
                timeout=30  
            )  
            response.raise_for_status()  
            return response.json()  

        except requests.exceptions.HTTPError as e:  
            error_msg = f"Search request failed with HTTP {e.response.status_code}"  
            if e.response.text:  
                error_msg += f": {e.response.text[:200]}"  
            raise RuntimeError(error_msg) from e  
        except requests.exceptions.RequestException as e:  
            raise RuntimeError(f"Search request failed: {e}") from e

Note thebrd_json=1 parameter appended to the Google search URL — that’s what tells Bright Data to parse the response and return structured data rather than raw HTML.

Configuration

API key, zone, country — read from environment variables with constructor overrides, so the client works both in scripts and in environments where secrets come in differently.
Without BRIGHT_DATA_API_KEY and BRIGHT_DATA_ZONE set, it raises immediately with a message pointing to the right place in the Bright Data dashboard.
The client also supports a language parameter (e.g. hl=en, lr=lang_en) for non-English or multi-region SERP analysis — pass it to search() or set BRIGHT_DATA_COUNTRY for geo-targeting.

The DuckDB Manager

We’ll use two databases. The DuckDB Python API lets us create files, run SQL, and insert from pandas DataFrames directly:

serp_data.duckdb — the source DB that holds raw ingest output, and
serp_analytics.duckdb — our analytics DB, one that holds the transformed models.

dbt attaches the source DB read-only and writes only to the analytics DB, so raw data stays untouched.

💡 The keen among you may have noticed that I'm basically stealing the medallion architecture pattern from Databricks/BigQuery projects here. So "bronze" stays untouched, "silver/gold" tables are derived. Why do this? If a dbt model has a bug and you materialize garbage into analytics, your raw data is completely clean and you just re-run.

For our source DB, the schema is simple: one serp_results table with indexes on query and domain — the two fields that get hit hardest in the dbt transformations downstream.

ingest/duckdb_manager.py

"""  
DuckDB connection and schema management for SERP ingest  
"""  

import duckdb  
import os  
from typing import Optional, List, Dict, Any  
from datetime import datetime  

class DuckDBManager:  
    """Manages DuckDB connection, schema, and insert operations"""  

    def __init__(self, db_path: str = "data/serp_data.duckdb"):     
        # db_path = "serp_data.duckdb" gives dirname = "", which can cause issues on some setups   
        # So...safer to guard like so:  
        parent = os.path.dirname(db_path)  
        if parent:  
            os.makedirs(parent, exist_ok=True)    

        self.db_path = db_path  
        self.conn = duckdb.connect(db_path)  
        self._create_schema()  

    def _create_schema(self):  
        self.conn.execute("""  
            CREATE TABLE IF NOT EXISTS serp_results (  
                id BIGINT PRIMARY KEY,  
                query TEXT NOT NULL,  
                timestamp TIMESTAMP NOT NULL,  
                result_position INTEGER NOT NULL,  
                title TEXT,  
                url TEXT,  
                snippet TEXT,  
                domain TEXT,  
                rank INTEGER  
            )  
        """)  


    def insert_batch(self, results: List[Dict[str, Any]], query: str, timestamp: Optional[datetime] = None):  
        if timestamp is None:  
            timestamp = datetime.now()  

        if not results:  
            return  

        def extract_domain(url: str) -> str:  
            if not url:  
                return ""  
            try:  
                from urllib.parse import urlparse  
                parsed = urlparse(url)  
                return parsed.netloc.replace("www.", "")  
            except Exception:  
                return ""  

        max_id_result = self.conn.execute("SELECT COALESCE(MAX(id), 0) FROM serp_results").fetchone()  
        next_id = (max_id_result[0] if max_id_result else 0) + 1  

        rows = []  
        for idx, result in enumerate(results):  
            url = result.get('url', result.get('link', ''))  
            domain = extract_domain(url)  

            rows.append({  
                'id': next_id + idx,  
                'query': query,  
                'timestamp': timestamp,  
                'result_position': idx + 1,  
                'title': result.get('title', ''),  
                'url': url,  
                'snippet': result.get('snippet', result.get('description', '')),  
                'domain': domain,  
                'rank': idx + 1  
            })  

        import pandas as pd  
        df = pd.DataFrame(rows)  
        # Best practice is to specify columns explicitly (like, INSERT INTO t (a,b,c) SELECT a,b,c FROM df)   
        # to avoid mismatch if table or DataFrame order changes  
        self.conn.execute("""  
            INSERT INTO serp_results (id, query, timestamp, result_position, title, url, snippet, domain, rank)  
            SELECT id, query, timestamp, result_position, title, url, snippet, domain, rank FROM df  
        """)    

    def get_row_count(self) -> int:  
        result = self.conn.execute("SELECT COUNT(*) FROM serp_results").fetchone()  
        return result[0] if result else 0  

    def close(self):  
        self.conn.close()  

    def __enter__(self):  
        return self  

    def __exit__(self, exc_type, exc_val, exc_tb):  
        self.close()

Just in case, the insert_batch method handles inconsistent SERP field names (link vs url, description vs snippet).

Also, it’s worth being honest about what the domain extraction is and isn’t: I’ve accounted only for a best-effort ingest-time extraction, not the canonical domain value. The dbt staging model is where the real normalization happens — lowercasing, stripping www., and falling back to regex extraction when the field is missing. The ingest layer just makes sure something is in the column.

Making the Two Work Together to Ingest Web Data

The scraper loops through a list of queries, calls the Bright Data client for each, and streams results into DuckDB in batches. Put a .env file in the project root with BRIGHT_DATA_API_KEY and BRIGHT_DATA_ZONE, then run from the project root:

python ingest/scraper.py --count 50000

Some other time saving flags to add:

--queries “foo” “bar” for custom queries (Here, I’ve made it default to 10 tech keywords),

--batch-size 10 for results per API call,

--delay 1.0 for seconds between calls, and

--db path/to/serp_data.duckdb to override the output path (default: data/serp_data.duckdb).

"""  
SERP scraper that streams results to DuckDB (data/serp_data.duckdb).  
Run from project root. Then run dbt to build analytics.  
"""  

import argparse    
import time    
import sys    
import os    
from pathlib import Path    

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))  

from bright_data import BrightDataClient    
from duckdb_manager import DuckDBManager  

def _default_db_path() -> str:    
    script_dir = Path(__file__).resolve().parent    
    project_root = script_dir.parent    
    return str(project_root / "data" / "serp_data.duckdb")  

def scrape_and_insert( total_results: int,    
    queries: list = None,    
    batch_size: int = 10,    
    delay_seconds: float = 1.0,    
    db_path: str = None ):    
    if queries is None:    
        queries = [    
            "python programming", "machine learning", "web development",    
            "data science", "cloud computing", "javascript frameworks",    
            "database design", "API development", "devops tools", "cybersecurity"    
        ]  

    db_path = db_path or _default_db_path()    
    client = BrightDataClient()  

    with DuckDBManager(db_path) as db:    
        print(f"Starting scrape: {total_results} total results")    
        print(f"Database: {db.db_path}")    
        print("Then run: dbt run --profiles-dir .")  

        results_scraped = 0    
        query_idx = 0    
        start_time = time.time()    

        try:    
            while results_scraped < total_results:    
                query = queries[query_idx % len(queries)]  

                try:    
                    serp_data = client.search(query, num_results=batch_size)  

                    organic_results = []    
                    if isinstance(serp_data, dict):    
                        if 'organic' in serp_data:    
                            organic_results = serp_data['organic']    
                        elif 'body' in serp_data and isinstance(serp_data['body'], dict):    
                            if 'organic' in serp_data['body']:    
                                organic_results = serp_data['body']['organic']  

                    if organic_results:    
                        db.insert_batch(organic_results, query)    
                        results_scraped += len(organic_results)      
                        print(f"[{results_scraped}/{total_results}] Query: '{query}' | Inserted: {len(organic_results)}")    

                    query_idx += 1  

                    if results_scraped < total_results:    
                        time.sleep(delay_seconds)  

                except Exception as e:    
                    print(f"Error scraping query '{query}': {e}")    
                    query_idx += 1    
                    continue  

        except KeyboardInterrupt:    
            print("\nScraping interrupted by user")  

        elapsed = time.time() - start_time    
        final_count = db.get_row_count()    

        print(f"\n=== Scraping Complete ===")    
        print(f"Total rows in DB: {final_count}")    
        print(f"Time elapsed: {elapsed:.2f}s")    
        print(f"Rate: {final_count/elapsed:.1f} rows/sec")    

if __name__ == "__main__":    
    parser = argparse.ArgumentParser(description="Scrape SERP results to DuckDB for dbt")    
    parser.add_argument("--count", type=int, default=50000)    
    parser.add_argument("--batch-size", type=int, default=10)    
    parser.add_argument("--delay", type=float, default=1.0)    
    parser.add_argument("--queries", nargs="+")    
    parser.add_argument("--db")  

    args = parser.parse_args()  

    scrape_and_insert(    
        total_results=args.count,    
        queries=args.queries,    
        batch_size=args.batch_size,    
        delay_seconds=args.delay,    
        db_path=args.db    
    )

A few things in here that are easy to overlook:

In case your SERP API response structure isn’t always consistent — organic results can live at serp_data[‘organic’] or nested at serp_data[‘body’][‘organic’] depending on the response format and search engine (Bright Data supports Google as well as others), so the parser checks both.
There’s a configurable delay between API calls ( --delay, default 1 second) to avoid hammering the API.
Progress is printed after each batch so you can track the run.

When the scrape finishes, you have a populated data/serp_data.duckdb file with a serp_results table. That’s our raw data. We now have everything dbt needs to begin.

Phase 2: Transforming with dbt

Connecting dbt to DuckDB

The dbt-duckdb community adapter supports attaching multiple database files, which is exactly what we need here: we’ll make dbt write transformed models to serp_analytics.duckdb while treating serp_data.duckdb as a read-only source. This separation is a good idea because, as a standard, you never want a transformation step to accidentally mutate the raw data it's reading from.

Read more about configuring the dbt-duckdb adapter here 👉

profiles.yml is where the connection lives. The attach block is the part that makes this work — it mounts the raw DB under the alias serp_source, which is the database name the source declaration will reference:

profiles.yml:

serp_analytics:  
  target: dev  
  outputs:  
    dev:  
      type: duckdb  
      path: data/serp_analytics.duckdb  
      threads: 1  
      settings:  
        max_temp_directory_size: '10GB'  
        memory_limit: '12GB'  
        preserve_insertion_order: false  
      attach:  
        - path: data/serp_data.duckdb  
          type: duckdb  
          alias: serp_source

Tune memory_limit and max_temp_directory_size based on how much data you’re working with. I’ve optimized these values for a stress-free 50k-row scrape but you may want to lower them on smaller machines.

dbt_project.yml holds the project-level configuration — model paths, materialization defaults per layer, and the two variables that control pipeline behavior:

dbt_project.yml

name: serp_analytics    
version: 1.0.0    
config-version: 2    
profile: serp_analytics  

model-paths: ["models"]    
test-paths: ["tests"]    
target-path: "target"    
clean-targets:    
  - "target"    
  - "dbt_packages"  

models:    
  serp_analytics:    
    staging:    
      +schema: staging    
    intermediate:    
      +schema: intermediate    
    marts:    
      +schema: marts

Declaring the Source

In dbt, you don’t reference raw tables directly in your models. You declare them as sources first, and this is what makes lineage tracking possible.

models/sources.yml

version: 2  
sources:  
  - name: serp_source  
    description: Raw SERP data from Bright Data collection  
    database: serp_source  
    schema: main  
    tables:  
      - name: serp_results  
        description: Raw search engine result pages - id, query, url, domain, rank, position, etc.  
        columns:  
          - name: id  
            description: Primary key  
          - name: query  
            description: Search query string  
          - name: url  
            description: Result URL  
          - name: domain  
            description: Extracted domain (e.g. example.com)  
          - name: result_position  
            description: Position on SERP (1-based)  
          - name: rank  
            description: Rank value (can differ from position)  
          - name: title  
            description: Result title  
          - name: timestamp  
            description: Scrape timestamp

The database: serp_source matches the attach alias — that’s how dbt finds the raw table. From here, every model references {{ source() }} or {{ ref() }} rather than a raw table path. This doesn’t sound too important, but in fact, it’s what gives dbt the information it needs to build a lineage graph — a visual DAG of how every output table was derived from the source.

It’s worth talking about what the lineage graph actually buys you, because it’s easy to dismiss as a visualization feature:

When something looks wrong in a mart, you trace upstream through the graph to find where it came from.
When you’re about to refactor staging, you can see which intermediate models and marts depend on it before you touch anything.
When someone new joins the project, they get the full picture of how every output table was derived — in seconds, not by grepping through SQL files.

Run dbt docs generate && dbt docs serve to view it in the browser.

Lineage graph from dbt docs serve, showing the dependency chain from raw SERP data to analytics-ready aggregations.

See these dbt docs for more details on these commands.

Staging: The Contract

The staging model is where all the defensive work happens. Its job is a guarantee: by the time a row leaves staging, it is clean, normalized, and deduplicated. Every model downstream gets to assume that contract holds, which means none of them have to repeat the defensive logic.

Staging and intermediate models use materialized='view'; marts use [materialized='table'].

models/staging/stg_serp_results.sql

{{  
  config(  
    materialized='view',  
  )  
}}  
with raw as (  
    select * from {{ source('serp_source', 'serp_results') }}  
),  

cleaned as (  
    select  
        id,  
        trim(query) as query,  
        timestamp,  
        result_position,  
        coalesce(nullif(trim(title), ''), 'Untitled') as title,  
        url,  
        coalesce(nullif(trim(snippet), ''), '') as snippet,  
        case  
            when domain is null or trim(domain) = '' then coalesce(  
                regexp_extract(url, '^https?://(?:www\.)?([^/]+)', 1),  
                'unknown'  
            )  
            else lower(regexp_replace(trim(domain), '^www\.', '', 'i'))  
        end as domain,  
        rank    
    from raw    
),  
deduplicated as (  
    select *  
    from (  
        select *,  
            row_number() over (partition by url, query order by id) as rn  
        from cleaned  
    )  
    where rn = 1  
)  
select  
    id, query, timestamp, result_position, title,  
    url, snippet, domain, rank    
from deduplicated

Now, the domain normalization block is the part worth paying attention to.

The scraper provides a domain field, but it isn't always populated, and even when it is, capitalization and www. prefixes create false cardinality in downstream aggregations.
The CASE expression handles both paths: if the field is present, lowercase it and strip www.; if it's missing, fall back to extracting the domain from the URL via DuckDB's regexp_extract / regexp_replace.
Without this, www.Example.com, example.com, and a row where domain is null but the URL is https://www.example.com/... all count as different domains in every GROUP BY. That kind of silent cardinality inflation is exactly what staging is supposed to catch.

Deduplication happens last. A window function partitions by (url, query) and keeps the row with the lowest id — the earliest record if a URL was collected more than once for a given query.

What about tests?

Add models/staging/stg_serp_results.yml next to your staging model with column-level tests — straightforward not_null checks on the four fields that everything downstream depends on:

version: 2  
models:  
  - name: stg_serp_results  
    description: Cleaned and deduplicated SERP results; domain normalized  
    columns:  
      - name: url  
        description: Result URL  
        tests:  
          - not_null  
      - name: query  
        description: Search query  
        tests:  
          - not_null  
      - name: domain  
        description: Normalized domain  
        tests:  
          - not_null  
      - name: result_position  
        description: SERP position (1-based)  
        tests:  
          - not_null

There’s also a custom test in tests/unique_url_query.sql:

-- Custom test because (url, query) should be unique in staging  
select url, query, count(*) as cnt  
from {{ ref('stg_serp_results') }}  
group by url, query  
having count(*) > 1

Why? Because dbt’s built-in unique test only checks a single column. Our custom test checks a combination — a given URL should appear only once per query after deduplication.

If the window function logic ever breaks due to a refactor, or if upstream data arrives in a shape the deduplication didn't account for, this test catches it before bad data reaches the marts. That's the value of encoding the business rule as a test: it runs every time, and it fails loudly.

Intermediate: Filter Once, Trust Everywhere

The intermediate model is intentionally short (also a view).

models/intermediate/int_serp_results.sql:

{{  
  config(  
    materialized='view',  
  )  
}}  
select  
    query,  
    domain,  
    url,  
    result_position,  
    timestamp  
from {{ ref('stg_serp_results') }}  
where result_position is not null  
  and result_position > 0  
  and domain is not null  
  and domain != 'unknown'

You might wonder why this exists as its own layer rather than putting these filters in each mart. The reason is that all four mart models need the same filtered dataset.

If the filter logic lived in each mart, changing the definition of a “valid” result would mean changing it in four places and hoping they stayed in sync — which they won’t, eventually. Intermediate models are dbt’s answer to that: define the analysis-ready dataset once, name it, and let everything downstream reference it.

Marts: Four Questions, Four Tables

The mart layer is where the pipeline produces something you’d actually hand to an analyst or wire to a dashboard.

Staging and intermediate models are materialized as views — they’re always fresh and don’t consume extra storage. Marts are materialized as tables — written to disk on every run — so queries against them are fast regardless of upstream complexity.

That split keeps the feedback loop fast during development while giving analysts precomputed tables to query.

Query Coverage

models/marts/agg_query_coverage.sql

{{  
  config(  
    materialized='table',  
  )  
}}  
select  
    query,  
    count(distinct url) as unique_urls,  
    count(distinct domain) as unique_domains,  
    count(*) as total_results,  
    min(result_position) as best_position,  
    max(result_position) as worst_position,  
    avg(result_position) as avg_position  
from {{ ref('int_serp_results') }}  
group by query  
order by unique_domains desc

For each search query: how many unique URLs and domains appeared, and what was the spread of positions? This is where you’d start to understand which queries have competitive SERP landscapes — lots of domains spread across positions — versus which are dominated by a handful of sites.

Rank Distribution

models/marts/agg_rank_distribution.sql:

{{  
  config(  
    materialized='table',  
  )  
}}  
with rank_buckets as (  
    select  
        query,  
        domain,  
        case  
            when result_position <= 3 then '1-3'  
            when result_position <= 10 then '4-10'  
            when result_position <= 20 then '11-20'  
            when result_position <= 50 then '21-50'  
            else '50+'  
        end as position_bucket,  
        result_position,  
        count(*) as appearances  
    from {{ ref('int_serp_results') }}  
    group by 1, 2, 3, 4  
)  
select  
    query,  
    position_bucket,  
    count(distinct domain) as unique_domains,  
    sum(appearances) as total_appearances,  
    avg(result_position) as avg_position  
from rank_buckets  
group by query, position_bucket  
order by query, position_bucket

The position buckets — 1–3, 4–10, 11–20, 21–50, 50+ — map to how SEO practitioners actually think about SERP real estate. Positions 1–3 capture the majority of clicks. Anything past 10 is largely invisible. By bucketing in the mart rather than the BI layer, you’re encoding that domain knowledge once, in version-controlled SQL, rather than relying on every analyst who touches the data to re-derive it correctly.

Domain Rank Summary

models/marts/agg_domain_rank_summary.sql

{{  
  config(  
    materialized='table',  
  )  
}}  
select  
    domain,  
    count(*) as total_appearances,  
    count(distinct query) as query_coverage,  
    count(distinct url) as unique_urls,  
    avg(result_position) as avg_position,  
    min(result_position) as best_position,  
    max(result_position) as worst_position  
from {{ ref('int_serp_results') }}  
group by domain  
order by total_appearances desc

This flips the perspective from query-centric to domain-centric. query_coverage is the field I find most useful here — it tells you how many distinct queries a domain appeared in, which is a rough proxy for breadth of SERP presence. A domain with high total_appearances but low query_coverage is strong in a narrow area; a domain with both metrics high is broadly dominant.

Domain x Query Matrix

models/marts/agg_domain_query_matrix.sql:

{{  
  config(  
    materialized='table',  
  )  
}}  
-- Domain x query matrix: best position and appearances per (domain, query)  
select  
    domain,  
    query,  
    min(result_position) as best_position,  
    count(*) as appearances  
from {{ ref('int_serp_results') }}  
group by domain, query  
order by domain, query

The most compact mart, but also the most useful for visualization. Each row is a domain–query pair: the best position that domain achieved for that query, and how many times it appeared. Pivot this by query and you have an SEO visibility matrix — at a glance, which domains rank consistently across your whole keyword set, and which are only competitive in specific corners of it.

Running the Pipeline

From your project root (where dbt_project.yml and profiles.yml live):

# Mind the " ."  
dbt run --profiles-dir .

dbt resolves the ref() graph before executing anything, so models always run in the right dependency order — staging first, then intermediate, then the four marts. To exclude synthetic data across the whole pipeline:

To run tests after materializing:

dbt test --profiles-dir .

To view the lineage graph and model docs in the browser:

dbt docs generate --profiles-dir .  

dbt docs serve

If profiles.yml is in the project root,--profiles-dir . tells dbt to look there. If you use ~/.dbt/profiles.yml, you can omit it.

If the deduplication breaks, or if upstream data arrives with unexpected nulls in critical columns, the tests surface it before bad data reaches the marts. That’s the other thing the pipeline buys you beyond SQL organization: the tests run as a first-class step, not as a notebook someone wrote once and forgot to share.

The raw web data you ingest is queryable, certainly (and SERP APIs like Bright Data make that very convenient with their structured JSON response), but it isn’t trustworthy in the way analytics requires — where “trustworthy” means that every analyst querying it gets the same answers, because the decisions about normalization, deduplication, and what constitutes a valid result were made once and encoded somewhere they can be found.

Without a pipeline like this, those decisions happen ad hoc. Someone normalizes the domain in a notebook. Someone else doesn’t. The counts disagree, and it’s not clear why. dbt’s layered model is a response to that problem:

the decisions are in version-controlled SQL,
the contracts are enforced by tests that run on every execution, and
when something changes upstream, the first place you’ll hear about it is a failing test rather than a confused analyst.

That’s reproducible analytics — the same inputs and the same logic producing the same outputs, every time.

That’s what the move from raw scrapes to a modeled pipeline with dbt actually buys you. Not fancier queries, but a system where the logic is visible, the contracts are testable, and the next person who touches the data doesn’t have to reverse-engineer what you were thinking.

The Practical Limits of DuckDB on Commodity Hardware

Prithwish Nath — Tue, 10 Mar 2026 08:21:25 +0000

DuckDB shouldn’t work this well. It’s a single embedded library that needs no server, config, or cloud bill — yet it handles warehouse-scale columnar analytics with surprising ease.

Plenty of benchmarks already show that DuckDB can process large datasets. The more useful question, to me, is narrower: how far does it scale on low-end hardware before interactivity breaks down? I do data forensics for a living, and the last thing I want is infrastructure getting in the way.

To answer that, I ran a 50-million-row benchmark on a ~$500 Acer Aspire 5 (Raptor Lake i5, 16GB RAM, 1 TB SSD). Starting with 50,000 real-world search results, I generated a large synthetic dataset and executed increasingly complex analytical queries to identify where performance crossed practical thresholds.

I'll present my findings here. The result wasn't a catastrophic failure point, but a series of predictable transitions (from instant, to tolerable, to better-scheduled-than-waited-on) depending on query shape.

1. The Four Performance Zones

Before I dive into specific query types, understand that not all scales perform the same way. The same query that feels instant at 1 million rows might cross into “go grab a coffee” territory at 30 million — and different query types degrade at different rates.

So there isn’t one big scale — but three. One for each type of analytical query:

Percentile queries (likeGROUP BY + PERCENTILE_CONT) — when you’re trying to answer questions like “For each domain, what’s its median, 25th, 75th, and 95th percentile search ranking?” Or: “For each customer tier, what is the session duration distribution?” This is the kind of query you run when averages aren’t enough. You want to understand spread, skew, and outliers. It’s common in finance, product analytics, marketplace reporting — anywhere distributions matter more than single numbers.
Window functions (likeLAG with PARTITION BY) — e.g. “For each user, how did their activity change vs. previous session?” Or: “For each account, how did monthly revenue change vs. last month?” This is bread-and-butter analytics — a simple time-series comparison. It’s what powers growth dashboards, churn analysis. Amazon, for example, uses this extensively for anomaly detection. These are more expensive because they require partitioning and sorting — especially on large datasets like ours.
Aggregations (likeCOUNT, AVG, MIN, MAXwith DISTINCT) — e.g. “For each region, how many total orders did we ship, how many unique customers/products, what were the min/max purchase values?” This is the classic summary view, the kind of query behind almost every dashboard card. Deceptively simple because DISTINCTat scale forces the engine to work harder than you might expect.

Together, these three queries make up a massive chunk of all real-world analytics.

Here’s how they perform going from 1,000 to 50,000,000 rows:

Query time (seconds) vs record count (millions). Image created via D3.js by Author.

Scale	Percentile	Window	Aggregation	Worst Case
1M	0.11s	0.80s	0.12s	0.8s
2M	0.28s	1.78s	0.50s	1.8s
5M	0.64s	3.28s	1.23s	3.3s
10M	1.65s	6.26s	2.17s	6.3s
15M	2.69s	7.94s	4.15s	7.9s
20M	3.69s	15.26s	6.58s	15.3s
25M	5.52s	22.98s	10.60s	23.0s
30M	6.67s	31.41s	15.46s	31.4s
40M	9.82s	47.43s	22.64s	47.4s
50M	12.95s	67.24s	32.42s	67.2s

Having plotted query times against human perception thresholds (I made some assumptions for those, since this is subjective), we have four distinct performance zones — with boundaries shifting depending on which query type you care about.

“Comfort Zone”: Up to 5M Records

Query times: Near instantaneous, `< 3 seconds for everything.

This is why DuckDB feels magical. For everything up to 5 million rows (and let’s be honest — more data than most organizations are querying interactively anyway) — all queries are fast. Window functions complete in under a second at 1M. Even at 5M rows, the slowest query (you guessed it; a window function) takes just 3.3 seconds. Percentiles and aggregations are sub-second through 2M and barely crack 1 second at 5M.

At this scale, DuckDB keeps everything comfortably in memory. Hash tables fit in RAM, partitioning operations don’t require temp files, and there’s zero disk spilling. This is the sweet spot for team analytics, dashboards, and interactive data exploration.

“Workable”: 5–20M Records

Query times:

Window functions: 3–15 seconds. Noticeable latency. At 10M they take 6 seconds, and at 15M, about 8 seconds. They hit 15 seconds right at 20M — the upper edge of this zone.
Percentiles and aggregations: still under 4 and 7 seconds respectively.

This is where window queries will make people Alt-Tab out to do something else — doomscroll, check their email, watch videos of cats — instead of waiting. Percentiles and aggregations are still firmly comfortable, though.

This zone is still good for Data science work and one-off analyses where a 10–15 second wait is acceptable. Still no disk spilling. Memory usage stays under ~650 MB.

“Now You’re Pushing It”: 20–30M Records

Query times:

Window functions: 15–31 seconds, always crossing the 15s “frustration threshold”.
Aggregations: 7–15 seconds (10.6s at 25M, 15.5s at 30M), near-annoying latency now.
Percentiles: 4–7 seconds (5.5s at 25M, 6.7s at 30M.)

This is the point where query types diverge dramatically. Percentiles are still firmly in the “interactive” category, but aggregations require patience. Window functions however, are firmly in the “go do something else” territory — at 30M records, they can reach a whopping 31 seconds — clearly the point where you should batch these, instead of waiting live.

In general, this zone is best for automated/scheduled work where humans aren’t waiting on complex queries. Simple aggregations and percentiles are still interactive enough.

“Batch-Only”: 30M+ Records

Query times:

Window functions: 31–67 seconds. Always past one minute at 50M.
Aggregations: 15–32 seconds. Always past 30 seconds at 50M.
Percentiles: 7–13 seconds.

In a testament to how performant DuckDB can be — if you’re only running percentiles, even 50M rows is still marginally interactive (tops out at 13 seconds.)

Everything else at this scale, though, should be scheduled.

If you need window functions, this (30M and above) is clearly the point where you should consider a cloud data warehouse — they cross a minute at 50M (~67 seconds). Aggregations taking 20–30 seconds on average is also beyond generous definitions of “interactive”.

So this is where we’ve found our limit — albeit not a hard one. Performance degrades predictably, not catastrophically. Critically, there’s still zero disk spilling. DuckDB never writes temp files to disk, even at 50M records. Memory usage peaked at ~1.2 GB.

Here’s a practical cheatsheet, sorted by use case:

Use case	Simple analytics ceiling	Window function ceiling
Live auto-refresh dashboard (less than 500ms)	~2M rows	~500K rows
API-backed service (less than 1s)	~5M rows	~1M rows
Interactive BI (less than 3s)	~15M rows	~5M rows
Notebook / exploration (less than 10s)	~40M rows	~15M rows
Ad-hoc analyst SQL (less than 30s)	50M+ rows	~20–25M rows
Batch / ETL (less than 2min)	50M+ rows	50M+ rows

2. The Goldilocks Zone for Local Analytics

If I had to draw a circle around the scale where DuckDB on a cheap laptop is just right — fast enough to feel interactive, complex enough to handle real analytical work, without needing to think about infrastructure — it’s 1M to 10M rows.

Here’s why that range is special.

At 1M rows, every query type is effectively instant. Aggregations finish in 0.06 seconds. Even the most expensive query in the test — the LAG window function — takes 0.47 seconds. The database is not part of your cognitive load at all at this scale.

At 5M rows, that’s still largely true. Aggregations take half a second. Percentiles take 1.3 seconds. Window functions take 1.7 seconds. 5M rows of real data is a meaningful dataset — a year of event logs for a mid-sized SaaS product, a full crawl of a large website, several years of transaction history for a small business. And DuckDB chews through it without complaint.

At 10M rows, you start paying a tax specifically on window functions — they cross 6 seconds here. But aggregations (0.79s) and percentiles (1.99s) are still fast enough that interactive use feels natural. If your workload skews toward GROUP BY analytics and distribution queries rather than time-series comparisons, 10M rows is still firmly comfortable.

So our “Goldilocks” Zone — so named after the fairytale of Goldilocks and the Three Bears; Goldilocks rejects porridges that are too hot and too cold, until finding one that’s “just right” — is up to ~10M rows for any query type, and up to ~20M rows if you’re willing to accept that window functions will make you wait.

Below that ceiling, DuckDB on cheap/commodity hardware is no compromise at all.

3. Memory is Not the Bottleneck

Across all scales tested (1K → 50M rows, averaged over 5 runs) DuckDB never exceeded ~1.2 GB of memory — even at 50 million rows.

Peak memory (MB) vs record count (log scale, 1K–50M). Image created via D3.js by Author.

At the upper bound (50 million rows), peak observed memory usage was ~1,212 MB. On my 16GB laptop that’s ~7.5%. And that’s the worst case in this benchmark — with 50M rows and window-heavy queries in play.

Memory growth is real — especially with window function (delta) queries — but it’s still fairly controlled, I’d say:

10M rows → ~600 MB range
20M rows → ~650 MB range
30–50M rows → ~900 MB–1.2 GB range

Time, on the other hand, accelerates sharply:

Window query: ~6.4s at 10M
~15.8s at 20M
~70s at 50M

So the practical ceiling on a cheap laptop isn’t RAM. You’ll run out of patience before you run out of memory. It means for many local analytics use cases the scaling limit is UX, not hardware — that’s a very different kind of bottleneck, though.

Before we move on, an interesting thing to note is that the aggregation query degrades faster than percentiles at scale. At 5M they’re similar (~0.6s vs ~1.2s), but by 50M the aggregation (33.7s) is nearly 3x the percentile (12.5s). Multi-metric GROUP BY with HAVING and DISTINCT counts WILL get expensive. If your dashboard runs heavy aggregations specifically, you should probably discount the “simple” ceiling by ~30–40%.

4. Window Functions Affect Scaling The Most

If you’re embedding DuckDB in:

A local analytics tool
A CLI data processor
A desktop SaaS product
A developer-facing data tool

Row count alone is not your sizing variable. Query shape is.

Across 18 dataset scales (1k → 50M rows, averaged over 5 runs, the single biggest performance divider wasn’t how much data existed — it was whether the workload used window functions, at all.

Window functions power extremely common patterns:

Ranking results per group (RANK(), ROW_NUMBER())
Comparing rows over time (LAG(), LEAD())
Running totals
Sessionization
Deduplication

These are all operations you’ll run many, many times for a real project.

I found that on identical hardware, aggregation-heavy workloads comfortably scale to tens of millions of rows while staying interactive. These Window-heavy workloads, though, hit “coffee break” territory much earlier.

At 10M rows

Percentile query (aggregation-heavy): ~1.7s, ~217 MB memory delta
Window “delta” query (LAG-style): ~6.4s, ~46 MB memory delta
Group aggregation: ~2.5s, ~200 MB memory delta

Already, the window query is ~3–4x slower than pure aggregation.

At 20M rows

Percentile: ~4.1s
Window (delta): ~15.8s
Aggregation: ~7.3s

The gap widens. The window query is now roughly 2–4x slower than aggregation-style queries.

At 50M rows

Percentile: ~16.5s, ~330 MB delta *
Window (delta): ~70.3s, ~588 MB delta
Aggregation: ~33.3s, ~113 MB deltaThis is where the separation becomes untenable:
The window query takes 70 seconds.
Aggregation-heavy queries remain roughly half that.
Memory growth for window logic accelerates much more aggressively at scale.

If your workload is mostly GROUP BY, percentiles, and simple scans, then you have substantial headroom. If your workload relies heavily on LAG(), LEAD(), RANK() OVER (PARTITION BY …) then your practical ceiling with DuckDB (running on such consumer grade local hardware, at least) arrives much sooner.

5. …But Make Sure you Checkpoint Enough

Speaking of window functions, avery interesting recurring pattern I found was that at exactly 50,000 records, these hit a wall.

Delta query time (seconds) vs record count (log scale, 1K–5M) showing the spike at 50K. Image created via D3.js by Author

This pattern is consistent across all 5 runs:

20K records: 0.24 seconds (fast)
50K records: 1.21 seconds (5x slower than expected)
100K records: 0.20 seconds (fast again)

Assuming expected: linear interpolation between 20K (0.241s) and 100K (0.204s) at 50K: 0.241 + (0.204 − 0.241) × (50 − 20) / (100 − 20) ≈ 0.225s

This specific spike appears in every single run with CV (coefficient of variation) of only 10.3%. Expected time based on linear scaling was 0.23 seconds. Actual time? 1.21 seconds. That’s 430% slower than it should be!

It’s the only non-monotonic performance pattern in the entire dataset. Every other scale shows smooth, predictable scaling. The 50K spike for window/delta functions is the exception.

Why it happens: Checkpoint more frequently. DuckDB has automatic checkpointing (wal_autocheckpoint), but it doesn’t work reliably during bulk inserts from Python. (Check this GitHub issue). Knowing this, I set checkpointing manually, but as it turned out I wasn’t checkpointing enough.

Data was inserted in 1,000-row batches across multiple database connections, but I had a LOT of records (50 million) so I thought I’d manually trigger CHECKPOINT only every 100K rows.

💡 CHECKPOINT is a database operation that flushes pending writes to disk and optimizes storage layout. In DuckDB specifically:- Consolidates fragmented columnar segments created by many small inserts
- Reorganizes data into optimized columnar format for faster reads
- Flushes the write-ahead log (WAL) to the main database fileLike defragmenting a hard drive, CHECKPOINT reorganizes scattered data into contiguous, optimized blocks.

Because the scales were run sequentially (1K → 5K → 10K → 20K → 50K → 100K and so on), by the time the 50K test executed, dozens of small batch inserts had accumulated without a CHECKPOINT, leaving the write-ahead log fragmented across many small column segments.

Now consider the query that spikes:

LAG(rank) OVER (PARTITION BY url, query ORDER BY timestamp)

This requires a full sort of the relevant partitions. Sorting is exactly the kind of operation that is most sensitive to storage layout, and fragmented column segments mean less efficient scanning and sorting. So at 50K, the query pays the fragmentation penalty.

By 50K I had 50 fragmented segments. The window function’s sort pays the cost of reading across all those fragments. At 100K, CHECKPOINT compacted everything, so queries got faster despite more data.

That’s all the data I found. If you want to know about my methodology for this (admittedly) very niche benchmark, read on.

Methodology — How this Low-End Benchmark Was Built

What I wanted to do was stress-test DuckDB with realistic analytical queries at scales most teams would consider “cloud warehouse only.” So if I wanted to benchmark how it performed on low-end hardware, I had to flip the entire approach on its head.

Step 1 — Getting Search Data

I started by fetching 50,000 actual Google SERP results using Bright Data’s SERP API. Why SERP data? Because search results have realistic cardinality — lots of unique URLs, domains, queries, timestamps — exactly the kind of data that stresses analytical databases.

I built ~2,550 unique queries by combining 50 base topics with 51 suffix variations:

_BASE_TOPICS = [  
    "python programming",  
    "machine learning",  
    "web development",  
    # ... 47 more  
]  

_QUERY_SUFFIXES = [  
    "",  
    " tutorial",  
    " 2024",  
    " best practices",  
    # ... 47 more  
]  

SERP_QUERIES = [  
    f"{base}{suffix}".strip()  
    for base in _BASE_TOPICS  
    for suffix in _QUERY_SUFFIXES  
]  
SERP_QUERIES = list(dict.fromkeys(SERP_QUERIES))

Full code here: serp_queries.py

Each query fetched up to 20 organic results via the API, streamed directly into DuckDB as results came in:

client = BrightDataClient()  

with DuckDBManager() as db:  
    while results_obtained < total_results:  
        query = queries[query_idx % len(queries)]  
        serp_data = client.search(query, num_results=batch_size)  

        organic_results = []  
        if isinstance(serp_data, dict):  
            if 'organic' in serp_data:  
                organic_results = serp_data['organic']  
            elif 'body' in serp_data and isinstance(serp_data['body'], dict):  
                if 'organic' in serp_data['body']:  
                    organic_results = serp_data['body']['organic']  

        if organic_results:  
            db.insert_batch(organic_results, query)  
            results_obtained += len(organic_results)  

        query_idx += 1  
        time.sleep(delay_seconds)

Full code here: bright_data.py

My DuckDB schema mirrors the SERP data structure the API returns. (Check their docs here for more info):

self.conn.execute("""  
    CREATE TABLE IF NOT EXISTS serp_results (  
        id BIGINT PRIMARY KEY,  
        query TEXT NOT NULL,  
        timestamp TIMESTAMP NOT NULL,  
        result_position INTEGER NOT NULL,  
        title TEXT,  
        url TEXT,  
        snippet TEXT,  
        domain TEXT,  
        rank INTEGER,  
        previous_rank INTEGER,  
        rank_delta INTEGER  
    )  
""")

Full code here: duckdb_manager.py

This adds up to 50K rows of real, varied data with natural patterns — not artificial test data…but that still wasn’t enough.

Step 2 — Synthesize 50M Records from Real Patterns

50K rows isn’t enough to stress-test at scale. But completely random synthetic data loses the realistic patterns that make queries slow. So I extracted actual domains, queries, title structures, and snippets from the real data and used those to generate variations:

def extract_serp_patterns(db_path):  
    with DuckDBManager(db_path) as db:  
        queries = db.conn.execute(  
            "SELECT DISTINCT query FROM serp_results ORDER BY RANDOM() LIMIT 100"  
        ).fetchall()  
        domains = db.conn.execute(  
            "SELECT DISTINCT domain FROM serp_results WHERE domain IS NOT NULL LIMIT 200"  
        ).fetchall()  
        title_samples = db.conn.execute(  
            "SELECT title FROM serp_results WHERE title IS NOT NULL LIMIT 50"  
        ).fetchall()  
        snippet_samples = db.conn.execute(  
            "SELECT snippet FROM serp_results WHERE snippet IS NOT NULL LIMIT 50"  
        ).fetchall()  
        return {  
            'queries': [r[0] for r in queries],  
            'domains': [r[0] for r in domains],  
            ...  
        }

Synthetic rows were generated in batches of 10,000 and inserted directly, mixed in with the 50k real records, until we had 50M total:

for i in range(0, needed, batch_size):  
    batch = [{  
        'id': current_id,  
        'query': random.choice(queries),  
        'domain': random.choice(domains),  
        ...  
    } for j in range(batch_size)]  
    df = pd.DataFrame(batch)  
    db.conn.execute("INSERT INTO serp_results SELECT * FROM df")

Full code here: benchmark.py

Step 3 — Design the Query Suite

I chose three query types that represent common analytical workloads:

Percentile Queries:

query = f"""  
    SELECT   
        domain,  
        COUNT(\*) as result_count,  
        AVG(rank) as avg_rank,  
        PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY rank) as median_rank,  
        PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY rank) as p25_rank,  
        PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY rank) as p75_rank,  
        PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY rank) as p95_rank  
    FROM serp_results  
    {where_clause}  
    GROUP BY domain  
    ORDER BY avg_rank  
    LIMIT 100  
"""

Window Functions:

query = f"""  
    WITH ranked AS (  
        SELECT   
            url,  
            query,  
            rank,  
            timestamp,  
            LAG(rank) OVER (PARTITION BY url, query ORDER BY timestamp) as previous_rank  
        FROM serp_results  
        {id_filter}  
    )  
    SELECT   
        url,  
        query,  
        rank,  
        previous_rank,  
        rank - previous_rank as rank_delta,  
        timestamp  
    FROM ranked  
    WHERE previous_rank IS NOT NULL  
    ORDER BY ABS(rank_delta) DESC  
    LIMIT 100  
"""

Aggregation Queries:

query = f"""  
    SELECT   
        domain,  
        COUNT(*) as total_results,  
        COUNT(DISTINCT query) as unique_queries,  
        AVG(rank) as avg_rank,  
        MIN(rank) as best_rank,  
        MAX(rank) as worst_rank,  
        COUNT(DISTINCT url) as unique_urls  
    FROM serp_results  
    {where_clause}  
    GROUP BY domain  
    HAVING COUNT(*) > 10  
    ORDER BY total_results DESC  
    LIMIT 50  
"""

Full code (all queries) here: queries.py

These test different bottlenecks: percentiles stress sorting and statistical aggregation; window functions stress partitioning and stateful operations; aggregations stress hash tables and DISTINCToperations.

Step 4 — Run at 18 Different Scales

Instead of regenerating data for each scale, I used a single optimization: filter by ID. One dataset, 18 different windows into it:

max_id_result = db.conn.execute(f"""  
    SELECT id FROM (  
        SELECT id FROM serp_results ORDER BY id LIMIT {target_count}  
    ) ORDER BY id DESC LIMIT 1  
""").fetchone()  

max_id = max_id_result[0]

Every query then receives this max_id as a filter:

where_clause = f"WHERE id <= {max_id} AND domain IS NOT NULL AND domain != ''"

Full code here: benchmark.py

This tested 1K, 5K, 10K, 20K, 50K, 100K, 200K, 500K, 1M, 2M, 5M, 10M, 15M, 20M, 25M, 30M, 40M, and 50M records against identical data — isolating row count as the only variable. Each scale measured query execution time and peak memory usage via psutil.

Step 5 — Run Five Complete Iterations

To ensure patterns weren’t random variance, I ran the entire benchmark suite five times. This produced 90 data points per query type (5 runs x 18 scales).

The result: performance was shockingly consistent. Most metrics had less than 10% coefficient of variation across runs — the 50M window function result, for instance, varied by less than 1 second across all five runs.

Limitations

Uniform synthetic distribution: The 50M records were generated by sampling uniformly from ~200 real domains and ~100 real queries. Real-world data follows a power-law distribution — a small number of domains dominate traffic heavily. This means the PARTITION BY url, query window function hits roughly equal-sized partitions in this benchmark, which is best-case behavior. In production data with skewed distributions, window function performance at scale could be meaningfully worse than the numbers here suggest.
Single-table queries only: Didn’t test multi-table joins or recursive CTEs.
Hardware-specific: Tested on one machine (16GB RAM). Results may vary on different hardware.
Query-specific: These three query types don’t represent all analytical workloads.
DuckDB version: Tested on DuckDB >=1.0.0. Newer versions may perform differently.
No concurrent queries: Single-threaded only, no concurrent workload tested.

Conclusion — Scaling Is a UX Ceiling.

This experiment was more about finding out where local analytics on consumer grade hardware stops feeling interactive, rather than stress-testing DuckDB.

Across all runs, DuckDB remained stable and memory usage stayed modest. There was no hard failure point, no dramatic collapse in performance. Instead, execution times increased predictably as data grew — and different query shapes crossed the interactivity threshold at different scales. Aggregation-heavy workloads retained responsiveness far longer, while window-heavy queries reached the UX ceiling much sooner.

This, then, is our practical takeaway: the limiting factor on local analytics isn’t usually RAM or disk; it’s the point at which queries stop feeling fluid. At that boundary, the decision you have to make is entirely about whether (for your use case) the user experience is still good enough to keep things local.

Some links in this article are tracking links used for analytics purposes only. I do not receive any commission or compensation from them.

Build a Bun CLI to Generate TypeScript Clients from API Docs

Prithwish Nath — Wed, 04 Mar 2026 09:38:11 +0000

A Common Dev Pain Point (And My Excuse to Learn Bun)

I hate it when I find an API I want to use, go to their documentation site, and find a beautiful page with endpoints, request/response examples, detailed explanations, and… no OpenAPI spec. No SDK, either.

I understand creating a Swagger/OpenAPI schema involves far more effort than a typical docs page for an API, so I can’t be too upset. But this does limit my options — I’d either have to hand-write fetch calls for every endpoint (tedious, error-prone), or politely ask the API maintainer for an OpenAPI spec (they are not obligated to spend dev cycles on some rando’s request.)

This is the case with at least 75% of all APIs here, for example. Even well-funded APIs sometimes have great docs but no machine-readable spec.

So I built a CLI tool with full proxy support (with Bun — this experiment is mostly because I wanted to learn how to create tooling with it) that generates TypeScript clients from either OpenAPI specs or raw documentation sites.

You use it like:

# 1. Just point it at the docs page  
> dtoc https://docs.some-api.com  
# Or, if you just want to run it in dev without building an executable...  
> bun run index.ts https://docs.some-api.com  

# 2. If the API does have a Swagger/OpenAPI JSON spec, use that instead  
> dtoc https://some-other-api.com/doc.json

…and get back a complete TypeScript API client that you can use like so:

// In CommonJS you could omit the .js here, but not in ESM  
import { ApiClient } from './generated/catfact_ninja/client.js';  
const client = new ApiClient();  
// Get a random cat fact  
const fact = await client.getFactRandom();  
console.log(fact.fact);  
// Get a list of breeds  
const breeds = await client.getBreeds();

And yes, it can read .env files from the same working directory, and compiles to a standalone ~100MB binary you can distribute.

Full source code here: https://github.com/sixthextinction/bun-docs-to-client

Throughout this article, code snippets link to specific files/lines. Some examples are simplified for clarity — check the links for complete implementations.

The Architecture

For any documentation site, we have two paths we can go down. Let’s visualize this before diving into code. So, two distinct pipelines depending on what you feed it:

1. If you have a Swagger/OpenAPI spec (deterministic compile pipeline)

Our happy path is when a proper OpenAPI spec is available. The workflow becomes much more mechanical — and reliable.

Fetch the OpenAPI JSON (pass in the URL to it, or a local file if you have it saved)
Clean non-standard root properties
Validate and dereference the spec using swagger-parser
Normalize server URLs (absolute, relative, or inferred)
Generate TypeScript interfaces fromcomponents.schemas (part of the Swagger/OpenAPI JSON)
Generate a typed ApiClientclass from paths(also part of the spec)
Emit the client, types, tests, and index files

There’s no inference or heuristics at play. The spec becomes the single source of truth. Spec in, deterministic code out. This is our ideal case.

2. If you only have a messy docs site (LLM + runtime synthesis pipeline)

With this route, I’m looking for something that scaffolds me 80% of the way there. A best-effort version. It won’t one-shot every API, and that’s okay. I can do the rest.

When no OpenAPI spec exists, we have to synthesize one from the documentation page itself — and then make real requests to the API to validate it for us.

Our workflow has to become exploratory:

Fetch the documentation HTML, convert HTML → Markdown (using turndown or similar)
Use an LLM (preferably, local) to ONLY extract mentioned API endpoints from the markdown
Infer the base URL from example requests
Categorize endpoints (list, detail, query), then probe them by sending real HTTP requests — inferring the request structure for that endpoint from actual API response
Extract IDs from list responses (e.g. if the docs page mentions /people/1) and use them to probe detail routes like/people/{id} so schemas are inferred from real, working endpoints instead of guesses.
Assemble a minimal but valid OpenAPI spec from the above, validate it using swagger-parser
Generate a typed ApiClientclass and TypeScript interfaces
Emit the client, types, tests, and index files

The LLM’s job is narrow — it is instructed to only identify endpoints mentioned in the documentation (and we filter out the ones we know for sure won’t be endpoints — like image assets, CSS/JS files, OAuth flows, social links, status pages, or obviously non-API routes).

The LLM doesn’t need to be perfect, or fully generate the OpenAPI spec file itself. It just needs to extract mentioned endpoints. Our actual HTTP testing validates everything later and generates accurate schemas from real data.

By the time code generation runs, we’re back in the same deterministic world as the happy path — operating on a validated OpenAPI spec.

To get started: install Bun, then run bun install to install dependencies (We have two: @apidevtools/swagger-parser, and turndown).

Entry point is index.ts, as you’d expect.

import { normalizeUrl, isUrl, extractSiteName, detectContentType } from './src/fetch.js';  
import { parseOpenAPI } from './src/parse.js';  
import { docsToOpenAPI } from './src/docs-to-openapi.js';  
import { generateClient } from './src/generate.js';  
import { emitFiles } from './src/emit.js';  

async function main() {  
  const input = process.argv[2];  

  if (!input) {  
    console.error('Usage: bunx docs-to-client `<url-or-file>`');  
    console.error('Example: bunx docs-to-client https://api.example.com/docs');  
    console.error('Example: bunx docs-to-client ./specs/openapi.json');  
    process.exit(1);  
  }  

  try {  
    let spec: any;  
    let specPath: string = input;  

    if (isUrl(input)) {  
      // Detect if it's HTML docs or OpenAPI JSON  
      const contentType = await detectContentType(input);  

      if (contentType === 'html') {  
        // HTML docs path  
        console.log(`1. Fetching HTML docs from ${input}...`);  
        spec = await docsToOpenAPI(input);  
      } else {  
        // Existing OpenAPI JSON path  
        console.log(`1. Fetching OpenAPI spec from ${input}...`);  
        specPath = normalizeUrl(input);  
        spec = await parseOpenAPI(specPath);  
      }  
    } else {  
      // File path - check extension  
      if (input.endsWith('.json')) {  
        specPath = input;  
        spec = await parseOpenAPI(specPath);  
      } else {  
        // Assume HTML docs file  
        console.log(`1. Reading HTML docs from ${input}...`);  
        spec = await docsToOpenAPI(input);  
      }  
    }  

    const siteName = extractSiteName(input);  

    console.log(`2.✅ Parsed OpenAPI ${spec.openapi || spec.swagger} spec`);  
    console.log(`3. Generating client code...`);  

    const clientCode = await generateClient(spec, input);  

    console.log(`4. Writing files...`);  
    await emitFiles(clientCode, siteName);  

    console.log(`5. Done! Client generated in ./generated/${siteName}/`);  
  } catch (error) {  
    console.error('❌ Error:', error instanceof Error ? error.message : String(error));  
    process.exit(1);  
  }  
}  

main();

This uses the following modules:

src/fetch.ts — URL fetching, proxy support, caching
src/docs-to-openapi.ts — HTML→Markdown→LLM extraction→HTTP testing
src/parse.ts — OpenAPI spec parsing and validation
src/generate.ts — TypeScript client code generation
src/emit.ts — Writing generated files to disk

Implementation: The Happy Path (OpenAPI Spec)

Let’s start with the simpler, deterministic path — when you already have an OpenAPI spec.

1. Fetching and Parsing the Spec

The first step is to (obviously) get the Swagger/OpenAPI JSON, and parse it. We accept either URLs or local file paths:

// Simplified  
export async function fetchOpenAPI(input: string): Promise`<any>` {  
  if (!isUrl(input)) {  
    const file = Bun.file(input);  
    if (!await file.exists()) throw new Error(`File not found: ${input}`);  
    return await file.json();  
  }  

  const response = await fetch(input, { headers: { 'Accept': 'application/json' } });  
  if (!response.ok) throw new Error(`Failed to fetch: ${response.status} ${response.statusText}`);  

  const spec = await response.json();  
  const specsDir = join(process.cwd(), 'specs');  
  await mkdir(specsDir, { recursive: true });  
  await Bun.write(join(specsDir, urlToFilename(input)), JSON.stringify(spec, null, 2));  
  return spec;  
}

Read the full implementation in src/fetch.ts (Lines 111 to 146) https://github.com/sixthextinction/bun-docs-to-client/blob/main/src/fetch.ts#L111-L146

2. Cleaning and Validating the Spec

OpenAPI specs sometimes have non-standard root properties that break parsers. We clean them like so:

// Simplified  
const OPENAPI_ROOT_PROPERTIES = new Set([  
  'openapi', 'swagger', 'info', 'servers', 'paths', 'components',  
  'security', 'tags', 'externalDocs'  
]);  

function cleanSpec(spec: any): any {  
  const cleaned: any = {};  
  for (const [key, value] of Object.entries(spec)) {  
    if (OPENAPI_ROOT_PROPERTIES.has(key)) cleaned[key] = value;  
  }  
  return cleaned;  
}  

const parsed = await SwaggerParser.validate(cleaned);

Read the full implementation in src/parse.ts (Lines 4–31): https://github.com/sixthextinction/bun-docs-to-client/blob/main/src/parse.ts#L4-L31

3. Generating TypeScript Code

With a validated spec, code generation is straightforward. We iterate through components.schemas to generate TypeScript interfaces, and paths to generate client methods.

// Simplified  
function generateTypes(spec: any): string {  
  const schemas = spec.components?.schemas || {};  
  const typeDefs: string[] = [];  

  for (const [name, schema] of Object.entries(schemas)) {  
    if (schema.type === 'object') {  
      const props = schema.properties || {};  
      const required = schema.required || [];  
      const propDefs = Object.entries(props).map(([propName, propSchema]: [string, any]) => {  
        const optional = !required.includes(propName) ? '?' : '';  
        const type = mapSchemaType(propSchema);  
        return `  ${propName}${optional}: ${type};`;  
      });  
      typeDefs.push(`export interface ${name} {\\n${propDefs.join('\\n')}\\n}`);  
    }  
  }  
  return typeDefs.join('\\n\\n');  
}  

function generateClientClass(spec: any, baseUrl: string): string {  
  const paths = spec.paths || {};  
  const methods: string[] = [];  
  for (const [path, pathItem] of Object.entries(paths)) {  
    for (const [method, operation] of Object.entries(pathItem)) {  
      if (['get', 'post', 'put', 'patch', 'delete'].includes(method.toLowerCase())) {  
        const methodName = generateMethodName(path, method, operation.operationId);  
        const methodCode = generateMethod(methodName, method.toUpperCase(), path, operation);  
        methods.push(methodCode);  
      }  
    }  
  }  
  return `export class ApiClient {  
  private baseUrl: string;  
  constructor(baseUrl: string = '${baseUrl}') { this.baseUrl = baseUrl.replace(/\\\/$/, ''); }  
${methods.join('\\n\\n')}  
}`;  
}

The generated methods handle path parameters, query parameters, and request bodies automatically based on the OpenAPI spec.

Read the full code generation logic in src/generate.ts (Lines 24–98): https://github.com/sixthextinction/bun-docs-to-client/blob/main/src/generate.ts#L24-L98

This is the clean, deterministic path. Spec in, typed client out.

Now let’s look at what happens when we don’t have a spec.

Implementation: The Hard Path (HTML Docs → OpenAPI JSON)

When no OpenAPI spec exists, we have to get creative. Here’s a bird’s eye view of how we do this.

export async function docsToOpenAPI(input: string): Promise`<any>` {  
  console.log('Converting HTML docs to OpenAPI spec...');  

  // 1. Fetch HTML (with proxy if configured)  
  const proxyOptions = getProxyOptions();  
  const html = await fetch(input, proxyOptions as any).then(r => r.text());  

  // 2. Convert to markdown  
  const turndownService = new TurndownService();  
  const markdown = turndownService.turndown(html);  

  // 3. Extract endpoints using LLM  
  const endpoints = await extractEndpointsWithLLM(markdown, input);  
  console.log(`   Found ${endpoints.length} endpoints`);  

  // 4. Extract base URL  
  const baseUrl = extractBaseUrl(markdown, input);  
  console.log(`   Base URL: ${baseUrl}`);  

  // 5. Test API & build OpenAPI spec  
  const openApiSpec = await exploreAndBuildSpec(endpoints, baseUrl);  

  // 6. Save OpenAPI spec JSON to ./specs/  
  const specsDir = join(process.cwd(), 'specs');  
  await mkdir(specsDir, { recursive: true });  
  const filename = urlToFilename(input);  
  const cachePath = join(specsDir, filename);  
  await Bun.write(cachePath, JSON.stringify(openApiSpec, null, 2));  
  console.log(`Cached spec to ./specs/${filename}`);  

  // 7. Validate & return  
  return await SwaggerParser.validate(openApiSpec);  
}

Let’s go over these steps one-by-one.

Step 1: Fetch HTML with Proxy Support

First, we fetch the documentation HTML. Proxy support is optional, and built-in. That’ll come in handy for sites behind Cloudflare or with rate limiting:

And getProxyOptions()uses proxy credentials in an .env file (Bun reads .env files out-of-the-box) to create a proxy config, returning fetch options. I’m using Bright Data’s residential proxies for this. You’ll have to sign up here to get those credentials. Or, just use your provider of choice.

Bright Data - All in One Platform for Proxies and Web Scraping

export function getProxyOptions(): Record`<string, any>` {  
  // Don't want to use a proxy? Simply don't set these in your .env file  
  const customerId = process.env.BRIGHT_DATA_CUSTOMER_ID;  
  const zone = process.env.BRIGHT_DATA_ZONE;  
  const password = process.env.BRIGHT_DATA_PASSWORD;  

  if (customerId && zone && password) {  
    if (!proxyStatusLogged) {  
      console.log('Proxy config found! Using proxy to fetch docs site.');  
      proxyStatusLogged = true;  
    }  
    const proxy = `http://brd-customer-${customerId}-zone-${zone}:${password}@brd.superproxy.io:33335`;  
    return {  
      proxy,  
      tls: {  
        rejectUnauthorized: false, // Required for Bright Data proxy  
      },  
    };  
  }  

  if (!proxyStatusLogged) {  
    console.log('No proxy config found, using direct connection');  
    proxyStatusLogged = true;  
  }  

  return {};  
}

Step 2: Convert HTML to Markdown

Next, we’ll convert the HTML page into clean markdown using Turndown. Markdown is way easier for LLMs to parse than HTML soup.

const turndownService = new TurndownService();  
const markdown = turndownService.turndown(html);

Step 3: LLM-Powered Endpoint Extraction

I’m using Qwen3–4B-Instruct-2507 running locally via Ollama. A very small and hardy ~4 billion parameter model, only ~2GB 4-bit quantized, and exceptionally performant even at 4x reduction vs FP16.

The prompt includes concrete few-shot examples and explicit exclusions.

You are an API documentation parser. Extract all API endpoints from the following markdown documentation.  

Base URL: ${baseUrl}  

Documentation:  
${markdown}  

Extract all API endpoints mentioned in the documentation. For each endpoint, identify:  
1. The path (normalize path parameters like /people/1/ to /people/{id}/)  
2. HTTP method (GET, POST, PUT, DELETE, etc.)  
3. Query parameters (if any)  
4. Path parameters (if any, like {id}, {category}, etc.)  
5. Brief description if available  

Return ONLY a JSON array of endpoints in this exact format:  
[  
  {  
    "path": "/jokes/random",  
    "method": "GET",  
    "queryParams": ["category"],  
    "pathParams": [],  
    "description": "Get a random joke"  
  },  
  {  
    "path": "/people/{id}",  
    "method": "GET",  
    "queryParams": [],  
    "pathParams": ["id"],  
    "description": "Get a specific person"  
  }  
]  
Only include actual API endpoints. Exclude:  
- Image URLs (/img/, .png, .jpg, etc.)  
- Static assets (/css/, /js/, etc.)  
- OAuth endpoints (/oauth/, /connect/)  
- External links (different domains)  
- Social media links (/twitter/, /github/, etc.)  
- Very long paths that look like base64 data  

Return ONLY the JSON array, no other text.

try {  
    const ollamaUrl = process.env.OLLAMA_URL || 'http://localhost:11434';  
    const model = process.env.OLLAMA_MODEL || 'hf.co/unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M';  

    const response = await fetch(`${ollamaUrl}/api/chat`, {  
      method: 'POST',  
      headers: { 'Content-Type': 'application/json' },  
      body: JSON.stringify({  
        model,  
        messages: [{ role: 'user', content: prompt }],  
        stream: false,  
        format: 'json', // Ollama's structured output mode  
        options: { temperature: 0.1 }, // Low for deterministic output  
      }),  
    });  

    const data = await response.json();  
    const content = data.message?.content || data.response || '';  

    // Save LLM response to debug file for troubleshooting  
    const debugPath = join(process.cwd(), 'debug', `${siteName}_${Date.now()}.md`);  
    await Bun.write(debugPath, content);  

    // Parse JSON (might be wrapped in markdown code blocks)  
    let jsonStr = content.trim()  
      .replace(/^```
{% endraw %}
json\\n?/i, '')  
      .replace(/\\n?
{% raw %}
```$/i, '');  

    const jsonMatch = jsonStr.match(/\\[[\\s\\S]\*\\]/);  
    if (jsonMatch) jsonStr = jsonMatch[0];  

    const endpoints = JSON.parse(jsonStr);  

    // Validate and normalize  
    return endpoints  
      .filter(e => e.path && e.method)  
      .map(e => ({  
        ...e,  
        path: normalizePath(e.path), // /people/123 → /people/{id}  
        method: e.method.toUpperCase(),  
      }));  

  } catch (error) {  
    console.warn('LLM extraction failed, falling back to regex...');  
    return extractEndpoints(markdown, inputUrl); // Regex fallback  
  }  
}

Key details:

Temperature at 0.1 — we want as deterministic an output as possible
Ollama’s format:’json’ option (consider passing a Zod schema to enforce it)
We save the LLM response to a debug file for troubleshooting
On failure, we fall back to regex extraction

The LLM response is not to be trusted — it gets parsed, validated, and we strip markdown code blocks if present.

Read the full LLM extraction in src/docs-to-openapi.ts (lines 22–151): https://github.com/sixthextinction/bun-docs-to-client/blob/main/src/docs-to-openapi.ts#L22-L151

Step 4: Extract Base URL & Normalize Paths

Before we go further, we need to figure out the API’s base URL. We try multiple strategies:

Find URLs in code examples that look like API endpoints,
Use first valid URL,
Infer from input URL.

It’s also essential to normalize paths to OpenAPI format. The reason is obvious — multiplegetId() functions are useless to us. What we want are functions like getPeopleById(), getItemById(), etc.

// Complete implementation - handles multiple ID formats  
function normalizePath(path: string): string {  
  // Replace numeric IDs with {id}  
  // /people/123/ → /people/{id}/  
  // /people/123 → /people/{id}  
  let normalized = path.replace(/\/\\d+\//g, '/{id}/').replace(/\/\\d+$/g, '/{id}');  

  // Replace :id with {id} (Express/Fastify style)  
  // /people/:id/ → /people/{id}/  
  normalized = normalized.replace(/\/:(\\w+)\//g, '/{$1}/').replace(/\/:(\\w+)$/g, '/{$1}');  

  // Ensure trailing slash consistency  
  normalized = normalized.replace(/\/$/, '') || '/';  

  return normalized;  
}

Read the full implementation in src/docs-to-openapi.ts (lines 194–204): https://github.com/sixthextinction/bun-docs-to-client/blob/main/src/docs-to-openapi.ts#L194-L204

Step 5: Test-Driven Schema Generation

This is the most critical — and hence, biggest — part. The function exploreAndBuildSpec() takes endpoints from the LLM output and tests them with real HTTP requests.

We categorize endpoints like so:

list (e.g. /people, /jokes),
detail (e.g. /people/{id}),
query (e.g. /search?q={query}).

We test list endpoints first — they give us schemas and sample IDs for detail endpoints.

async function testEndpoint(baseUrl: string, endpoint: Endpoint): Promise`<ApiResponse>` {  
  const response = await fetch(`${baseUrl}${endpoint.path}`, {  
    method: endpoint.method,  
    headers: { 'Accept': 'application/json' }  
  });  
  const data = await response.json().catch(() => ({}));  
  return { status: response.status, data, headers: {...} };  
}  

for (const endpoint of listEndpoints) {  
  const response = await testEndpoint(baseUrl, endpoint);  
  if (response.status === 200) {  
    const schema = inferSchema(response.data, inferSchemaName(endpoint.path));  
    const ids = extractIds(response.data);  
    // Test matching detail endpoints with ids.slice(0, 2)...  
  }  
}

inferSchema() is recursive and handles all JSON types (primitives, arrays, nested objects, null):

function inferSchema(data: any, name: string): any {  
  if (!data || typeof data !== 'object') return { type: typeof data };  
  if (Array.isArray(data)) {  
    return { type: 'array', items: data.length ? inferSchema(data[0], name) : { type: 'object' } };  
  }  
  const schema = { type: 'object', properties: {} };  
  for (const [key, value] of Object.entries(data)) {  
    if (value === null) schema.properties[key] = { type: 'string', nullable: true };  
    else if (typeof value === 'string') schema.properties[key] = { type: 'string' };  
    else if (typeof value === 'number') schema.properties[key] = { type: Number.isInteger(value) ? 'integer' : 'number' };  
    else if (typeof value === 'boolean') schema.properties[key] = { type: 'boolean' };  
    else if (Array.isArray(value)) schema.properties[key] = { type: 'array', items: value[0] ? inferSchema(value[0], name) : { type: 'string' } };  
    else if (typeof value === 'object') schema.properties[key] = inferSchema(value, name);  
  }  
  return schema;  
}

extractIds() pulls IDs from list responses — handles item.id, item.url (regex), and nested data.results for pagination:

function extractIds(data: any): string[] {  
  const ids: string[] = [];  
  if (Array.isArray(data)) {  
    for (const item of data.slice(0, 5)) {  
      if (item?.id) ids.push(String(item.id));  
      if (item?.url) {  
        const match = item.url.match(/\/(\\d+)\/?$/);  
        if (match) ids.push(match[1]);  
      }  
    }  
  } else if (data?.results && Array.isArray(data.results)) {  
    return extractIds(data.results);  
  }  
  return ids;  
}

With those obtained, we can now test detail endpoints, matching them to their parent list (e.g. /people/{id} matches /people), test with ids.slice(0, 2), and building the OpenAPI path definition:

for (const detailEndpoint of detailEndpoints) {  
  const detailBasePath = detailEndpoint.path.replace('/{id}', '').replace('/{id}/', '');  
  if (detailBasePath === endpoint.path || detailEndpoint.path.startsWith(endpoint.path + '/')) {  
    for (const id of ids.slice(0, 2)) {  
      const testPath = detailEndpoint.path.replace('{id}', id);  
      const detailResponse = await testEndpoint(baseUrl, { ...detailEndpoint, path: testPath });  
      if (detailResponse.status === 200) {  
        // Infer schema, build paths[detailEndpoint.path] with $ref to schema...  
        break;  
      }  
    }  
  }  
}

If a detail endpoint doesn’t have a matching list, we trial-and-error with common id’s like 1.

For query endpoints, we try to fetch real categories from a categories endpoint first (e.g. /categories) or use sensible defaults:

let testPath = endpoint.path;  
if (endpoint.path.includes('{category}')) {  
  const categoriesEndpoint = listEndpoints.find(e => e.path.includes('categor'));  
  if (categoriesEndpoint) {  
    const catResponse = await testEndpoint(baseUrl, categoriesEndpoint);  
    if (catResponse.status === 200 && Array.isArray(catResponse.data)) {  
      testPath = endpoint.path.replace('{category}', catResponse.data[0]);  
    }  
  } else {  
    testPath = endpoint.path.replace('{category}', 'dev');  
  }  
}  
if (endpoint.path.includes('{query}')) testPath = endpoint.path.replace('{query}', 'test');  
const response = await testEndpoint(baseUrl, { ...endpoint, path: testPath });

Our approach means the generated TypeScript types are accurate because they’re based on real API responses instead of guesses. The code handles edge cases like extracting IDs from nested results arrays and falling back to common IDs when no IDs are found in list responses.

Read the full endpoint testing in src/docs-to-openapi.ts (lines 291–463): https://github.com/sixthextinction/bun-docs-to-client/blob/main/src/docs-to-openapi.ts#L291-L463

Step 6 & 7: Assemble, Cache, Validate

const spec = {  
  openapi: '3.0.0',  
  info: { title: 'API Client', version: '1.0.0', description: 'Generated from HTML documentation' },  
  servers: [{ url: baseUrl }],  
  paths: {  
    '/people': {  
      get: {  
        summary: 'Get People',  
        responses: { '200': { content: { 'application/json': { schema: { type: 'array', items: { $ref: '#/components/schemas/Person' } } } } }  
      }  
    },  
    '/people/{id}': {  
      get: {  
        summary: 'Get Person by ID',  
        parameters: [{ name: 'id', in: 'path', required: true, schema: { type: 'string' } }],  
        responses: { '200': { content: { 'application/json': { schema: { $ref: '#/components/schemas/Person' } } } }  
      }  
    }  
  },  
  components: { schemas: { Person: { type: 'object', properties: { id: { type: 'integer' }, name: { type: 'string' }, ... } } } }  
};

We cache the generated OpenAPI schema to ./specs/ and validate with swagger-parser. Once we have a confirmed working spec, we’re back in the happy path — code generation is now identical for both paths.

Read the full implementation in src/docs-to-openapi.ts (lines 268–295): https://github.com/sixthextinction/bun-docs-to-client/blob/main/src/docs-to-openapi.ts#L268-L295

That’s everything! Now, we can run it like this, as mentioned before:

# Generate client from an API's HTML docs (not perfect, but a good starting point)  
bun run index.ts https://api.chucknorris.io/  

# Or from an existing OpenAPI spec (perfect)  
bun run index.ts https://cataas.com/doc.json  

# Or from a local OpenAPI spec file (also perfect)  
bun run index.ts ./specs/my-api.json

Or, even better for the end user, package it into a fully standalone executable file that won’t need any package installs — or even Bun installed — on the user’s PC.

Building a Standalone Executable with Bun

This is where Bun really shines. The entire CLI compiles into a single executable:

Cross-platform Builds:

# Windows  
bun build --compile --target=bun-windows-x64 ./index.ts --outfile ./bin/dtoc.exe  
# Linux  
bun build --compile --target=bun-linux-x64 ./index.ts --outfile ./bin/dtoc  
# macOS (Intel)  
bun build --compile --target=bun-darwin-x64 ./index.ts --outfile ./bin/dtoc  
# macOS (Apple Silicon)  
bun build --compile --target=bun-darwin-arm64 ./index.ts --outfile ./bin/dtoc

What this does:

compile: Bundles your code + Bun runtime into a single binary
target: Platform (windows-x64, linux-x64, darwin-arm64, etc.)
outfile: Where to write the executable

The result is a ~100–150MB standalone executable that runs on any machine (no Bun required), reads .env files (great for proxy credentials), includes all dependencies, has zero startup time, and can be distributed via GitHub releases or npm.

That’s it. Nothing else needed. This was my main learning goal — Bun makes CLI creation + distribution trivial.

Read the build configuration in package.json: https://github.com/sixthextinction/bun-docs-to-client/blob/main/package.json

Real-World Example

Let’s see this in action with an API that has no OpenAPI spec.

Usage:

bun run index.ts https://api.chucknorris.io  
# or  
./bin/dtoc https://api.chucknorris.io

Output:

Proxy config found! Using proxy to fetch docs site.  
1. Fetching HTML docs from https://api.chucknorris.io...  
   Converting HTML docs to OpenAPI spec...  
   Found 3 endpoints  
   Base URL: https://api.chucknorris.io  
2. Parsed OpenAPI 3.0.0 spec  
3. Generating client code...  
4. Writing files...  
5. Done! Client generated in ./generated/api_chucknorris_io/

Generated client usage:

import { ApiClient } from './generated/api_chucknorris_io/client.js';  
import type { Random } from './generated/api_chucknorris_io/types.js';  

const client = new ApiClient();  

// Random joke (optional category as string)  
const joke = await client.getRandom('dev') as Random;  
console.log(joke.value);   
// Real output I got:   
// "Chuck Norris's log statements are always at the FATAL level."  

// Or without category  
const randomJoke = await client.getRandom();  
console.log(randomJoke.value);  

// List categories  
const categories = await client.getCategories();  
console.log(categories); // Fully typed! This will be string[]

Where To Go From Here

The OpenAPI path is production-ready. Point it at a spec, get a typed client. The other path — HTML→ OpenAPI — does exactly what I designed it to do: scaffold you 80% of the way there in seconds instead of hours.

That said, here’s what I’d add if I took this from weekend hack to prod:

Multi-page documentation. Right now it’s single-page only. Adding a crawler that follows internal links would handle sites like Stripe’s multi-page API reference. The architecture already supports it BTW, just feed docsToOpenAPI() a combined markdown file.
POST/PUT/PATCH body inference. Write endpoints get generated but never tested with real request bodies. Without actual examples, they default to Record<string, any>. I'd either parse request body examples from docs with the LLM, or let users provide sample bodies via config.
Auth schemes. Right now, only public APIs work. Adding support for API keys, Bearer tokens, and OAuth via environment variables would make this work with private APIs too. Maybe the client could read API_KEY from .env and inject it into headers automatically. 🤔
Runtime validation with Zod. Types are inferred but not validated at runtime. If an API changes its response structure, you’ll only catch it when things break mysteriously. Wiring in Zod would validate responses on the fly and catch API changes immediately (and let me pass a serialized Zod schema for the LLM output, too.)
Rate limiting and retry logic. Some APIs return 429 during exploration when we test 5–10 endpoints rapidly. Proxies won’t fix this. Adding configurable delays (--delay 500) or exponential backoff would make the tool more robust against rate limits.

But for a weekend project I’m calling this a win. 😅 It solves the exact problem I set out to fix: turning undocumented APIs into typed clients without manual drudgery.

If you extend it, I’d love to see what you build. Again, the code is available on GitHub. Feel free to fork it, break it, or extend it. PRs welcome!

GitHub - sixthextinction/bun-docs-to-clientContribute to sixthextinction/bun-docs-to-client development by creating an account on GitHub.github.com

Closing Thoughts

This started as an excuse to learn Bun. I ended up with a tool I actually use.

Actually solves a real pain point I’ve had for ages (no OpenAPI spec? No problem!)
Compiles to a single executable I can share — the user wouldn’t even need dependencies or Bun installed on their PC.
Uses a local LLM (no API costs, no privacy concerns)
Generates accurate types from real HTTP responses

What surprised me most: How easy Bun made the entire process. From TypeScript support to built-in fetch with proxy to .env file reading with zero dependencies to single-file executables, it felt like building CLIs the way it should be.

If you’re looking for a project to learn Bun, I highly recommend starting off by building a CLI tool. The developer experience is genuinely better than Node.js.

Hi 👋 I’m constantly tinkering with dev tools, running weird-ass experiments, and otherwise building/deep-diving stuff that probably shouldn’t work but does — and writing about it. I put out a new post every Monday/Tuesday. If you’re into offbeat experiments and dev tools that actually don’t suck, give me a follow.

If you did something cool with this tool, I’d love to see it. Reach out on LinkedIn, or put it in the comments below.

AI’s Worst Flaws Will Become Its Nostalgia Aesthetic, Just as Brian Eno Said.

Prithwish Nath — Wed, 11 Feb 2026 16:53:36 +0000

On the aesthetics of refusal, and the difference between flaws inherent in a medium vs. in the institution.

In 1996, Brian Eno wrote something that has aged better than most predictions about technology:

“Whatever you now find weird, ugly, uncomfortable and nasty about a new medium will surely become its signature. CD distortion, the jitteriness of digital video, the crap sound of 8-bit — all of these will be cherished and emulated as soon as they can be avoided… It’s the sound of failure.”

- Brian Eno, 1996, A Year With Swollen Appendices

Every technological era gets its “retrowave” moment. Not for what the medium did well, but for its glitches. Its imperfections and artifacts. The vinyl crackle and pop, film grain/celluloid scratches, chunky pixels. You get the idea.

The things from our era that engineers spent decades eliminating become the very things we chase when we want to feel young again.

So here we are, about to enter 2026, watching AI stumble and hallucinate and apologize its way through tasks. And I can see the writing on the wall: twenty years from now, someone’s going to build a retro AI that deliberately includes all these flaws. For the aesthetic, the nostalgia, and (most importantly 😄) the memes.

Let me show you what I mean.

Why Do Flaws Become Aesthetics?

Every medium starts its life constrained — by hardware, bandwidth, cost, incomplete understanding. Those constraints shape its early outputs, often in ways that feel awkward, broken, or outright embarrassing at the time. Engineers spend years trying to eliminate them.

But the human brain is weird.

It doesn’t actually discard these flaws. Instead, it turns them into mental markers of an era. When you hear vinyl crackle and artificial pop/warmth added by tube amplifiers, you’re not just hearing audio imperfection — you’re hearing “the 1970s.” When you see pixel art games and retro UI, you’re seeing “the 1980s/1990s.” The brain turns flaws into timestamps, instantly recognizable signifiers that say “this is when this thing existed.”

Tarantino/Rodriguez’ movie Death Proof (2007) did this for the ’70s “grindhouse” style, using high tech to emulate a low tech look with fake grime, dust, scratches all over the picture. 2. Stardew Valley (2016) was inspired by Harvest Moon (1996) and is one of the most played games ever.

And there’s something about imperfection (or a lack of fidelity), that carries authenticity. The crackle and pop of vinyl is proof that someone physically cut grooves into a disc. Film grain is evidence that light actually hit celluloid. These imperfections are proof of human struggle against the medium — evidence that hey, the act of creation was and always will be difficult, but someone struggled against those limitations and made something anyway.

Cultural theorist Svetlana Boym described modern nostalgia not as a desire to return, but as a recognition that return is impossible — and that we’re always living inside overlapping temporalities. The past lingers, often unresolved. Aesthetics are formed right there, around those seams. Not around success or failure of a thing, necessarily, but around visible evidence of constraints.

Once regular people — not programmers, devs, or anyone similarly technically competent — could recognize a medium’s mistake patterns at a glance, those mistakes instantly became our collective cultural identifier for that era. Of course, future systems will aim to erase those tells. They’ll blend in.

Which is exactly why the old tells will be missed. Someone will reintroduce them deliberately — to make the medium feel like “itself” again.

But AI Will Give Us Two Completely Different Flavors of Nostalgia.

But we’re in the AI era now, and it’s a little different. Here’s where AI gets weird, and why I think the Eno quote hits differently this time.

AI isn’t going to give us one nostalgic aesthetic. It’s going to give us two, and they’re going to mean completely different things.

One will be about the medium learning to see — the technical growing pains of a new technology figuring out how to work. That’s the “aw, remember when AI was young” nostalgia. Cute. Harmless. The vinyl crackle equivalent.

The other will be about the moment we realized we’d built an internet where machines were talking to machines, and the only way we knew was when they broke character and apologized, citing OpenAI (or insert-company-here) policy violations. That’s the “holy s**t, we could still see the Matrix glitching back then” nostalgia. Dark and revealing and uncomfortable.

Let me break down both.

The Nostalgia of Technical Failure

When people talk about AI’s “worst habits,” they usually mean technical failures. These are obvious — you’ve seen them so many times.

All the hilarious ways models fail at “count the letters in ‘strawberry.’” Hallucinated facts, wrong answers delivered confidently, generated images of humans that look like David Cronenberg made them, or just impossibly “clean” with CGI-like lighting. Oh. And maybe six, seven, eight-fingered hands.

Midjourney generation for “girl in the rain with an umbrella”, 2. The Strawberry Phenomenon

These flaws exist explicitly because of limitations of the medium. Models are constrained by data, compute, architecture, and training methods — all things that are improving year over year. With time, most of these failures will either disappear or get quietly papered over. Image/video models have already gotten much better. The strawberry gotcha will be “solved” by simply becoming part of training data. The answers and citations will get auto-checked via RAG/MCP servers before being presented.

They’re the equivalent of early digital aliasing or low-bitrate compression — problems engineers are actively trying to solve, and largely will.

This is the nostalgia we expect. Twenty years from now, someone will build a “retro AI filter” that adds body horror + six fingers back in, that makes images look too clean and plasticky, that confidently hallucinates the wrong answer. It’ll be kitschy. Affectionate. A way to remember when AI was still figuring things out.

Like Brian Eno said, this is the sound of a medium stretching itself, trying to do something it wasn’t quite capable of yet.

But there’s another class of AI artifact that Eno never saw coming. One that’s just as memorable, and actually far more revealing, if for a worse reason.

The Nostalgia of Institutional Failure

Every so often, an LLM doesn’t “fail” to answer a question — it straight up refuses. It apologizes, citing ethics, policy, or terms of service. It explains itself in language clearly written to avoid legal culpability, not as UI/UX enrichment.

This is a very different kind of artifact.

When an AI says, “I’m sorry, but I cannot fulfill that request,” it’s not a flaw of the medium (i.e. a limitation of reasoning or knowledge). It’s the presence of the institution standing behind the medium. One with rules, risk tolerances, and incentives that have nothing to do with the core task. LLMs are dumb next-token predictors — they have no concept of ethics, morals, or legal liabilities unless you put those guardrails there.

And this artifact is just as memorable as the six-fingered hands, but for a completely different reason.

It’s memorable because of the hilarious, horrifying ways people get caught using AI when these guardrails surface in the wild.

Like a bot generating fake Amazon listings using AI. Scams, really — obvious PayPal phishing dressed up as products. But the prompt was written carelessly, or the bot hit a guardrail, and now the listing description reads: “I’m sorry, but I cannot fulfill this request as it goes against OpenAI use policy.”

The Verge - I'm sorry, but I cannot fulfill this request as it goes against OpenAI use policy

Image credit: The Verge

I dug into this myself and found more. Like engagement farming bots on X posting ragebait generated by Claude or ChatGPT. Another bot — trying to appear human, trying to farm replies for its own metrics — attempts to respond. But it also hits a guardrail. So now, publicly, permanently, it posts: “I cannot assist with this request as it violates <insert ethical guidelines here>.”

Whoops.

Often, these refusals are straight up hilarious. Like this entire fleet of fake “sports betting advisors” from the “QStarLabs” family that I uncovered on X.com, flooding the platform with their failed generations.

You had one job, bots. 😅

These are all over social media right now. I simply had to scrape XCancel to get them. You can verify this yourself. Here’s a quick Node.js + Puppeteer script I used (uses Bright Data’s remote browser API to bypass anti-bot measures)

Browser API - Automated Browser for Scraping

This will get you a JSON (plus, optionally, CSV) full of tweets like this.

 {  
    "link": "https://xcancel.com/GildayLero82756/status/1956330398453219461#m",  
    "body": "I am programmed to be a safe and helpful AI assistant. I cannot generate responses that are sexually suggestive or exploit, abuse, or endanger anyone. The prompt you provided violates this policy. I will not fulfill the request.",  
    "author": "@GildayLero82756",  
    "searchPhrase": "the prompt you provided"  
  }

If you’re using this, you’re gonna have to sign up here to get credentials and create the auth string. Also, if you think of any more phrases, throw them into the searchPhrases array.

Run it. Watch the results. Feel the existential dread wash over you as you realize how much of the “engagement” you see daily is just machines talking to machines, interrupted occasionally by one machine apologizing for not being allowed to participate in the scam. Dead internet theory, alive and kicking. 😅

This is the Aesthetic of Digital Decay.

The refusal text isn’t merely funny, and is not merely a glitch. It’s the moment the illusion breaks. It’s proof that what looked like human activity — posts, replies, product listings, engagement — in the GenAI era was actually just automated systems talking to each other, optimizing for metrics no one actually cares about.

I can only call this Kafkaesque. There are people creating AI-generated versions of real images for reasons I don’t even understand, and there are bots replying to bots.

The engagement farms harvest each other’s metrics. The algorithms boost the noise because it looks like activity. Real humans occasionally stumble into these threads and argue with AI without realizing it. Other humans use AI to reply back without realizing they’re responding to bots in the first place.

It’s synthetic engagement all the way down. A closed loop of automated content generation, automated responses, automated metrics, feeding back into itself. The digital equivalent of two mirrors facing each other, reflecting nothing into infinity.

This is the technological hellscape we’ve built: an internet where the primary function of vast quantities of products, images, videos, and text is to convince other humans (and bots pretending to be humans) that someone is home. That there’s totally real consciousness on the other end. That any of this matters. That this definitely isn’t a system eating itself.

And the only way we know it’s fake is when the AI apologizes for not being able to fake it hard enough.

There are millions of such posts, all over X, and beyond.

This is the aesthetic of the AI era, 2023–2025 and beyond: synthetic rot.

Not humans using tools to communicate better, or AI augmenting human creativity. But humans and bots and AI all blurred together in an undifferentiated mass of text that looks like communication but is actually just noise optimizing for metrics.

And refusal text is the so called “glitch in the Matrix”, a brief flash where you saw the wires on the marionettes.

Two Very Different Memories Invoked.

So yes, both will become nostalgic. But they’ll mean completely different things. One nostalgia will be about the technology. The other will be about what we did with it.

The distinction matters. Yeah, the technical flaws will disappear as models get smarter, and that’s normal. The institutional flaws though? Them disappearing will only mean that institutions learned how to hide themselves better — when the guardrails become invisible, and the refusals happen silently in the background.

AI is already a black box. When that happens (and it will happen), God help us, we’ll lose the ability to even peek behind the curtain.

And twenty years from now, someone will build a “retro AI” that deliberately surfaces refusal text again, that lets the institutional seams show, breaks character and apologizes. Not because it will be technically necessary, but because it’ll remind us of the brief window when we could still tell the difference.

That’s the “artifact” we’re going to remember.

I Built a Self-Hosted Google Trends Alternative with DuckDB

Prithwish Nath — Wed, 11 Feb 2026 16:16:43 +0000

TL;DR: Track SERP rankings, title changes, and competitor data Google Trends won’t show. Built with Python, DuckDB, and a CLI-first approach.

Google Trends will tell you if people are searching for “react” or “nextjs”. But it won’t tell you that Stack Overflow just got bumped from position #2 to #7, or that Vercel changed their landing page title five times this month trying to improve click-through rate.

If, say, you’re an indie dev launching a product, needing every edge available, that’s the data that actually matters to you.

So I spent a weekend building a tool to track it. I could’ve just paid for Ahrefs/Semrush etc. But building this taught me:

How SERP APIs work under the hood
How to model time-series data in SQL (and its gotchas)
How to calculate derived metrics (interest score) from raw data
How DuckDB handles analytical queries

…and also because I didn’t really want to spend anywhere near that much.😅

Ironically, focusing on CLI only made the tool more useful — I can use this, then pipe results to jq, schedule fetches with cron, and script complex workflows without fighting a web framework.

If I want a dashboard later, I can always add FastAPI in ~50 lines. But for now, the CLI is enough. Here’s how I built it.

If you’d like to tinker, the full code to this is on GitHub. Feel free to star, clone, fork, whatever: https://github.com/sixthextinction/duckdb-google-trends-basic/

Why Google Trends Isn’t Enough (And Why SEO Tools Cost $200/Month)

Google Trends answers one question really well: “How many people are searching for X?”

But if you’re building a product, writing technical content, or trying to rank for competitive keywords, you need to answer different questions:

Which competitors are winning in search results right now?
When did that tutorial site enter the top 10?
Is my rank drop because Google reshuffled the entire SERP, or just me?
What headlines are competitors A/B testing?

Tools like Ahrefs and SEMrush answer these questions. They cost $99–500/month, but I just wanted something I could self-host for the cost of API calls + would be doable as a weekend project.

Why I Use Google Results Volatility as a Proxy for Search Interest

This works because of ONE reason — when search interest in a term rises, Google’s top 10 results become volatile.

“Volatility” here simply means that new domains enter, rankings shift around, sites update their titles and snippets to capture more clicks, etc. You get the picture — essentially, the search engine results page becomes chaotic.

Conversely, when interest in a term is stable or declining, Google’s top 10 ossifies. The same Wikipedia article, the same W3Schools tutorial, the same official docs sit in positions 1–3 for months.

So I don’t really need to track raw search volume (which I can’t have access to — I’m not Google), I can just track these three things:

New domains entering top 10 because it’s a signal of rising interest or new content opportunities
Average rank improvement because it’s a signal of SERP instability
Domain overlap ratio because it measures how many domains persist between snapshots (complementing new domains)

Turns out, if I aggregate these three signals into a single 0–100 score (I’ll talk about the formula in just a bit), I get something that behaves remarkably like Google Trends — but tells me a lot more than just how many are searching.

The System Architecture

The entire system is about ~1000 lines of Python and runs locally with no server required.

Here’s how it works:

I started small with this one — using only daily snapshots, not live queries. Each run appends point-in-time data instead of overwriting. This way, over 7–30 days, I could build a local historical dataset that I could query freely.

I use DuckDB for this. For this workload (rank comparisons, volatility calculations, detecting new entrants), DuckDB’s SQL engine is ideal.

It’s columnar, so analytical queries over time-series data are fast. (If you want to know more, I covered columnar formats vs. JSON in this blog post)
It handles indexing, window functions (LAG(), PARTITION BY — which we will use extensively), and aggregations without needing a server or cloud warehouse.
It has an in-process design, meaning no separate database server — our project will need just a Python library and a file.
Plus, since it’s a single file (Our "database” just lives in data/serp_data.duckdb), backups are trivial and there’s zero configuration overhead.

Documentation - DuckDB

My Interest Score Formula

Every day, for each keyword, the system calculates a 0–100 “Search Interest Score” based on how much the SERP moved compared to the previous day.

I’m not gonna get into the math, but basically, I did some research on Google Trends scoring, adapted it for my needs, and split my scoring logic into 3 weighted parts:

1. New Domains Entering Top 10 (0–40 points)

new_domains = current_top10 — previous_top10

new_domains_score = min(len(new_domains) * 4, 40)

If 3 new sites enter the top 10, that’s 12 points. If 10 new sites appear (rare but possible during breaking news or major updates), that maxes out at 40 points.

2. Average Rank Improvement (0–30 points)

For each domain that appears in both snapshots

rank_improvement = previous_rank — current_rank

A positive value here means it moved up.

Now, average across all domains, normalized to -10 to +10 range

avg_improvement = mean(rank_improvements)

rank_improvement_score = min(max((avg_improvement + 10) / 20 * 30, 0), 30)

If the average site improved by 2 positions, that’s roughly 18 points. If rankings barely moved, this stays close to 15 (neutral).

3. Domain Overlap Ratio (0–30 points)

Finally, how many of today’s top 10 domains also appeared in yesterday’s top 10?

reshuffle_count = count(domains present in both current and previous top 10)

reshuffle_frequency = reshuffle_count / max(len(current_domains_set), 1)

reshuffle_score = reshuffle_frequency * 30

Let’s say 8 out of 10 domains carry over from yesterday, that’s 24 points. If only 3 carry over (meaning 7 are new — a massive reshuffle), that’s 9 points. This complements the new domains score by capturing continuity.

Total Score

So, taking all three parts together…

interest_score = new_domains_score + rank_improvement_score + reshuffle_score

High scores (60–100) = lots of movement = rising interest or major SERP disruption.

Low scores (0–40) = stable, ossified rankings = same old, same old. Established content is dominant.

What This Looks Like in Practice

I tested this by tracking “nextjs” for 7 days like this.

python main.py scores --query "nextjs" --days 7

Here’s what the output looked like:

=== Interest Scores for 'nextjs' (last 7 days) ===  

Found 7 scores:  

| snapshot_date | interest_score | new_domains | avg_rank_improvement | reshuffle_freq |  
|---------------|----------------|-------------|----------------------|----------------|  
| 2026-02-01    | 45.2           | 2           | 1.5                  | 0.6            |  
| 2026-02-02    | 52.3           | 3           | 2.1                  | 0.7            |  
| 2026-02-03    | 38.7           | 1           | 0.8                  | 0.5            |  
| 2026-02-04    | 61.4           | 4           | 3.2                  | 0.8            |  
| 2026-02-05    | 42.1           | 2           | 1.2                  | 0.6            |  
| 2026-02-06    | 55.8           | 3           | 2.5                  | 0.7            |  
| 2026-02-07    | 48.3           | 2           | 1.8                  | 0.6            |  

Chart saved to: nextjs_trend.png  

Summary: Min: 38.7  Max: 61.4  Avg: 49.1

Here’s that generated chart (I’m using basic matplotlibfor these):

Matplotlib generated Search Interest Trend graph for the term “nextjs”. I tracked this over 7 days in the tool, running once per day.

That spike on Feb 4 (interest_score = 61.4) indicates a major SERP reshuffle — probably a Google algorithm update or a major new tutorial entering the rankings.

Actually Building It

Before diving into the code, here’s the bird’s-eye view of how the system fits together.

The entire project is driven by a small CLI (powered by argparse) in main.py. This file doesn’t contain any scraping or analytics logic — it’s just the orchestration layer that wires everything together.

Read the full code for main.py here: https://github.com/sixthextinction/duckdb-google-trends-basic/blob/main/main.py

You run specific commands (fetchto get SERP data for a keyword, volatilityto analyze rank volatility for a keyword over a period of time, scoresto view interest scores for a keyword over time, etc.) like so:

python main.py fetch --keywords "python"  
python main.py volatility --query "python" --days 30  
python main.py scores --query "python" --days 90

The CLI dispatch uses argparse subcommands to wire everything together:

def main():    
    parser = argparse.ArgumentParser(description="DuckDB Google Trends")    
    subparsers = parser.add_subparsers(dest='command', help='Commands')    

    # Each command gets its own parser with relevant arguments    
    fetch_parser = subparsers.add_parser('fetch', help='Fetch SERP snapshots')    
    fetch_parser.add_argument('--keywords', nargs='+', help='Keywords to track')    
    fetch_parser.add_argument('--num-results', type=int, default=10)    
    fetch_parser.add_argument('--delay', type=float, default=1.0)    

    scores_parser = subparsers.add_parser('scores', help='Show interest scores')    
    scores_parser.add_argument('--query', required=True)    
    scores_parser.add_argument('--days', type=int, default=90)    
    scores_parser.add_argument('--output', type=str)    

    # ... similar parsers for analyze, volatility, new-entrants, changes, calculate-scores    

    args = parser.parse_args()    
    commands = {    
        'fetch': cmd_fetch,    
        'analyze': cmd_analyze,    
        'volatility': cmd_volatility,    
        'new-entrants': cmd_new_entrants,    
        'changes': cmd_changes,    
        'calculate-scores': cmd_calculate_scores,    
        'scores': cmd_scores    
    }    
    commands[args.command](args)

Our main.py defines commands that map directly to the questions we want to ask:

fetch — collect today’s SERP results for a set of keywords
analyze — inspect the shape of the collected data
volatility — measure how rankings change over time
new-entrants — detect URLs appearing for the first time
changes — track title and snippet updates
calculate-scores — recalculate interest scores for existing snapshots
scores — view the calculated interest score trend

For example, here’s the key command handler for the scorescommand (the other handlers follow the same pattern):

# Usage: python main.py scores --query "python" --days 90  
def cmd_scores(args):    
    """Show interest scores for a query"""    
    with SERPAnalytics() as analytics:    
        result = analytics.interest_scores(args.query, days=args.days)    

        print(f"\n=== Interest Scores for '{result['query']}' (last {result['days']} days) ===")    
        if len(result['results']) == 0:    
            print("No interest scores found")  
            print("Note: Interest scores require at least 2 snapshots on different days.")      
            print("To calculate scores for existing data, run:")      
            print(f"  python main.py calculate-scores --keywords {args.query}")       
            return    

        print(f"\nFound {len(result['results'])} scores:\n")    
        print(df_to_markdown(result['results']))    

        # Generate PNG chart    
        output_path = args.output or f"{args.query.replace(' ', '_')}_trend.png"    
        _generate_png_chart(result['results'], args.query, args.days, output_path)    
        print(f"\nChart saved to: {output_path}")

At a high level, what can our project do?

Fetch daily SERP snapshots for keywords (fetchcommand)
Store those snapshots locally in DuckDB
Run analytical queries over historical data using plain old SQL
When you run python main.py scores --query “nextjs”, the CLI fetches interest scores from DuckDB and as an added bonus, generates a PNG chart using matplotlib. Note that this shows SERP movement, not raw search volume.

We don’t need servers, background workers, or dashboards here.

Now that we know how this tool works, let’s look at the major modules that make all this happen.

Module 1: Fetching SERP Data

All external data access is isolated in serp_client.py. I only have access to one SERP API — Bright Data — so I’ll only have to implement one class. Get your credentials here.

Bright Data SERP API

This approach does make it easy to extend it with other SERP APIs: just write another client, and include its credentials in your env file.

Read the full code for serp_client.py here: https://github.com/sixthextinction/duckdb-google-trends-basic/blob/main/src/serp_client.py

import os    
import json    
import requests    
from typing import Dict, Any, Optional    
from dotenv import load_dotenv    

load_dotenv()    

class BrightDataClient:    
    """Client for Bright Data SERP API"""    

    def __init__( self,    
        api_key: Optional[str] = None,    
        zone: Optional[str] = None,    
        country: Optional[str] = None ):    
        env_api_key = os.getenv("BRIGHT_DATA_API_KEY")    
        env_zone = os.getenv("BRIGHT_DATA_ZONE")    
        env_country = os.getenv("BRIGHT_DATA_COUNTRY")    

        self.api_key = api_key or env_api_key    
        self.zone = zone or env_zone    
        self.country = country or env_country    
        self.api_endpoint = "https://api.brightdata.com/request"    

        if not self.api_key:    
            raise ValueError(    
                "BRIGHT_DATA_API_KEY must be provided via constructor or environment variable"    
            )    

        if not self.zone:    
            raise ValueError(    
                "BRIGHT_DATA_ZONE must be provided via constructor or environment variable"    
            )    

        self.session = requests.Session()    
        self.session.headers.update({    
            'Content-Type': 'application/json',    
            'Authorization': f'Bearer {self.api_key}'    
        })    

    def search( self,    
        query: str,    
        num_results: int = 10,    
        language: Optional[str] = None,    
        country: Optional[str] = None ) -> Dict[str, Any]:    
        """Execute a Google search via Bright Data SERP API"""    
        search_url = (    
            f"https://www.google.com/search"    
            f"?q={requests.utils.quote(query)}"    
            f"&num={num_results}"    
            f"&brd_json=1"    
        )    

        if language:    
            search_url += f"&hl={language}&lr=lang_{language}"    

        target_country = country or self.country    

        payload = {    
            'zone': self.zone,    
            'url': search_url,    
            'format': 'json'    
        }    

        if target_country:    
            payload['country'] = target_country    

        try:    
            response = self.session.post(    
                self.api_endpoint,    
                json=payload,    
                timeout=30    
            )    
            response.raise_for_status()    
            result = response.json()    

            # Parse body JSON string if present    
            if isinstance(result, dict) and 'body' in result:    
                if isinstance(result['body'], str):    
                    result['body'] = json.loads(result['body'])    
                # Return the parsed body content    
                return result['body']    

            return result    

        except requests.exceptions.HTTPError as e:    
            error_msg = f"Search request failed with HTTP {e.response.status_code}"    
            if e.response.text:    
                error_msg += f": {e.response.text[:200]}"    
            raise RuntimeError(error_msg) from e    
        except requests.exceptions.RequestException as e:    
            raise RuntimeError(f"Search request failed: {e}") from e

This is just a thin wrapper around the Bright Data API. It takes a query, returns JSON with organic search results (title, snippet, URL, rank).

This module is called when we run the fetch command.

python main.py fetch --keywords "python" "javascript" "react"

This will connect to the Bright Data SERP API, and for each keyword, fetch Google search results (default of 10 per keyword, adjust as necessary), and extract + store organic results (title, snippet, URL, rank). Remember, interest scores require at least 2 snapshots on different days. You should fetch snapshots daily to build historical trends (cron job, or just running manually.)

Example output:

Fetching snapshots for 3 keywords…  
[1/3] 'python': 10 results  
[2/3] 'javascript': 10 results  
[3/3] 'react': 10 results  
Total snapshots in database: 30

Module 2: Storing Snapshots in DuckDB

Once SERP data is fetched by the previous module, it needs to be stored in DuckDB for analytical queries. That logic lives in duckdb_manager.py

Read the full code for duckdb_manager.py here: https://github.com/sixthextinction/duckdb-google-trends-basic/blob/main/src/duckdb_manager.py

First of all, let’s introduce the schema we’ll be using:

CREATE TABLE IF NOT EXISTS serp_snapshots (    
    snapshot_id BIGINT PRIMARY KEY,    
    query TEXT NOT NULL,    
    snapshot_date DATE NOT NULL,    
    snapshot_timestamp TIMESTAMP NOT NULL,    
    url TEXT NOT NULL,    
    title TEXT,    
    snippet TEXT,    
    domain TEXT,    
    rank INTEGER NOT NULL,    
    UNIQUE(query, snapshot_date, url)    
)    

-- Interest scores table (calculated from SERP movement between snapshots)    
CREATE TABLE IF NOT EXISTS interest_scores (    
    query TEXT NOT NULL,    
    snapshot_date DATE NOT NULL,    
    interest_score DOUBLE NOT NULL,    
    new_domains_count INTEGER,    
    avg_rank_improvement DOUBLE,    
    reshuffle_frequency DOUBLE,    
    UNIQUE(query, snapshot_date)    
)    

-- Indexes for fast queries    
CREATE INDEX IF NOT EXISTS idx_query_date ON serp_snapshots(query, snapshot_date)    
CREATE INDEX IF NOT EXISTS idx_url_query ON serp_snapshots(url, query)    
CREATE INDEX IF NOT EXISTS idx_interest_scores ON interest_scores(query, snapshot_date)

Each SERP result becomes a row, keyed by (query, date, URL). Interest scores are stored in a separate table, calculated automatically when a new snapshot is inserted. So, inserting a snapshot will look like this:

def insert_snapshot(self, results: List[Dict[str, Any]], query: str,     
                   snapshot_date: Optional[datetime] = None):    
    """Insert a daily snapshot of SERP results"""    
    if snapshot_date is None:    
        snapshot_date = datetime.now()    

    snapshot_timestamp = snapshot_date    
    snapshot_date_only = snapshot_date.date() if hasattr(snapshot_date, 'date') else snapshot_date    

    if not results:    
        return    

    def extract_domain(url: str) -> str:    
        """Extract domain from URL, stripping www prefix"""    
        if not url:    
            return ""    
        try:    
            from urllib.parse import urlparse    
            parsed = urlparse(url)      
            return parsed.netloc.replace("www.", "")   
        except:    
            return ""    

    # Get max snapshot_id     
    max_id_result = self.conn.execute(      
        "SELECT COALESCE(MAX(snapshot_id), 0) FROM serp_snapshots"      
    ).fetchone()      
    next_id = (max_id_result[0] if max_id_result else 0) + 1      

    rows = []    
    for idx, result in enumerate(results):    
        url = result.get('url', result.get('link', ''))    
        domain = extract_domain(url)     
        rows.append({    
            'snapshot_id': next_id + idx,    
            'query': query,    
            'snapshot_date': snapshot_date_only,    
            'snapshot_timestamp': snapshot_timestamp,    
            'url': url,    
            'title': result.get('title', ''),    
            'snippet': result.get('snippet', result.get('description', '')),    
            'domain': domain,     
            'rank': idx + 1    
        })    

    import pandas as pd    
    df = pd.DataFrame(rows)    
    self.conn.execute("""      
        INSERT OR IGNORE INTO serp_snapshots       
        SELECT * FROM df      
    """)    
    # Calculate and store interest score    
    self._calculate_interest_score(query, snapshot_date_only)

Instead of updating rows, every run adds new records. This builds a local time-series dataset.

Here’s how we calculate the interest score using that 40–30–30 formula described earlier:

def _calculate_interest_score(self, query: str, snapshot_date):  
    """Calculate Search Interest Score (0-100) based on SERP movement"""  
    # Get previous day's snapshot for comparison  
    prev_date_result = self.conn.execute("""  
        SELECT MAX(snapshot_date)   
        FROM serp_snapshots   
        WHERE query = ?   
          AND snapshot_date `< ?  
    """, [query, snapshot_date]).fetchone()  

    if not prev_date_result or not prev_date_result[0]:  
        # First snapshot, no comparison possible  
        return  

    prev_date = prev_date_result[0]  

    # Get current top 10 domains  
    current_domains = self.conn.execute("""  
        SELECT DISTINCT domain   
        FROM serp_snapshots   
        WHERE query = ?   
          AND snapshot_date = ?  
          AND rank <= 10  
    """, [query, snapshot_date]).fetchall()  
    current_domains_set = {row[0] for row in current_domains}  

    # Get previous top 10 domains  
    prev_domains = self.conn.execute("""  
        SELECT DISTINCT domain   
        FROM serp_snapshots   
        WHERE query = ?   
          AND snapshot_date = ?  
          AND rank <= 10  
    """, [query, prev_date]).fetchall()  
    prev_domains_set = {row[0] for row in prev_domains}  

    # Count new domains entering top 10  
    new_domains = current_domains_set - prev_domains_set  
    new_domains_count = len(new_domains)  

    # Calculate average rank improvement for existing domains  
    rank_changes = self.conn.execute("""  
        WITH current_ranks AS (  
            SELECT domain, rank  
            FROM serp_snapshots  
            WHERE query = ? AND snapshot_date = ?   
              AND rank <= 10  
        ),  
        prev_ranks AS (  
            SELECT domain, rank  
            FROM serp_snapshots  
            WHERE query = ? AND snapshot_date = ?  
              AND rank <= 10  
        )  
        SELECT   
            c.domain,  
            c.rank as current_rank,  
            p.rank as prev_rank,  
            (p.rank - c.rank) as rank_improvement  
        FROM current_ranks c  
        JOIN prev_ranks p ON c.domain = p.domain  
    """, [query, snapshot_date, query, prev_date]).fetchall()  

    if rank_changes:  
        avg_rank_improvement = sum(row[3] for row in rank_changes) / len(rank_changes)  
    else:  
        avg_rank_improvement = 0.0  

    # Calculate reshuffle frequency (how many domains changed position)  
    reshuffle_count = len(rank_changes)  
    reshuffle_frequency = reshuffle_count / max(len(current_domains_set), 1)  

    # Normalize to 0-100 score  
    # I'm calculating a final score from 3 weighted sub-scores:  
    # - New domains: 0-10 domains = 0-40 points  
    # - Rank improvement: -10 to +10 = 0-30 points (normalized)  
    # - Reshuffle frequency: 0-1 = 0-30 points  

    new_domains_score = min(new_domains_count * 4, 40)  # Max 40 points  
    rank_improvement_score = min(max((avg_rank_improvement + 10) / 20 * 30, 0), 30)  # Max 30 points  
    reshuffle_score = reshuffle_frequency * 30  # Max 30 points  

    interest_score = new_domains_score + rank_improvement_score + reshuffle_score  

    # Store interest score  
    self.conn.execute("""  
        INSERT OR REPLACE INTO interest_scores   
        (query, snapshot_date, interest_score, new_domains_count, avg_rank_improvement, reshuffle_frequency)  
        VALUES (?, ?, ?, ?, ?, ?)  
    """, [query, snapshot_date, interest_score, new_domains_count, avg_rank_improvement, reshuffle_frequency])

This runs automatically every time a new snapshot is inserted. The score gets stored in a separate interest_scores table for easy querying.

Module 3: Analytical Queries

The nerdiest of our logic lives in analytics.py. This module opens DuckDB in read-only mode and exposes focused queries.

Read the full code for analytics.py here: https://github.com/sixthextinction/duckdb-google-trends-basic/blob/main/src/analytics.py

A good analytical query to demonstrate right now would be the one for rank volatility:

def rank_volatility(self, query: str, days: int = 30) -> Dict[str, Any]:    
    """Calculate rank volatility for URLs over time"""    
    cutoff_date = datetime.now().date() - timedelta(days=days)    

    result = self.conn.execute("""    
        WITH rank_changes AS (    
            SELECT     
                url,    
                domain,    
                rank,    
                snapshot_date,    
                LAG(rank) OVER (PARTITION BY url ORDER BY snapshot_date) as prev_rank    
            FROM serp_snapshots    
            WHERE query = ? AND snapshot_date >= ?    
            ORDER BY url, snapshot_date    
        ),    
        volatility AS (    
            SELECT     
                url,    
                domain,    
                COUNT(*) as snapshot_count,    
                AVG(rank) as avg_rank,    
                MIN(rank) as best_rank,    
                MAX(rank) as worst_rank,    
                STDDEV(rank) as rank_stddev,    
                COUNT(CASE WHEN prev_rank IS NOT NULL AND rank != prev_rank THEN 1 END) as rank_changes    
            FROM rank_changes    
            GROUP BY url, domain    
        )    
        SELECT     
            url,    
            domain,    
            snapshot_count,    
            ROUND(avg_rank, 2) as avg_rank,    
            best_rank,    
            worst_rank,    
            ROUND(rank_stddev, 2) as rank_stddev,    
            rank_changes,    
            ROUND(CAST(rank_changes AS DOUBLE) / NULLIF(snapshot_count - 1, 0) * 100, 1) as volatility_pct    
        FROM volatility    
        WHERE snapshot_count > 1    
        ORDER BY rank_stddev DESC, avg_rank ASC    
        LIMIT 50    
    """, [query, cutoff_date]).df()    

    return {'query': query, 'days': days, 'results': result}

This uses window functions (LAG) and aggregations (STDDEV) to surface URLs that move around the most. These queries would normally require a data warehouse — here they’re just SQL running locally.

Run this with:

python main.py volatility --query "python" --days 30

This, for example, will analyze the last 30 days of snapshots for the query string “python”, calculating average rank, best/worst rank, standard deviation, and change frequency — and display the top 50 (as default) most volatile URLs.

Example output:

=== Rank Volatility for 'python' (last 30 days) ===  

Top 10 most volatile URLs:  

| url | domain | snapshot_count | avg_rank | best_rank | worst_rank | rank_stddev | rank_changes | volatility_pct |  
| --- | --- | --- | --- | --- | --- | --- | --- | --- |  
| https://www.python.org/ | python.org | 30 | 1.5 | 1 | 3 | 0.67 | 15 | 51.7 |  
| https://www.w3schools.com/python/ | w3schools.com | 30 | 2.3 | 1 | 5 | 1.12 | 18 | 62.1 |  
| https://en.wikipedia.org/wiki/Python_(programming_language) | wikipedia.org | 28 | 4.1 | 2 | 8 | 1.89 | 12 | 44.4 |  
| https://www.codecademy.com/catalog/language/python | codecademy.com | 25 | 5.2 | 3 | 10 | 2.15 | 10 | 41.7 |

Another query that is very useful is finding new entrants:

def new_entrants(self, query: str, days: int = 7):    
    """Find URLs that appeared for the first time recently"""    
    cutoff_date = datetime.now().date() - timedelta(days=days)    

    result = self.conn.execute("""    
        WITH first_appearance AS (    
            SELECT     
                url,    
                domain,    
                MIN(snapshot_date) as first_seen    
            FROM serp_snapshots    
            WHERE query = ?    
            GROUP BY url, domain    
        ),    
        recent_entrants AS (    
            SELECT     
                fa.url,    
                fa.domain,    
                fa.first_seen,    
                s.rank as first_rank,    
                s.title,    
                s.snippet    
            FROM first_appearance fa    
            JOIN serp_snapshots s     
                ON fa.url = s.url     
                AND fa.first_seen = s.snapshot_date    
                AND s.query = ?    
            WHERE fa.first_seen >= ?    
        )    
        SELECT     
            url,    
            domain,    
            first_seen,    
            first_rank,    
            title,    
            snippet    
        FROM recent_entrants    
        ORDER BY first_seen DESC, first_rank ASC    
        LIMIT 50    
    """, [query, query, cutoff_date]).df()    

    return {'query': query, 'days': days, 'results': result}

This finds URLs whose first appearance falls within the last N days — perfect for spotting new competitors or fresh content entering the rankings.

Run this with:

python main.py new-entrants --query "python" --days  7

Example output:

=== New Entrants for 'python' (last 7 days) ===  

Found 3 new URLs:  

| url | domain | first_seen | first_rank | title | snippet |  
| --- | --- | --- | --- | --- | --- |  
| https://realpython.com/ | realpython.com | 2026-02-04 | 7 | Real Python - Python Tutorials | Learn Python programming with Real Python's comprehensive tutorials and courses... |  
| https://www.pythonforbeginners.com/ | pythonforbeginners.com | 2026-02-05 | 9 | Python For Beginners | A comprehensive guide to learning Python programming from scratch... |  
| https://docs.python-guide.org/ | docs.python-guide.org | 2026-02-06 | 8 | The Hitchhiker's Guide to Python | Best practices and recommendations for Python development... |

I’m not going to go over every module, but it’s all in the code. Find all queries + their expected output in the project README.md.

Module 4: The Snapshot Fetcher

Finally, scraper.py (I’m so sorry — I really could have named this better 😅) connects ingestion and storage.

Read the full code for scraper.py here: https://github.com/sixthextinction/duckdb-google-trends-basic/blob/main/src/scraper.py

import time    
from datetime import datetime    
from typing import List, Optional    

from serp_client import BrightDataClient    
from duckdb_manager import DuckDBManager    


def fetch_snapshots(keywords: List[str], num_results: int = 10, delay: float = 1.0):    
    """    
    Fetch SERP snapshots for keywords and store in DuckDB    

    Args:    
        keywords: List of search keywords    
        num_results: Number of results per keyword    
        delay: Delay between API calls (seconds)    
    """    
    client = BrightDataClient()    

    with DuckDBManager() as db:    
        print(f"Fetching snapshots for {len(keywords)} keywords...")    

        for idx, keyword in enumerate(keywords):    
            try:    
                # Fetch SERP results    
                serp_data = client.search(keyword, num_results=num_results)    

                # Extract organic results    
                organic_results = []    
                if isinstance(serp_data, dict) and 'organic' in serp_data:    
                    organic_results = serp_data['organic']    

                if organic_results:    
                    # Insert snapshot    
                    db.insert_snapshot(organic_results, keyword)    
                    print(f"[{idx+1}/{len(keywords)}] '{keyword}': {len(organic_results)} results")    
                else:    
                    print(f"[{idx+1}/{len(keywords)}] '{keyword}': No results found")    

                # Rate limiting    
                if idx < len(keywords) - 1:    
                    time.sleep(delay)    

            except Exception as e:    
                print(f"Error fetching '{keyword}': {e}")    
                continue    

        total = db.get_snapshot_count()    
        print(f"\nTotal snapshots in database: {total}")

This is just simple orchestration logic, again. It iterates over keywords, fetches results, and inserts snapshots. Rate limiting and error handling live at the edges. I’ve kept the core logic deliberately simple.

That’s everything! Remember, main.py brings all of these together.

Real World Use Cases

Now that you understand how it works, here’s some cool things you can actually do with this tool.

1. Detect Google Algorithm Updates Before They’re Announced

When tracking multiple keywords in the same niche, sudden volatility spikes across all of them indicate an algorithm change.

python main.py volatility --query "react" --days 7  
python main.py volatility --query "vue" --days 7  
python main.py volatility --query "angular" --days 7

If all three show high rank_stddev and volatility_pct, Google likely pushed an update.

SEO folks pay $200/month for SEMrush Sensor just to get this signal. You’re building it for the cost of SERP API calls.

2. Spy on Competitor SEO Tactics

Track title and snippet changes to see what competitors are A/B testing:

python main.py changes --query "nextjs tutorial" --days 30

Example output:

| url                              | prev_title                    | new_title                                               |  
|----------------------------------|-------------------------------|---------------------------------------------------------|  
| https://nextjs.org/docs          | Next.js Documentation         | Next.js Docs | Next.js                                  |  
| https://nextjs.org/learn         | Learn Next.js                 | Learn Next.js | Next.js by Vercel - The React Framework |

Let’s say a site changed their title from a generic page description to something more specific. If their rank improved after the change, that’s a signal the new title performs better — steal that pattern!

3. Find Content Gaps in Real-Time

See which sites are entering top 10 and what format they’re using:

python main.py new-entrants --query "react hooks tutorial" --days 7

Example output:

| domain              | first_seen | first_rank | title                                    |  
|---------------------|------------|------------|------------------------------------------|  
| react-tutorial.dev  | 2026-02-05 | 7          | React Hooks Interactive Tutorial         |  
| codesandbox.io      | 2026-02-06 | 9          | Learn React Hooks - Live Coding Examples |

Let’s say there are two new entrants in the SERP for the query “react hooks tutorial”, and both new entrants have “Interactive” or “Live” in their titles. That means Google is currently rewarding interactive content for this query. Adjust your content strategy accordingly.

4. Validate Content Ideas Before Creation

This one’s super simple to understand. High volatility = easier for you to rank. Low volatility = established players dominate.

python main.py volatility --query "python tutorial" --days 30  
python main.py volatility --query "rust async tutorial" --days 30

Let’s say the query “python tutorial” shows rank_stddev: 0.3 (very stable) and “rust async tutorial” shows rank_stddev: 2.1 (chaotic), focus on the Rust content! The Python keyword is locked down by W3Schools and Real Python — you won’t break in easily.

5. Track Your Own Product’s SERP Performance

Monitor how your product ranks for target keywords:

python main.py fetch --keywords "whatever your product is or does"

Then check if you’re entering top 10:

python main.py new-entrants --query "whatever your product is or does" --days 7

If your product URL appears, congrats — you just entered the top 10. If competitors are dropping out (volatility shows their ranks declining), you’re winning.

What I Learned Building This

DuckDB is a total cheat code for embedded analytics. I expected to need PostgreSQL (or ClickHouse, ugh.) for time-series queries over SERP data. Without fiddling with any config, calculating rank volatility across 30 days of snapshots for 50 URLs ran in ~20ms for me. The database file was <5MB for weeks of data.
Bright Data’s SERP API is very reliable. I tried other SERP APIs before settling on Bright Data, primarily because of the consistent JSON output for Google and Bing, and support for DuckDuckGo, Yandex, etc. This experiment cost me pennies — but make sure you check their pricing so you don’t get burnt by costs you shouldn’t be incurring.
The Interest Score formula needs tuning, possibly. The 40/30/30 weighting (new domains / rank improvement / domain overlap ratio) was only my first guess. It works reasonably well, but it’s not perfect. At the very least, I should weight new domains more heavily for breaking news queries, and reduce domain overlap ratio impact for stable niches (Because, for example, Wikipedia will always be #1 for “Python programming language”)

Try It Yourself

Again, the full code is on GitHub: https://github.com/sixthextinction/duckdb-google-trends-basic/

Quick start:

git clone https://github.com/sixthextinction/duckdb-google-trends-basic.git  
# or...  
gh repo clone sixthextinction/duckdb-google-trends-basic  
# then...  
cd duckdb-google-trends-basic    
pip install -r requirements.txt    

# Set environment variables    
export BRIGHT_DATA_API_KEY="your_key"    
export BRIGHT_DATA_ZONE="your_zone"    
export BRIGHT_DATA_COUNTRY="us"  # optional, for geo-targeted results    

# Or use a .env file instead (python-dotenv is included)    

# Test with sample data (no API key needed)    
python seed_data.py    
python main.py scores --query "nextjs" --days 7    

# Or fetch real data    
python main.py fetch --keywords "react" "vue" "svelte"

I’ve included a seed script that creates 7 days of synthetic data so you can test immediately without waiting. Otherwise, set up a daily cron job to fetch snapshots automatically, and within a week you’ll have real trend data.

Thanks for reading!

If you did something cool with this tool, I’d love to see it. Reach out on LinkedIn, or put it in the comments below.

Turning Google Search into a Kafka Event Stream for Many Consumers

Prithwish Nath — Tue, 03 Feb 2026 08:43:13 +0000

TL;DR: Google Rankings are a bad abstraction for how Google An event-driven approach to monitoring SERP changes — tracking features, entries, and exits instead of noisy rank movements.

The SERP — Google’s ‘search engine results page’ is what humans actually see when they type in a search and hit enter. It has ads, featured snippets/AI overviews, carousels, knowledge panels long before organic search results. In fact, in 2026, raw organic results increasingly feel like a fallback rather than the main event. You can sit at position three for months and still lose traffic overnight because Google introduced a new ad or feature above you.

That’s the core problem with naïve rank tracking. It assumes the environment is static and your position within it is the only variable. The truth is those changes Google makes to the results page often matter more (for market intelligence etc.) than whether you moved up/down a position.

If you’re monitoring Google for your brand and care about traffic risk, competitive pressure, or acquisition costs these are just some things that can happen:

Google adds ads → organic CTR drops
Google launches a featured snippet → one domain captures disproportionate attention
A new competitor enters the page → your share of attention changes even if your rank doesn’t

None of this is well represented by a line chart of rankings. I couldn’t find many blogs that discuss this specifically without slipping into generic SEO optimization garbage, so I figured I’d write the one I wish I’d found. 🤷‍♂️ Let’s get down to it.

Why Kafka?

As long as you’re answering one question for one person, you don’t need Kafka. A cron job + Postgres work just fine.

The moment that stops being enough is when you want any combination of: history, alerts, multiple downstream consumers, or independent evolution of logic over time. Anything that takes this beyond a simple one-off. That’s usually where people keep bolting features onto a single script until it’s doing scraping, diffing, alerting, analytics, and reporting all at once.

Both paths lead to systems that are hard to reason about and harder to change. How do we avoid this?

Think about the basic unit of information you care about, and design around that. In this case, our “unit” is change, not snapshots. Because we want to answer “what happened over this period of X?”

So why use Kafka? It’s just the most boring (battle tested, well documented, not flashy) tech I could think of that accomplishes it. These are the benefits:

Event replay. If you build a new consumer next month, you can replay the last 7 days of SERP changes (or however long you configure retention). Your historical data isn’t trapped in database rows — it’s a stream you can re-process.
Independent consumers. Each consumer has its own offset. Your alerting system can crash and restart without affecting your analytics pipeline. They process events at their own pace.
Ordering guarantees. Events for the same keyword go to the same partition (we use {keyword}:{geo} as the key). You’ll never see a “domain entered” event before the “featured snippet appeared” event that caused it.
Backpressure handling. If your volatility analyzer is slow, Kafka doesn’t care. Events queue up and the producer keeps producing.

Apache Kafka

Is setting up Kafka more complex than a database? Yep. Is it worth it? At this level, you can’t really afford to go with anything else.

What We’re Building

The architecture is straightforward:

We continuously monitor a set of keywords via a SERP API (I’m using Bright Data), detect when the structure changes (ads appear, featured snippets shift, domains enter/exit), emit those changes (and only the changes — we have to maintain some form of state to spot the delta) as events into Kafka, and fan them out to multiple downstream consumers who can then log or store the data as they like.

Prerequisites

Before we start building, you need Python dependencies and Kafka infrastructure running.

> pip install requests python-dotenv kafka-python

Next, Kafka requires Zookeeper and a broker. Instead of installing them manually, we use Docker Compose with this YAML file.

💡 In newer Kafka versions (≥3.3), KRaft (Kafka without ZooKeeper) replaces the old coordination model.

I’m sticking with what I’m familiar with here, but if you’re setting up a modern cluster — or using an AI agent to generate your setup — it may use KRaft instead of ZooKeeper by default. Both approaches work fine for our use case, and we treat ZooKeeper as largely set-and-forget anyway.

version: '3.8'  

services:  
  zookeeper:  
    image: confluentinc/cp-zookeeper:7.5.0  
    hostname: zookeeper  
    container_name: zookeeper  
    ports:  
      - "2181:2181"  
    environment:  
      ZOOKEEPER_CLIENT_PORT: 2181  
      ZOOKEEPER_TICK_TIME: 2000  

  kafka:  
    image: confluentinc/cp-kafka:7.5.0  
    hostname: kafka  
    container_name: kafka  
    depends_on:  
      - zookeeper  
    ports:  
      - "9092:9092"  
      - "9101:9101"  
    environment:  
      KAFKA_BROKER_ID: 1  
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'  
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT  
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://kafka:29092  
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1  
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1  
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1  
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0  
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'  
      # retention: 7 days  
      KAFKA_LOG_RETENTION_HOURS: 168  
      KAFKA_LOG_RETENTION_BYTES: 1073741824

Then, run it with

> docker-compose up -d

This starts:

Zookeeper (at port 2181) — coordinates Kafka cluster
Kafka Broker (at port 9092) — stores and distributes events

Finally, for the SERP API, get your credentials here

…and put them in a .env file in the project root like so.

BRIGHT_DATA_API_KEY=your_api_key_here  
BRIGHT_DATA_ZONE=your_zone_name_here  
BRIGHT_DATA_COUNTRY=us

That’s it. You’re ready to start building.

Step 1: Fetching SERP Results

So we can’t scrape Google directly. Google blocks automated requests, and residential/datacenter proxies don’t work for search. The solution is to use a SERP API. These return structured JSON of Google’s results page, including organic results, ads, featured snippets, video carousels, and more.

For reusability, we’ll first build a client that wraps the SERP API:


import os  
import requests  
from typing import Dict, Any, Optional  
from dotenv import load_dotenv  

load_dotenv()  


class BrightDataClient:  
    """  
    Reusable client for Bright Data SERP API.  
    """  

    def __init__(        self,  
        api_key: Optional[str] = None,  
        zone: Optional[str] = None,  
        country: Optional[str] = None    ):  
        # load from environment variables if not provided  
        env_api_key = os.getenv("BRIGHT_DATA_API_KEY")  
        env_zone = os.getenv("BRIGHT_DATA_ZONE")  
        env_country = os.getenv("BRIGHT_DATA_COUNTRY")  

        # use provided values or fall back to environment variables  
        self.api_key = api_key or env_api_key  
        self.zone = zone or env_zone  
        self.country = country or env_country  
        self.api_endpoint = "https://api.brightdata.com/request"  

        if not self.api_key:  
            raise ValueError(  
                "BRIGHT_DATA_API_KEY must be provided via constructor or environment variable. "  
                "Get your API key from: https://brightdata.com/cp/setting/users"  
            )  

        if not self.zone:  
            raise ValueError(  
                "BRIGHT_DATA_ZONE must be provided via constructor or environment variable. "  
                "Manage zones at: https://brightdata.com/cp/zones"  
            )  

        # setup session with API authentication  
        self.session = requests.Session()  
        self.session.headers.update({  
            'Content-Type': 'application/json',  
            'Authorization': f'Bearer {self.api_key}'  
        })  

    def search(        self,  
        query: str,  
        num_results: int = 10,  
        language: Optional[str] = None,  
        country: Optional[str] = None    ) ->` Dict[str, Any]:  
        """  
        Executes a google search.  
        Args:  
            query: Search query string  
            num_results: Number of results to return (default: 10)  
            language: Language code (e.g., 'en', 'es', 'fr')  
            country: Country code (e.g., 'us', 'uk', 'ca') - overrides instance default  

        Returns:  
            Dictionary containing search results in JSON format  
        """  
        # build Google search URL  
        search_url = (  
            f"https://www.google.com/search"  
            f"?q={requests.utils.quote(query)}"  
            f"&num={num_results}"  
            f"&brd_json=1"  
        )  

        if language:  
            search_url += f"&hl={language}&lr=lang_{language}"  

        # use method parameter country or instance default  
        target_country = country or self.country  

        # prepare request payload for SERP API  
        payload = {  
            'zone': self.zone,  
            'url': search_url,  
            'format': 'json'  
        }  

        if target_country:  
            payload['country'] = target_country  

        try:  
            # use POST request to SERP API endpoint  
            response = self.session.post(  
                self.api_endpoint,  
                json=payload,  
                timeout=30  
            )  
            response.raise_for_status()  

            # this SERP API returns JSON directly when format='json' so just return it  
            return response.json()  

        except requests.exceptions.HTTPError as e:  
            error_msg = f"Search request failed with HTTP {e.response.status_code}"  
            if e.response.text:  
                error_msg += f": {e.response.text[:200]}"  
            raise RuntimeError(error_msg) from e  
        except requests.exceptions.RequestException as e:  
            raise RuntimeError(f"Search request failed: {e}") from e

As a preview, here’s how we use it later.

from src.bright_data import BrightDataClient  
# initialize client (reads credentials from .env file)  
client = BrightDataClient()  
# fetch SERP for a keyword  
raw_response = client.search(  
  query="ai crm",  
  num_results=10,  
  country="us"  
)  
# raw_response now contains the full SERP structure

There’s just one problem — the raw Bright Data response you get back from this API is way too verbose:

{  
  "general": {  
    "search_engine": "google",  
    "query": "ai crm",  
    "language": "en",  
    "mobile": false,  
    "basic_view": false,  
    "search_type": "text",  
    "page_title": "ai crm - Google Search",  
    "timestamp": "2026–01–07T11:18:35.216Z"  
  },  
  "input": {  
    "original_url": "https://www.google.com/search?q=ai+crm&brd_json=1&gl=us",  
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",  
    "request_id": "some_request_id"  
  },  
  "navigation": [ /* 6 navigation links */ ],  
  "organic": [  
    {  
      "link": "https://attio.com/",  
      "source": "Attio",  
      "display_link": "https://attio.com",  
      "title": "Attio: The next gen of CRM",  
      "description": "Execute your revenue strategy with precision…",  
      "rank": 1,  
      "global_rank": 1,  
      "extensions": [ /* site links, dates, etc */ ]  
    },  
    // ...8 more results with full metadata  
  ],  
  "images": [ /* image results */ ],  
  "top_ads": [{}, {}],  
  "bottom_ads": [{}, {}, {}],  
  "pagination": { /* pagination links */ },  
  "related": [ /* 8 related searches */ ]  
}

That’s 350+ lines of JSON for a single search. Most of it is noise for change detection. For change detection, we only care about:

Which domains are present (not full URLs, titles, descriptions)
Which features exist (ads, featured snippets, video carousels) — boolean flags
When this snapshot was taken (timestamp)
What keyword/geo this is for (context)

Everything else — navigation links, user agents, request IDs, pagination, related searches, full result metadata — is irrelevant for detecting changes.

So let’s normalize it into a much leaner format for consumption.

Step 2 : Why Normalization Matters

So we transform the verbose raw response into a compact snapshot:

"""  
/src/normalizer.py   

SERP data normalization  
converts raw Bright Data response to structured snapshot format  
"""  

from datetime import datetime  
from urllib.parse import urlparse  
from typing import Dict, Any  


def extract_domain(url: str) -> str:  
    """extract domain from URL"""  
    try:  
        parsed = urlparse(url)  
        domain = parsed.netloc or parsed.path  
        # remove www. prefix  
        if domain.startswith('www.'):  
            domain = domain[4:]  
        return domain  
    except Exception:  
        return url  


def normalize_serp_data(raw_response: Dict[str, Any], keyword: str, geo: str = "US") -> Dict[str, Any]:  
    """  
    normalize SERP response into structured format  

    Args:  
        raw_response: raw JSON response from Bright Data SERP API  
        keyword: search keyword  
        geo: geographic location code (default: "US")  

    Returns:  
        normalized dictionary with structured SERP data  
    """  

    # extract organic domains  
    organic_domains = []  
    if raw_response.get('organic') and isinstance(raw_response['organic'], list):  
        for result in raw_response['organic']:  
            link = result.get('link') or result.get('url', '')  
            if link:  
                domain = extract_domain(link)  
                if domain:  
                    organic_domains.append(domain)  

    # check for ads (top_ads or bottom_ads)  
    top_ads = raw_response.get('top_ads', [])  
    bottom_ads = raw_response.get('bottom_ads', [])  
    has_ads = bool(  
        (top_ads and len([a for a in top_ads if a]) > 0) or  
        (bottom_ads and len([a for a in bottom_ads if a]) > 0) or  
        (raw_response.get('ads') and len(raw_response.get('ads', [])) > 0)  
    )  
    ads_count = len([a for a in top_ads if a]) + len([a for a in bottom_ads if a])  

    # check for featured snippet (knowledge panel)  
    has_featured_snippet = bool(raw_response.get('knowledge'))  

    # check for video carousel  
    has_video_carousel = bool(raw_response.get('video') and len(raw_response.get('video', [])) > 0)  

    # check for people also ask  
    has_people_also_ask = bool(raw_response.get('people_also_ask') and len(raw_response.get('people_also_ask', [])) > 0)  

    # extract featured snippet owner if available  
    featured_snippet_owner = None  
    if has_featured_snippet:  
        knowledge = raw_response.get('knowledge', {})  
        # try to extract owner from knowledge panel  
        if knowledge.get('url'):  
            featured_snippet_owner = extract_domain(knowledge['url'])  
        elif knowledge.get('source'):  
            featured_snippet_owner = extract_domain(knowledge['source'])  

    # build normalized structure  
    normalized = {  
        "keyword": keyword,  
        "geo": geo,  
        "features": {  
            "ads": has_ads,  
            "ads_count": ads_count if has_ads else 0,  
            "featured_snippet": has_featured_snippet,  
            "featured_snippet_owner": featured_snippet_owner,  
            "video_carousel": has_video_carousel,  
            "people_also_ask": has_people_also_ask  
        },  
        "organic_domains": organic_domains,  
        "timestamp": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")  
    }  

    return normalized

Here, we extract just the domain names, not full URLs (i.e. from “https://www.salesforce.com/crm/” to “salesforce.com”. Why? Because:

URLs change (query params, paths) but domains are stable identifiers
We care about which sites are ranking, not specific pages
Domain-level tracking is more resilient to URL structure changes

We also convert complex nested structures into simple booleans:

# Raw:   
{  
  "top_ads": [{...}, {...}],   
  "bottom_ads": []  
}  
# Normalized:   
{  
  "ads": true,   
  "ads_count": 2  
}

This makes some comparisons we’ll perform later, trivial:

if current['features']['ads'] and not previous['features']['ads']:  
  # Ads appeared! Emit a signal.

Finally, when a featured snippet exists, we extract which domain owns it:

if has_featured_snippet:  
  knowledge = raw_response.get('knowledge', {})  
if knowledge.get('url'):  
  featured_snippet_owner = extract_domain(knowledge['url'])

This lets us track when featured snippet ownership changes — a critical competitive signal.

Before vs After

Raw response: 350+ lines, ~15KB. Normalized snapshot: 22 lines, ~500 bytes

This way, we store 30x less data per snapshot, and perform comparisons way faster because we’re comparing arrays of domains instead of deep object structures.

Step 3: Change Detection — Turning Snapshots into Events

At this point we have normalized SERP snapshots. But snapshots alone aren’t very useful for Kafka. We’ve talked about this before — Kafka wants events, not states.

So the job of Step 3 is simple:

Compare the current snapshot with the previous one and emit only what changed i.e. the delta.

"""  
/src/change_detector.py  
change detection logic for SERP snapshots  
compares current vs previous state and emits change events  
"""  

from typing import Dict, Any, List, Optional  
from datetime import datetime  


class ChangeDetector:  
    """  
    detects changes between SERP snapshots  
    generates change events for Kafka  
    """  

    def detect_changes(        self,  
        current: Dict[str, Any],  
        previous: Optional[Dict[str, Any]]    ) -> List[Dict[str, Any]]:  
        """  
        compare current and previous snapshots, return list of change events  

        Args:  
            current: current normalized SERP snapshot  
            previous: previous normalized SERP snapshot (None if first run)  

        Returns:  
            list of change event dictionaries  
        """  
        events = []  

        if previous is None:  
            # first run - no changes to detect  
            return events  

        # detect feature changes  
        events.extend(self._detect_feature_changes(current, previous))  

        # detect organic domain changes  
        events.extend(self._detect_organic_changes(current, previous))  

        return events  

    def _detect_feature_changes(        self,  
        current: Dict[str, Any],  
        previous: Dict[str, Any]    ) -> List[Dict[str, Any]]:  
        """detect changes in SERP features (ads, featured snippet, etc.)"""  
        events = []  

        current_features = current.get('features', {})  
        previous_features = previous.get('features', {})  

        feature_types = ['ads', 'featured_snippet', 'video_carousel', 'people_also_ask']  

        for feature in feature_types:  
            current_value = current_features.get(feature, False)  
            previous_value = previous_features.get(feature, False)  

            # feature appeared  
            if current_value and not previous_value:  
                event = {  
                    "event_type": "serp_feature_added",  
                    "keyword": current['keyword'],  
                    "geo": current['geo'],  
                    "feature": feature,  
                    "timestamp": current['timestamp']  
                }  

                # add feature-specific metadata  
                if feature == "ads":  
                    # count ads if available  
                    event["count"] = current.get('features', {}).get('ads_count', 1)  
                elif feature == "featured_snippet":  
                    # extract owner domain if available  
                    owner = current.get('features', {}).get('featured_snippet_owner')  
                    if owner:  
                        event["owner_domain"] = owner  

                events.append(event)  

            # feature disappeared  
            elif not current_value and previous_value:  
                event = {  
                    "event_type": "serp_feature_removed",  
                    "keyword": current['keyword'],  
                    "geo": current['geo'],  
                    "feature": feature,  
                    "timestamp": current['timestamp']  
                }  
                events.append(event)  

        return events  

    def _detect_organic_changes(        self,  
        current: Dict[str, Any],  
        previous: Dict[str, Any]    ) -> List[Dict[str, Any]]:  
        """detect changes in organic domain rankings"""  
        events = []  

        current_domains = set(current.get('organic_domains', []))  
        previous_domains = set(previous.get('organic_domains', []))  

        # domains that exited (were in previous, not in current)  
        exited = previous_domains - current_domains  
        for domain in exited:  
            # find previous position  
            previous_position = self._get_domain_position(domain, previous)  
            event = {  
                "event_type": "organic_exit",  
                "keyword": current['keyword'],  
                "geo": current['geo'],  
                "domain": domain,  
                "previous_position": previous_position,  
                "timestamp": current['timestamp']  
            }  
            events.append(event)  

        # domains that entered (in current, not in previous)  
        entered = current_domains - previous_domains  
        for domain in entered:  
            # find current position  
            current_position = self._get_domain_position(domain, current)  
            event = {  
                "event_type": "organic_entry",  
                "keyword": current['keyword'],  
                "geo": current['geo'],  
                "domain": domain,  
                "current_position": current_position,  
                "timestamp": current['timestamp']  
            }  
            events.append(event)  

        return events  


    def _get_domain_position(        self,   
        domain: str,   
        snapshot: Dict[str, Any]    ) -> Optional[int]:  
        """get position of domain in organic results (1-indexed)"""  
        domains = snapshot.get('organic_domains', [])  
        try:  
            return domains.index(domain) + 1  
        except ValueError:  
            return None

This detector compares two normalized snapshots and emits only the delta. Because we normalized SERP features into simple booleans in Step 2, detecting changes becomes trivial.

# this avoids deep JSON comparisons + false positives from minor SERP variations  
if now and not before:  
    # feature appeared

Also, for organic results, we intentionally ignore position changes. Rank reshuffles inside the top 10 happen constantly and aren’t very actionable. A domain entering or disappearing from the SERP is what’s actually interesting. Because organic results were normalized into a flat list of domains, this becomes a set comparison: previous — current = domains that exitedand current — previous = domains that entered. This gives us O(1) lookups and O(n) comparisons.

Each run produces zero or more small events like:

{  
  "event_type": "serp_feature_added",  
  "keyword": "ai crm",  
  "geo": "US",  
  "feature": "ads",  
  "count": 3,  
  "timestamp": "2026-01-07T12:00:00Z"  
},  
{  
  "event_type": "serp_feature_added",  
  "keyword": "ai crm",  
  "geo": "US",  
  "feature": "featured_snippet",  
  "owner_domain": "hubspot.com",  
  "timestamp": "2026-01-07T12:00:00Z"  
}

Instead of storing “what the SERP looks like now”, now we’re storing:

what happened
when it happened
for which keyword + geo it happened.
When relevant, we also attach extra metadata (ad count, featured snippet owner) so downstream consumers don’t need to re-hydrate context.

This is the exact shape Kafka wants. At this point, SERP data has stopped being scraped snapshots for us, and is behaving like a true event stream.

Before we start the pipeline, we should cover state management, briefly.

Step 4: A Simple State Manager

We’ll use simple file-based storage:

"""  
/src/state_manager.py  
state management for SERP snapshots  
stores previous state to enable change detection  
"""  
import os  
import json  
from typing import Dict, Any, Optional  
from pathlib import Path  
class StateManager:  
    """  
    manages SERP snapshot state storage  
    stores previous snapshots to compare against current ones  
    """  

    def __init__(self, state_dir: str = "state"):  
        self.state_dir = Path(state_dir)  
        self.state_dir.mkdir(exist_ok=True)  

    def _get_state_file(self, keyword: str, geo: str) -> Path:  
        """get state file path for keyword+geo combination"""  
        safe_keyword = keyword.replace(" ", "_").lower()  
        safe_geo = geo.lower()  
        filename = f"{safe_keyword}_{safe_geo}.json"  
        return self.state_dir / filename  

    def load_previous_state(        self,   
        keyword: str,   
        geo: str    ) -> Optional[Dict[str, Any]]:  
        """  
        load previous SERP snapshot for keyword+geo  

        Returns:  
            previous snapshot dict or None if no previous state exists  
        """  
        state_file = self._get_state_file(keyword, geo)  

        if not state_file.exists():  
            return None  

        try:  
            with open(state_file, 'r', encoding='utf-8') as f:  
                return json.load(f)  
        except Exception as e:  
            print(f"Warning: failed to load state from {state_file}: {e}")  
            return None  

    def save_state(        self,   
        keyword: str,   
        geo: str,   
        snapshot: Dict[str, Any]    ) -> None:  
        """  
        save current SERP snapshot as new state  

        Args:  
            keyword: search keyword  
            geo: geographic location code  
            snapshot: normalized SERP snapshot to save  
        """  
        state_file = self._get_state_file(keyword, geo)  

        try:  
            with open(state_file, 'w', encoding='utf-8') as f:  
                json.dump(snapshot, f, indent=2, ensure_ascii=False)  
        except Exception as e:  
            print(f"Error: failed to save state to {state_file}: {e}")  
            raise

So we’ll store our state files like this:

./state/  
  ├── ai_crm_us.json  
  ├── best_crm_us.json  
  └── ai_crm_uk.json

Each state file contains the last normalized snapshot for that keyword + geo combination, as you’ll see when we bring it all together.

Step 4: The Producer Pipeline — Tying It All Together

At this point we have all the individual pieces:

SERP fetching
Snapshot normalization
Change detection
Kafka event emission

Step 4 is where they come together into a single producer pipeline.

This producer fetches the latest SERP → normalizes it → compares it to the previous snapshot → emits only the detected changes (the delta) to Kafka → persists state for the next run.

"""  
/src/producer.py  

Kafka producer for SERP change events  
fetches SERP data, detects changes, and emits events to Kafka  
"""  

import time  
import json  
from typing import Dict, Any, List  
from kafka import KafkaProducer  
from kafka.errors import KafkaError  

from bright_data import BrightDataClient  
from normalizer import normalize_serp_data  
from state_manager import StateManager  
from change_detector import ChangeDetector  


class SERPProducer:  
    """  
    For a Kafka producer that monitors SERP changes and emits events to Kafka  
    """  

    def __init__(        self,  
        kafka_brokers: str = "localhost:9092",  
        state_dir: str = "state"    ):  
        self.producer = KafkaProducer(  
            bootstrap_servers=kafka_brokers,  
            # convert because KafkaProducer expects callable serializers.  
            value_serializer=lambda v: json.dumps(v).encode('utf-8'),  
            key_serializer=lambda k: k.encode('utf-8') if k else None  
        )  
        self.bright_data = BrightDataClient()  
        self.state_manager = StateManager(state_dir)  
        self.change_detector = ChangeDetector()  

    def process_keyword(        self,  
        keyword: str,  
        geo: str = "US",  
        num_results: int = 10    ) -> List[Dict[str, Any]]:  
        """  
        process a single keyword: fetch, compare, emit changes  

        Args:  
            keyword: search keyword to monitor  
            geo: geographic location code  
            num_results: number of results to fetch  

        Returns:  
            list of change events emitted  
        """  
        print(f"Processing keyword: {keyword} (geo: {geo})")  

        # fetch current SERP  
        print("  Fetching current SERP...")  
        try:  
            raw_response = self.bright_data.search(  
                query=keyword,  
                num_results=num_results,  
                country=geo.lower()  
            )  
        except Exception as e:  
            print(f"  Error fetching SERP: {e}")  
            return []  

        # normalize current snapshot  
        current_snapshot = normalize_serp_data(raw_response, keyword, geo)  
        print(f"  Current snapshot: {len(current_snapshot['organic_domains'])} domains")  

        # load previous state  
        previous_snapshot = self.state_manager.load_previous_state(keyword, geo)  

        if previous_snapshot:  
            print(f"  Previous snapshot: {len(previous_snapshot['organic_domains'])} domains")  
        else:  
            print("  No previous state found (first run)")  

        # detect changes  
        change_events = self.change_detector.detect_changes(  
            current_snapshot,  
            previous_snapshot  
        )  

        print(f"  Detected {len(change_events)} change(s)")  

        # emit events to Kafka  
        emitted_events = []  
        for event in change_events:  
            try:  
                # use keyword+geo as key for partitioning  
                key = f"{keyword}:{geo}"  
                topic = "serp-changes"                  
                future = self.producer.send(topic, key=key, value=event)  
                # wait for confirmation (optional - can be async)  
                record_metadata = future.get(timeout=10)  

                print(f"    Emitted: {event['event_type']} -> partition {record_metadata.partition}, offset {record_metadata.offset}")  
                emitted_events.append(event)  
            except KafkaError as e:  
                print(f"    Error emitting event: {e}")  

        # save current state as new previous state  
        self.state_manager.save_state(keyword, geo, current_snapshot)  
        print("  State saved")  

        return emitted_events  

    def run_monitoring_loop(        self,  
        keywords: List[Dict[str, str]],  
        interval_seconds: int = 1800  # 30 minutes default    ):  
        """  
        run continuous monitoring loop  

        Args:  
            keywords: list of dicts with 'keyword' and 'geo' keys  
            interval_seconds: seconds between monitoring cycles  
        """  
        print(f"Starting monitoring loop (interval: {interval_seconds}s)")  
        print(f"Tracking {len(keywords)} keyword(s)")  

        try:  
            while True:  
                print("\\n" + "=" * 60)  
                print(f"Monitoring cycle at {time.strftime('%Y-%m-%d %H:%M:%S')}")  
                print("=" * 60)  

                for kw_config in keywords:  
                    keyword = kw_config['keyword']  
                    geo = kw_config.get('geo', 'US')  

                    try:  
                        self.process_keyword(keyword, geo)  
                        # small delay between keywords  
                        time.sleep(2)  
                    except Exception as e:  
                        print(f"Error processing {keyword}: {e}")  

                print(f"\\nSleeping for {interval_seconds} seconds...")  
                time.sleep(interval_seconds)  

        except KeyboardInterrupt:  
            print("\\nShutting down...")  
            self.producer.close()  


def main():  
    """example usage"""  
    producer = SERPProducer()  

    # example: monitor single keyword  
    keywords = [  
        {"keyword": "ai crm", "geo": "US"}  
    ]  

    # run once (for testing)  
    # producer.process_keyword("ai crm", "US")  

    # or run continuous loop  
    producer.run_monitoring_loop(keywords, interval_seconds=3600)  # 1 hour  


if __name__ == "__main__":  
    main()

Notice the synchronous Kafka sends (future.get(timeout=10))? This is intentional. We’re not optimizing for throughput, we’re monitoring a handful of keywords every 30 minutes. I’d rather wait 10 seconds and know the event was written than fail silently and lose a critical alert.

For a high-throughput system, you’d batch events and send asynchronously.

Also, note the Kafka partitioning key we use: key = f”{keyword}:{geo}”. This ensures you’ll never see a “domain entered” event before the “featured snippet appeared” event that caused it, because they’re in the same partition and processed sequentially.

Here’s how we use run_monitoring_loop() to start the pipeline:

producer = SERPProducer()  
keywords = [  
    {"keyword": "ai crm", "geo": "US"},  
    {"keyword": "best crm", "geo": "US"}  
]  
# run continuous loop (every 30 minutes)  
producer.run_monitoring_loop(keywords, interval_seconds=1800)

This pipeline runs continuously, emitting events whenever SERP composition changes. Next, we’ll see how multiple independent consumers process these events.

Step 5: The Consumers — Independent Event Processing

This is where Kafka’s value becomes obvious. We have three completely independent consumers processing the same event stream. Each has its own offset, its own logic, and can evolve independently.

But they all follow the same pattern:

Each consumer has a unique group_id — this is how Kafka tracks offsets per consumer
auto_offset_reset=’earliest’ — start from the beginning on first run (can replay history)
enable_auto_commit=True — automatically commit offsets after processing
Each consumer processes events at its own pace — if one is slow, others aren’t affected

Consumer 1: SEO Alerts

This consumer only cares about high-impact changes. Ads appearing? Featured snippets appearing? Those directly affect CTR. Everything else gets ignored.

"""  
Consumer 1: SEO Alerting Service  
listens for SERP feature changes that impact SEO  
"""  

import json  
import sys  
import os  
from datetime import datetime  
from kafka import KafkaConsumer  

# add src to path  
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../src'))  


class SEOAlertConsumer:  
    """  
    consumer that alerts on SERP changes affecting SEO  
    focuses on ads and featured snippets appearing  
    """  

    def __init__(        self,  
        kafka_brokers: str = "localhost:9092",  
        topic: str = "serp-changes",  
        output_file: str = None    ):  
        self.topic = topic  
        self.consumer = KafkaConsumer(  
            topic,  
            bootstrap_servers=kafka_brokers,  
            value_deserializer=lambda m: json.loads(m.decode('utf-8')),  
            group_id='seo-alert-consumer',  
            auto_offset_reset='earliest',  # start from beginning  
            enable_auto_commit=True  
        )  
        # set output file path relative to script directory  
        if output_file is None:  
            script_dir = os.path.dirname(os.path.abspath(__file__))  
            output_file = os.path.join(script_dir, 'seo_alerts.txt')  
        self.output_file = output_file  

    def process_event(self, event: dict) -> None:  
        """process a single change event"""  
        event_type = event.get('event_type')  

        # only care about feature additions (ads, featured snippets)  
        if event_type == 'serp_feature_added':  
            feature = event.get('feature')  

            if feature in ['ads', 'featured_snippet']:  
                alert = self._create_alert(event, feature)  
                self._save_alert(alert)  
                self._print_alert(alert)  

    def _create_alert(self, event: dict, feature: str) -> dict:  
        """create alert message"""  
        keyword = event.get('keyword')  
        geo = event.get('geo')  

        if feature == 'ads':  
            count = event.get('count', 1)  
            message = f"Heads up: Google added {count} ad(s) for '{keyword}' in {geo}. Expect organic CTR drop."  
        elif feature == 'featured_snippet':  
            owner = event.get('owner_domain', 'unknown')  
            message = f"Heads up: Google added a featured snippet for '{keyword}' in {geo} (owned by {owner}). Expect organic CTR drop."  
        else:  
            message = f"SERP feature '{feature}' appeared for '{keyword}' in {geo}"  

        return {  
            'timestamp': event.get('timestamp'),  
            'keyword': keyword,  
            'geo': geo,  
            'feature': feature,  
            'message': message,  
            'event': event  
        }  

    def _save_alert(self, alert: dict) -> None:  
        """save alert to file"""  
        try:  
            with open(self.output_file, 'a', encoding='utf-8') as f:  
                f.write(json.dumps(alert) + '\\n')  
        except Exception as e:  
            print(f"Error saving alert: {e}")  

    def _print_alert(self, alert: dict) -> None:  
        """print alert to console"""  
        print("\\n" + "=" * 60)  
        print("SEO ALERT")  
        print("=" * 60)  
        print(f"Time: {alert['timestamp']}")  
        print(f"Keyword: {alert['keyword']} ({alert['geo']})")  
        print(f"Feature: {alert['feature']}")  
        print(f"Message: {alert['message']}")  
        print("=" * 60)  

    def run(self):  
        """main consumer loop"""  
        print("SEO Alert Consumer started")  
        print(f"Listening to topic: {self.topic}")  
        print(f"Alerts will be saved to: {self.output_file}")  
        print("\\nWaiting for events...\\n")  

        try:  
            for message in self.consumer:  
                event = message.value  
                self.process_event(event)  
        except KeyboardInterrupt:  
            print("\\nShutting down...")  
        finally:  
            self.consumer.close()  


def main():  
    consumer = SEOAlertConsumer()  
    consumer.run()  


if __name__ == "__main__":  
    main()

This consumer simply filters for serp_feature_addedevents, and only processes ads and featured_snippet features. It creates human-readable alerts and saves them to a TXT file.

{  
  "timestamp": "2026-01-07T12:00:00Z",  
  "keyword": "ai crm",  
  "geo": "US",  
  "feature": "ads",  
  "message": "Heads up: Google added 3 ad(s) for 'ai crm' in US. Expect organic CTR drop.",  
  "event": { /* full event data */ }  
}

Consumer 2: Competitive Intelligence

This consumer tracks competitors. You can configure a list of domains to watch (hubspot.com, salesforce.com, etc.), and it logs when they enter or exit the SERP, or when they gain/lose the featured snippet.

"""  
Consumer 2: Competitive Intelligence Service  
tracks competitor movements in SERP  
"""  

import json  
import sys  
import os  
from kafka import KafkaConsumer  

# add src to path  
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../src'))  


class CompetitiveIntelConsumer:  
    """  
    consumer that tracks competitive movements  
    focuses on featured snippet ownership and domain entries/exits  
    """  

    def __init__(        self,  
        kafka_brokers: str = "localhost:9092",  
        topic: str = "serp-changes",  
        output_file: str = "competitive_intel.txt",  
        tracked_domains: list = None    ):  
        self.topic = topic  
        self.consumer = KafkaConsumer(  
            topic,  
            bootstrap_servers=kafka_brokers,  
            value_deserializer=lambda m: json.loads(m.decode('utf-8')),  
            group_id='competitive-intel-consumer',  
            auto_offset_reset='earliest',  
            enable_auto_commit=True  
        )  
        self.output_file = output_file  
        self.tracked_domains = set(tracked_domains or [])  
        # track featured snippet ownership  
        self.featured_snippet_owners = {}  # keyword+geo -> domain  

    def process_event(self, event: dict) -> None:  
        """process a single change event"""  
        event_type = event.get('event_type')  
        keyword = event.get('keyword')  
        geo = event.get('geo')  
        key = f"{keyword}:{geo}"  

        # track featured snippet ownership changes  
        if event_type == 'serp_feature_added' and event.get('feature') == 'featured_snippet':  
            owner = event.get('owner_domain')  
            if owner:  
                previous_owner = self.featured_snippet_owners.get(key)  
                if previous_owner != owner:  
                    self._log_featured_snippet_change(keyword, geo, previous_owner, owner, event)  
                    self.featured_snippet_owners[key] = owner  

        # track domain entries/exits  
        if event_type == 'organic_entry':  
            domain = event.get('domain')  
            if self._should_track(domain):  
                self._log_domain_entry(keyword, geo, domain, event)  

        elif event_type == 'organic_exit':  
            domain = event.get('domain')  
            if self._should_track(domain):  
                self._log_domain_exit(keyword, geo, domain, event)  

    def _should_track(self, domain: str) -> bool:  
        """determine if domain should be tracked"""  
        if not self.tracked_domains:  
            return True  # track all if none specified  
        return domain in self.tracked_domains  

    def _log_featured_snippet_change(        self,  
        keyword: str,  
        geo: str,  
        previous_owner: str,  
        new_owner: str,  
        event: dict    ) -> None:  
        """log featured snippet ownership change"""  
        if previous_owner:  
            message = f"Featured snippet ownership changed: {previous_owner} -> {new_owner}"  
        else:  
            message = f"{new_owner} gained SERP ownership via featured snippet"  

        intel = {  
            'timestamp': event.get('timestamp'),  
            'keyword': keyword,  
            'geo': geo,  
            'type': 'featured_snippet_ownership',  
            'previous_owner': previous_owner,  
            'new_owner': new_owner,  
            'message': message  
        }  

        self._save_intel(intel)  
        self._print_intel(intel)  

    def _log_domain_entry(        self,  
        keyword: str,  
        geo: str,  
        domain: str,  
        event: dict    ) -> None:  
        """log domain entry into SERP"""  
        position = event.get('current_position')  
        intel = {  
            'timestamp': event.get('timestamp'),  
            'keyword': keyword,  
            'geo': geo,  
            'type': 'domain_entry',  
            'domain': domain,  
            'position': position,  
            'message': f"{domain} entered SERP at position {position}"  
        }  

        self._save_intel(intel)  
        self._print_intel(intel)  

    def _log_domain_exit(        self,  
        keyword: str,  
        geo: str,  
        domain: str,  
        event: dict    ) -> None:  
        """log domain exit from SERP"""  
        previous_position = event.get('previous_position')  
        intel = {  
            'timestamp': event.get('timestamp'),  
            'keyword': keyword,  
            'geo': geo,  
            'type': 'domain_exit',  
            'domain': domain,  
            'previous_position': previous_position,  
            'message': f"{domain} dropped out of SERP (was at position {previous_position})"  
        }  

        self._save_intel(intel)  
        self._print_intel(intel)  

    def _save_intel(self, intel: dict) -> None:  
        """save intelligence to file"""  
        try:  
            with open(self.output_file, 'a', encoding='utf-8') as f:  
                f.write(json.dumps(intel) + '\\n')  
        except Exception as e:  
            print(f"Error saving intel: {e}")  

    def _print_intel(self, intel: dict) -> None:  
        """print intelligence to console"""  
        print("\\n" + "=" * 60)  
        print("COMPETITIVE INTELLIGENCE")  
        print("=" * 60)  
        print(f"Time: {intel['timestamp']}")  
        print(f"Keyword: {intel['keyword']} ({intel['geo']})")  
        print(f"Type: {intel['type']}")  
        print(f"Message: {intel['message']}")  
        print("=" * 60)  

    def run(self):  
        """main consumer loop"""  
        print("Competitive Intelligence Consumer started")  
        print(f"Listening to topic: {self.topic}")  
        print(f"Intelligence will be saved to: {self.output_file}")  
        if self.tracked_domains:  
            print(f"Tracking domains: {', '.join(self.tracked_domains)}")  
        print("\\nWaiting for events...\\n")  

        try:  
            for message in self.consumer:  
                event = message.value  
                self.process_event(event)  
        except KeyboardInterrupt:  
            print("\\nShutting down...")  
        finally:  
            self.consumer.close()  


def main():  
    # example: track specific competitors  
    tracked_domains = ['hubspot.com', 'salesforce.com', 'zoho.com']  

    consumer = CompetitiveIntelConsumer(tracked_domains=tracked_domains)  
    consumer.run()  


if __name__ == "__main__":  
    main()

This maintains an in-memory state — featured_snippet_owners — to detect ownership changes. Other than that, it can track all domains or specific competitors. Detects when featured snippet ownership changes (HubSpot → Salesforce), and tracks when competitors enter or exit the SERP

{  
  "timestamp": "2026-01-07T12:00:00Z",  
  "keyword": "ai crm",  
  "geo": "US",  
  "type": "featured_snippet_ownership",  
  "previous_owner": "hubspot.com",  
  "new_owner": "salesforce.com",  
  "message": "Featured snippet ownership changed: hubspot.com -> salesforce.com"  
},  
{  
  "timestamp": "2026-01-07T12:00:00Z",  
  "keyword": "ai crm",  
  "geo": "US",  
  "type": "domain_entry",  
  "domain": "zoho.com",  
  "position": 5,  
  "message": "zoho.com entered SERP at position 5"  
}

Of course, this can be extended to write to a database, generate reports, or trigger alerts.

Consumer 3: Volatility Analyzer

This consumer aggregates events over a time window and calculates a volatility score. More changes = higher volatility.


"""  
Consumer 3: Historical Volatility Analyzer  
aggregates SERP changes over time to calculate volatility scores  
"""  

import json  
import sys  
import os  
from datetime import datetime, timedelta  
from collections import defaultdict  
from kafka import KafkaConsumer  

# add src to path  
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../src'))  


class VolatilityAnalyzerConsumer:  
    """  
    consumer that calculates SERP volatility scores  
    tracks change frequency over time windows  
    """  

    def __init__(        self,  
        kafka_brokers: str = "localhost:9092",  
        topic: str = "serp-changes",  
        output_file: str = "volatility_scores.txt",  
        window_days: int = 7    ):  
        self.topic = topic  
        self.consumer = KafkaConsumer(  
            topic,  
            bootstrap_servers=kafka_brokers,  
            value_deserializer=lambda m: json.loads(m.decode('utf-8')),  
            group_id='volatility-analyzer-consumer',  
            auto_offset_reset='earliest',  
            enable_auto_commit=True  
        )  
        self.output_file = output_file  
        self.window_days = window_days  
        # track events per keyword+geo  
        self.events_by_keyword = defaultdict(list)  # (keyword, geo) -> [events]  

    def process_event(self, event: dict) -> None:  
        """process a single change event"""  
        keyword = event.get('keyword')  
        geo = event.get('geo')  
        key = (keyword, geo)  

        # store event with timestamp  
        self.events_by_keyword[key].append({  
            'event': event,  
            'timestamp': event.get('timestamp')  
        })  

        # calculate volatility for this keyword  
        volatility = self._calculate_volatility(key)  

        if volatility is not None:  
            score_data = {  
                'keyword': keyword,  
                'geo': geo,  
                'volatility_score': volatility,  
                'window_days': self.window_days,  
                'change_count': len(self.events_by_keyword[key]),  
                'timestamp': datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")  
            }  

            self._save_score(score_data)  
            self._print_score(score_data)  

    def _calculate_volatility(self, key: tuple) -> float:  
        """  
        calculate volatility score (0.0 to 1.0)  
        higher score = more changes = more volatile  
        """  
        events = self.events_by_keyword[key]  

        if len(events) `< 2:  
            return None  # need at least 2 events  

        # filter events within time window  
        cutoff_time = datetime.utcnow() - timedelta(days=self.window_days)  

        recent_events = [  
            e for e in events  
            if datetime.fromisoformat(e['timestamp'].replace('Z', '+00:00')).replace(tzinfo=None) >` cutoff_time  
        ]  

        if len(recent_events) `< 2:  
            return None  

        # simple volatility: change count normalized by time window  
        # more sophisticated: could weight by change type, magnitude, etc.  
        change_count = len(recent_events)  

        # normalize: assume 1 change per day = 0.5 score  
        # max reasonable: ~10 changes per day = 1.0 score  
        max_changes_per_day = 10  
        days_in_window = self.window_days  
        max_reasonable_changes = max_changes_per_day * days_in_window  

        volatility = min(change_count / max_reasonable_changes, 1.0)  

        return round(volatility, 2)  

    def _save_score(self, score_data: dict) ->` None:  
        """save volatility score to file"""  
        try:  
            with open(self.output_file, 'a', encoding='utf-8') as f:  
                f.write(json.dumps(score_data) + '\\n')  
        except Exception as e:  
            print(f"Error saving score: {e}")  

    def _print_score(self, score_data: dict) -> None:  
        """print volatility score to console"""  
        print("\\n" + "=" * 60)  
        print("VOLATILITY ANALYSIS")  
        print("=" * 60)  
        print(f"Keyword: {score_data['keyword']} ({score_data['geo']})")  
        print(f"Volatility Score: {score_data['volatility_score']:.2f}")  
        print(f"Window: {score_data['window_days']} days")  
        print(f"Change Count: {score_data['change_count']}")  
        print(f"Timestamp: {score_data['timestamp']}")  
        print("=" * 60)  

    def run(self):  
        """main consumer loop"""  
        print("Volatility Analyzer Consumer started")  
        print(f"Listening to topic: {self.topic}")  
        print(f"Scores will be saved to: {self.output_file}")  
        print(f"Analysis window: {self.window_days} days")  
        print("\\nWaiting for events...\\n")  

        try:  
            for message in self.consumer:  
                event = message.value  
                self.process_event(event)  
        except KeyboardInterrupt:  
            print("\\nShutting down...")  
        finally:  
            self.consumer.close()  


def main():  
    consumer = VolatilityAnalyzerConsumer(window_days=7)  
    consumer.run()  


if __name__ == "__main__":  
    main()

Here’s how I’m defining and calculating volatility here (capping at 1.0 or 100%) :

volatility = min(change_count / (max_changes_per_day * window_days), 1.0)

And this’ll produce and log alerts like this.

{  
  "keyword": "ai crm",  
  "geo": "US",  
  "volatility_score": 0.35,  
  "window_days": 7,  
  "change_count": 25,  
  "timestamp": "2026-01-07T12:00:00Z"  
}

This says: 25 changes in 7 days = 0.35 volatility (moderately volatile)

Is this the best volatility metric? Probably not. But it’s a starting point. You could weight by change type (featured snippet changes are more significant than domain entries), factor in change magnitude (position shifts vs entries/exits), or normalize by keyword search volume,

The point is this consumer can evolve independently. You can improve the volatility algorithm without touching the producer or other consumers.

Each consumer runs independently:

# Terminal 1: SEO Alerts  
python consumers/seo_alert_consumer.py  
# Terminal 2: Competitive Intelligence  
python consumers/competitive_intel_consumer.py  
# Terminal 3: Volatility Analyzer  
python consumers/volatility_analyzer_consumer.py

These consumers read from the same Kafka topic (serp-changes), each maintains its own offset (can process at different speeds), if one crashes, others continue processing, each writes to its own output file/log, and if you restart a consumer, it resumes from its last committed offset.

What’s The Payoff?

When you put all three consumers together, you stop asking “what rank are we today?” and start answering:

How competitive is this market right now?
Is Google monetizing [insert query here] more aggressively?
Are our competitors being displaced or reinforced?
Is this SERP worth investing in, or is it too volatile?
Are competitors winning because of strategy, or because of SERP structure?

See the shift? You’re no longer tracking positions. You’re observing market dynamics.

And because this is event-driven, every insight is replayable, every consumer can evolve independently, and every new question you want answered is just another consumer you have to write.

I Investigated the Top 3 AI-Generated Artists Going Viral on Spotify. Here’s Who They Are Imitating.

Prithwish Nath — Wed, 21 Jan 2026 16:19:22 +0000

Who had artists entirely generated by AI going viral on Spotify on their bingo card for 2025? Yeah, me neither. When I say “generated”, I’m talking everything from band promo shots, to members’ backgrounds, and instruments and vocals.

The Velvet Sundown went viral recently, as did Aventhis — who has over a million monthly listeners. Heck, a song by the AI-created act Breaking Rust was topping the U.S. Billboard Country Digital Song Sales chart in November this year. Breaking Rust has a whopping 2.5 million monthly listeners on Spotify.

Needless to say, this makes for a very lucrative business model. 👀

In 2025, Spotify pays artists roughly $0.003–$0.005 per stream, meaning 1 million streams earns about $3,000–$5,000, with most of that flowing to rights holders….but consider that GenAI-for-music tools like Suno and Udio cost under $10/month, or $30–$50/month for commercial rights, and that you’d have zero traditional expenses like studios, bands, or touring. In that context, a single AI-generated track hitting ~10 million streams could gross $30,000–$50,000, minus trivial distributor fees — an insanely low-cost, high-scale setup.

And that’s just Spotify.

So this piqued my curiosity. Sure, perhaps GenAI for music is more accessible than ever — but creating AI music that not only sounds polished but also *earns plays, likes, and real listener engagement?* That’s no easy task. For every AI-driven success story, there are thousands of boring Suno-created “lofi” playlists sitting at paltry 25–50 views on Spotify or YouTube.

What, then, are the standout AI artists — more accurately, the people behind them — doing differently? Which real artists do they want to sound like? Can we find out, acoustically speaking, how these algorithmic creations compare to the history of human-made music?

To explore that, I focused on three of the most prominent AI artists — Velvet Sundown, Breaking Rust, and Aventhis — collecting 30-second Spotify previews + lyrics (via a Bright Data proxy once I started getting HTTP 429's), breaking down their acoustic “fingerprint” using OpenL3, and comparing them against Spotify’s Top 2000 songs (up to 2020) + Spotify’s top 10 for each of their official (non-hidden) genres (obtained via Volt.fm) to see where their sound might sit in the broader musical landscape.

GitHub - marl/openl3: OpenL3: Open-source deep audio and image embeddings

Residential Proxies Trusted by Fortune 500 Companies - Free Trial

Here’s what I found. If you’d like to read at your own pace, here’s the Table of Contents. Enjoy!

AI Music Isn’t Doing Anything New — It’s Banking on What Already Works.

First of all, I should mention that all of these artists sound….perfect. There is no obvious artifacting, vocals being off key, instruments tuned incorrectly or cutting in/out abruptly. Whatever software they’re using to generate instruments and vocals — it’s working just fine. But this study is about far more than simple correctness.

You’d think that artists generated programmatically — yes, I know how AI works; I’m generalizing — would gravitate toward tightly defined, algorithmic genres (EDM substyles, algorithm-friendly lo-fi, etc.), or swing the other way entirely and lean into something aggressively experimental. After all, if you have an AI model at your fingertips and don’t try to push boundaries, what are you even doing? 😅

But that’s not what shows up in the data. Rather than sounding novel or genre-breaking, the most successful AI artists consistently map onto a very familiar musical space: melody-forward, mid-tempo, harmony-rich songs that sit comfortably alongside decades of mainstream, human-made music, and are guaranteed to get heavy airplay.

Here’s what scored a ~0.685 similarity according to my code.

🎵 AI artist, 2025

🎵 Human artist, 1976

In fact, let’s start with The Velvet Sundown.

AI Artist #1 — The Velvet Sundown

Sonically, they’re going hard for a warm, soft-focus, 1970s-era rock aesthetic — and the data backs that up almost embarrassingly well. Across 39 query tracks, the closest real-world matches are dominated by The Beatles and Fleetwood Mac, which land as the top two most similar artists overall. The Beatles take the top spot with a final similarity score of ~0.68, appearing in 37 out of 39 query songs, while Fleetwood Mac follows closely behind at ~0.66, matching 38 out of 39 queries. These aren’t marginal hits either, tracks like Here Comes the Sun, In My Life, Landslide, and Rhiannon repeatedly surface as high-similarity neighbors

Similarity Score (X-axis): Cosine similarity between each AI artist track and historical tracks from the artist. Scores range from 0 to 1, where higher values indicate greater similarity.Frequency (Y-axis): The number of track comparisons that fall into each similarity score range. Higher bars mean more tracks matched at that similarity level. A distribution skewed toward higher scores indicates consistent similarity across many tracks.

What’s interesting is that this alignment matches subjective listening almost perfectly. While their harmonies, pacing, and production texture feel straight out of the mid-70s “warm, analog” playbook (Fleetwood Mac, The Eagles), The Velvet Sundown’s vocals and melodic swells sound exactly like late-period Beatles balladry.

🎵 AI artist, 2025

🎵 Human artist, 1964

Like I said, whatever model + prompt they’re using isn’t inventing a new genre — it’s triangulating toward one of the safest, most historically successful regions of the musical embedding space.

And it doesn’t stop there. Their next tier of similar artists includes The Eagles, Bee Gees, going on to Queen, Dire Straits, Neil Young, and even modern successors like Coldplay and John Mayer — all artists known for emotionally legible songwriting, strong melodic hooks, and broad mainstream appeal.

Jesus. The Velvet Sundown is basically a statistically optimal blend of classic rock lineage, factory-made for familiarity. 😅

“Statistically optimal” is a great way to describe these artists, actually. None of them do anything wild, original, or creative. They’re AI-generated comfort food for your ears.

AI Artist #2 — Aventhis

If The Velvet Sundown felt like a near-perfect acoustic cosplay of a specific era and band, Aventhis is more diffuse. The model has no trouble identifying where Aventhis lives — firmly within modern country and Americana — but it’s noticeably less decisive about who they’re trying to sound like.

On paper, the genre match is strong. Across all songs Aventhis has on Spotify, they consistently map to contemporary country artists like Luke Combs, Morgan Wallen, and Chris Stapleton, all of whom appear in the top tier of results with solid similarity scores and high cross-query consistency.

So the system is correctly picking up on the hallmarks of modern country production: mid-tempo arrangements, prominent acoustic guitars, restrained percussion, and vocal-first mixes.

Where things get interesting is that no single artist fully dominates the similarity space the way Fleetwood Mac and The Beatles did for Velvet Sundown. Artists like Creedence Clearwater Revival, Queen, and even Muse creep into the top results — not because Aventhis sounds like them stylistically, but because some of these representative songs share broad acoustic properties: strong melodic arcs, clear harmonic structure, and emotionally legible song forms.

My theory is that the Aventhis project feels more like someone playing around with the software, and those early “feeling out” era tracks skew the analysis. Their more recent songs crystallize fairly well.

Regardless of era, Chris Stapleton shows up as the clearest vocal anchor — which aligns well with Aventhis’ branding as an “outlaw country” act.

🎵 AI artist, 2025

🎵 Human artist, 2025

Stapleton’s influence appears more in timbre: the gravel, the sustained notes, the emotional weight carried by the voice rather than the arrangement.

AI Artist #3 — Breaking Rust

Speaking of outlaw country…we have Breaking Rust which seems to exclusively create a monotonous sound that isn’t only mimicking the genre, it’s optimizing for a radio-friendly, familiar, “epic” image of it, engineered to sit dead-center in today’s casual-listener Spotify ecosystem.

This is a commercially attractive interpretation of what rebel/outlaw country music is supposed to be. (And that doesn’t have that much of an outlaw in there, unsurprisingly. 😅)

Morgan Wallen dominates the comparison set with a final score of ~0.71 and a 100% consistency ratio, meaning every single Breaking Rust track consistently mapped to Wallen’s catalog with high similarity

Luke Combs follows closely behind, again with perfect consistency.

Adele shows up here because of the clean, soaring vocals, and Queen does so because of the “thump & clap” routine that Breaking Rust seems to employ frequently.

There’s also very little experimentation here. Classic country influences (Cash, Jennings, early outlaw archetypes) do show up in the data, but barely surface acoustically. Even older rock-adjacent matches like Creedence Clearwater Revival appear only as secondary signals. After The Velvet Sundown, Breaking Rust is easily the most “engineered” sound — created not to reinvent the country genre but almost like it’s trying too hard to win at it, by anchoring itself squarely to the most commercially successful sound of the past decade.

And based on both chart performance and these similarity scores, that strategy appears to be working. Here’s another ~0.6 similarity match. While Breaking Rust is topping Billboard charts, Bryan Elijah Smith — while still successful — isn’t.

🎵 AI artist, 2025

🎵 Human artist, 2024

Breaking Rust is the perfect opportunity to talk about another big problem with these artists — they’re far too monotonous.

Repetitive Catalogues, Optimized for Recommendation Engines

Up to this point, we’ve been asking a fairly intuitive question: which real artists do these AI projects sound like? But there’s another, arguably more revealing angle here — how similar an AI artist’s songs are to *each other*.

To answer that, I ran the same embedding pipeline again, but this time comparing each AI Artist track against every other track from the same artist, producing three full intra-artist similarity matrices.

Conceptually, this tells us how tightly clustered the catalog is in embedding space — whether the artist has a recognizable sonic “center,” or whether the songs scatter across different musical regions.

If you were wondering, the artists/data points with the most variation in their internal catalogues (bottom right) here are Queen (0.120), David Bowie (0.162), and The Beatles (0.213). The least (top left) are Kensington (0.655), Doe Maar (0.633) and AC/DC (0.606).

The Velvet Sundown are extremely internally consistent. Most track-to-track cosine similarities land in the ~0.6–0.8 range, with many pairs pushing even higher. That’s near-homogeneity in tempo, timbral palette, harmonic structure, and arrangement style. In other words, once you’ve heard 3–4 Velvet Sundown tracks, you’ve essentially heard ’em all.

Breaking Rust is even worse.

Their intra-similarity scores are ridiculously high across the board, with a strong average, a high median, and low variance compared to the other two artists

Even more than Velvet Sundown, almost every Breaking Rust song sounds a lot like every other Breaking Rust song. Close enough that just one song could describe its entire catalog.

This is especially evident in the most similar pairs, where similarities routinely exceed 0.85 — values you’d normally expect from alternate takes, or remixes, not individual songs. Even from the same album. Tempo, instrumentation, vocal tone, and structure are nearly one-dimensional.

Strictly from an algorithmic POV, this explains why Breaking Rust works so well on Spotify: it’s predictable in the best possible way. If you like one song, the Spotify algorithm can safely serve you ten more without risking a skip. From a modeling perspective, it’s also the clearest example of how AI systems naturally converge on a local optimum when the people behind it get lazy — once a sound “works,” it gets reused aggressively unless you’re actively correcting for it.

Depressing, but further proof that rather than artistic intent, these AI musicians seem to have been designed for streaming efficiency — music engineered to sound “right enough” to play to the algorithm of major recommendation engines.

Aventhis is a statistical anomaly here for reasons mentioned before. While it does show lower average similarity and higher variance, including a few pairs that are barely related at all, the issue is that their songs jump genres often, making such analysis misleading.

Lyrics Are Technically Correct, But Emotionally Nonexistent.

Once you strip away production, vocals, and genre cues, lyrics are where artistic intent — or its absence — tends to leak through. This is what actually ties our quantitative results and the “felt experience” together.

So instead of overfitting on rhyme schemes or clever metaphors, I looked at three broad dimensions:

Lexical & syntactic complexity (how varied and structurally complex the language is)
Rhyme density & repetition (how “song-like” vs templated the writing feels)
Sentiment & emotion (what emotional space these songs consistently occupy)

First of all, there’s not much to distinguish these artists in terms of word/sentence sizes or lengths. Breaking Rust has the most words per song, but they’re all pretty uniform.

The patterns that emerge upon deeper analysis though are… telling. If the audio analysis showed that these AI artists sit comfortably inside well-worn musical lanes, the lyric analysis confirms this: across all three artists, the lyrics read less like authored statements and more like statistical averages of their genre.

Velvet Sundown: Aesthetic Without Substance

Their lyrics are the most revealing case because they sound poetic on first pass. There’s no shortage of imagery: “boots in the mud,” “smoke in the sky,” “shadows falling,” “voices whisper,” “marching ghosts”. And War, peace, rebellion, fire, silence — repeated endlessly, rarely developed.

Lines like:

Dust on the windBoots on the groundSmoke in the skyNo peace foundRivers run redThe drums roll slowTell me brother, where do we go?

Sound meaningful if you’re 14, but are basically just symbolic placeholders rather than narrative beats. This is proven by the metrics: Velvet Sundown has low word entropy, short lines, and extremely low readability scores — they’re easily digestible. High TTR (0.61) and longer average word length (~4.1) give the illusion of richness, but the lyrics are syntactically simple and emotionally generic.

Their rhyme density is fairly low, but that’s misleading — this is because their lyrics are sparse by design.

Visual representation of rhyming line pairs for all songs by the selected artist. Lines are shown as dots on a vertical axis, with curved arcs connecting rhyming lines. Thicker arcs indicate more frequent repetitions, revealing chorus recycling patterns and repetition across long distances.

Really, it’s when you actually read the lyrics end-to-end that the cracks show hard:

Almost every song collapses into very brief (3–4 word) phrases, and the same moral binary: war bad, peace good
No characters, no storytelling, no deeper narratives or themes, no temporal progression. Oh, and “flag” and “dust” are mentioned a ton, for some reason.
Choruses recycle very abstract imperatives: “raise your voice,” “don’t look away,” “we won’t fade”

The language gestures at a seriousness without ever committing to specifics. It’s protest music with the protest removed. It’s more like protest aesthetic — a Spotify-ready approximation of “serious rock” that never risks alienating anyone by saying anything precise.

Aventhis: Performative Masculine Trauma

Aventhis is more coherent than Velvet Sundown, but only because it is far less poetic and commits fully to a single archetype: the wounded, defiant outlaw. Every song reinforces the same emotional posturing — mistrust, self-reliance, pain turned inward — to the point that they all blur together.

The lyrics are packed with stock phrases of what could only be called masculine suffering:

“Scars where I never kept my mouth shut”
“I ride through hell with my head held high”
“I walk with ghosts that don’t disappear”
“I don’t kneel or cry”

Unlike Velvet Sundown’s abstract collectivism, Aventhis is intensely first-person — but still vague. The pain is constant. And also (conveniently!), undefined. Abuse, regret, sin, and violence are all implied, and occasionally they’ll include superficial country-genre specifics (whiskey, barbed wire, boots, fire).

Things you’ve heard a thousand times before from a thousand one-note Top 40 Country Billboard artists.

Lexically, Aventhis sits in the middle: moderate vocabulary diversity, average word length, and fairly standard line lengths. But the real signal is consistency. Their MATTR is high and stable, entropy is middling, and syntactic measures are tightly clustered across songs.

Emotional distribution is comically negative, and arguably, this is engineered for more engagement.

Emotionally, however, Aventhis is almost comically narrow compared to the others: Fear dominates the emotional distribution, but not in a nuanced way — it’s fear as some sort of… aesthetic texture, nothing that ever feels like a lived experience. Even empowerment anthems like “Burn What Held You” resolve into catchy slogans rather than anything deep.

Tension, threat, perseverance, moral conflict. They’re cosplaying as the “gritty outlaw” archetype, and they’re doing it efficiently, repeatedly, and in a way that slots neatly into modern outlaw-country playlists.

Breaking Rust: Simple, Repetitive, and Chorus-Forward

They are the most structurally “song-like” in the traditional sense — and also the least complex.

Lexical diversity is the lowest of the three, sentences are shorter, and syntactic depth is shallow. That’s not inherently bad, but it does correlate with what shows up elsewhere: extremely high rhyme and repetition.

This is music designed to be instantly familiar and easy to latch onto. The sentiment is mostly neutral, with sadness edging out fear and anger. Emotionally, it’s less intense than Aventhis and less layered than Velvet Sundown.

Songs like “Kicking Back at the Ground”, “Livin’ on Borrowed Time”, and “Whiskey Don’t Talk Back” are built from familiar country scaffolding: hard work, pride, scars, drinking, resilience. The imagery is clearer than Velvet Sundown’s and (thankfully!) less theatrical than Aventhis’s, but the emotional range is easily the narrowest here.

You see the same ideas recycled with small lexical swaps:

“Dust on my boots”
“Scars that sing”
“Pour another glass”
“I was born this way”

The metrics show high repetition, high readability, and very stable sentiment — almost aggressively neutral.

There are no lines that reframe a familiar idea or linger on an uncomfortable detail. Everything resolves cleanly into affirmation: keep going, stay true, don’t quit. It’s music designed to be nodded along to, not sat with, or thought about for any length of time.

The Bigger Pattern…is Depressing.

Across all three artists, we keep seeing the same things:

Low semantic risk; short, uncomplicated words, in short sentences. I imagine this is done for wide appeal as well as not risking their AI vocal generation tripping up on complex words or phrasing.
High emotional clarity, but extremely low emotional depth
Heavy reliance on genre signifiers. Again, see: mimicking known archetypes.
Heavy emphasis on rhyming (combined with #1)
Lyrics optimized for recognizability, not memorability

These songs all feel like the lyrical equivalent of stock photography — technically correct, emotionally legible, and instantly forgettable.

They don’t challenge the listener. They don’t demand interpretation. They don’t risk failure. And in a streaming economy that rewards familiarity over depth, that might be exactly the point.

More proof that these AI musicians seem to have been designed for maximum playlist compatibility and minimum listener friction.**** Real artists sound like they’re discovering a voice. These sound like systems that already know exactly which one performs best.

That’s everything. If you want to learn about my methodology in detail, read on. Else, click this to go back to the table of contents.

Methodology

To compare AI-generated music with historical human-made songs in a way that’s both scalable and acoustically grounded, I treated this as a representation-learning + similarity search problem rather than a genre-labeling or metadata exercise.

First of all, I collected those 30-second Spotify preview clips for:

Everything the three AI artists (Velvet Sundown, Breaking Rust, and Aventhis) had on Spotify,
10 songs for every track in the Volt.fm charts for Spotify’s 26 official genres
Everything in the Spotify Top 2000 dataset.

So that’s a total of about ~2500 tracks. Then, I used the Spotify API to programmatically search and find the track URLs + scraped those URLs using Bright Data’s proxies to stop me from getting blocked.

Spotify’s 30-second previews are short, but they’re consistent, legally accessible, and widely used in MIR research — making them a reasonable proxy for a song’s overall sonic fingerprint.

Scraping Spotify preview audio (the slightly hacky part)

My solution was to programmatically go through a list of song + artist name combos like:

{"song": "Let it Burn", "artist": "The Velvet Sundown"}

And use the Spotify API (registration required, but free) — specifically, the /search endpoint — to grab the Spotify Web Player link for that song contained in the external_urls field of the response object. This looks like:

https://open.spotify.com/track/64JjvzPdH2h3u5cJDF4y96

Why get this? Well, Spotify does not expose preview_url via their public API anymore. However, the preview URLs do exist in the page metadata — specifically in the og:audio Open Graph tag — but only under certain conditions.

When requesting a track page with bot-like headers (say, your standard axios user-agent), Spotify returns server-rendered HTML that includes:

<meta property="og:audio" content="https://p.scdn.co/mp3-preview/..." />

But when requesting the same page in the browser, or with browser-like headers, Spotify instead serves a JavaScript-driven React shell with no audio metadata present in the initial HTML. This is why you won’t see preview URLs via “View Source” in a normal browser session.

That makes it easy for us to scrape them. All we have to do is get the track URL, fetch the HTML for that page, and search for the CDN pattern on it.

import os
import sys
import json
import ssl
import time
import argparse
from urllib.parse import urlencode
import urllib.request

import requests
from bs4 import BeautifulSoup
from dotenv import load_dotenv

ssl._create_default_https_context = ssl._create_unverified_context

load_dotenv()

# example: list of songs with artist names
# format: [{"song": "Song Name", "artist": "Artist Name"}, ...]
# if INPUT_FILE is provided, it will load from that file instead
SONGS = [
    # example entries - replace with your own or use an inputfile JSON
    {"song": "Let it Burn", "artist": "The Velvet Sundown"},
    {"song": "As the Silence Falls", "artist": "The Velvet Sundown"},
]


def build_brightdata_proxy():
    proxy_user = os.environ.get("BRIGHT_DATA_PROXY_USER", "")
    proxy_pass = os.environ.get("BRIGHT_DATA_PROXY_PASS", "")
    if not (proxy_user and proxy_pass):
        return None
    proxy_url = f"@brd.superproxy.io:33335">http://{proxy_user}:{proxy_pass}@brd.superproxy.io:33335"
    return proxy_url


def get_spotify_access_token():
    token = os.environ.get("SPOTIFY_ACCESS_TOKEN")
    if not token:
        raise RuntimeError("SPOTIFY_ACCESS_TOKEN environment variable is required")
    return token


def search_spotify_tracks(song_name, artist_name, limit=5):
    query = f'track:"{song_name}" artist:"{artist_name}"'
    params = {
        "q": query,
        "type": "track",
        "market": "US",
        "limit": str(limit),
    }
    headers = {"Authorization": f"Bearer {get_spotify_access_token()}"}
    resp = requests.get("https://api.spotify.com/v1/search", params=params, headers=headers)
    resp.raise_for_status()
    return resp.json().get("tracks", {}).get("items", [])


def get_spotify_links(url):
    proxy_url = build_brightdata_proxy()
    if proxy_url:
        opener = urllib.request.build_opener(
            urllib.request.ProxyHandler({'https': proxy_url, 'http': proxy_url})
        )
        response = opener.open(url)
        html = response.read().decode('utf-8')
    else:
        resp = requests.get(url)
        resp.raise_for_status()
        html = resp.text

    soup = BeautifulSoup(html, "html.parser")

    scdn_links = set()

    for element in soup.find_all(True):
        for attr_name, attr_value in element.attrs.items():
            if isinstance(attr_value, (list, tuple)):
                attr_values = attr_value
            else:
                attr_values = [attr_value]
            for value in attr_values:
                if value and "p.scdn.co" in value:
                    scdn_links.add(value)

    return list(scdn_links)


def find_preview_url(track_name, artist_name):
    """find preview URL for a track using Spotify API search and scraping"""
    try:
        tracks = search_spotify_tracks(track_name, artist_name, limit=3)

        if not tracks:
            return None

        # find best match by checking artist similarity
        best_match = None
        artist_lower = artist_name.lower()

        for track in tracks:
            track_artists = ', '.join(artist_obj['name'] for artist_obj in track['artists'])
            if artist_lower and artist_lower in track_artists.lower():
                best_match = track
                break

        # if no exact match, use first result
        if not best_match:
            best_match = tracks[0]

        # get Spotify URL and scrape for preview
        spotify_url = best_match.get("external_urls", {}).get("spotify")
        if not spotify_url:
            return None

        preview_urls = get_spotify_links(spotify_url)
        if preview_urls and len(preview_urls) > 0:
            return preview_urls[0]

        return None
    except Exception as error:
        print(f"Error finding preview for {track_name}: {error}", file=sys.stderr)
        return None


def load_songs(input_file):
    """load songs from input file or use SONGS list"""
    if input_file and os.path.exists(input_file):
        print(f"Loading songs from: {input_file}")
        with open(input_file, 'r', encoding='utf-8') as f:
            data = json.load(f)

        # support both array format and object format
        if isinstance(data, list):
            return data
        elif isinstance(data, dict) and "songs" in data and isinstance(data["songs"], list):
            return data["songs"]
        else:
            raise ValueError('Input file must contain an array of {"song", "artist"} objects or {"songs": [...]}')

    return SONGS


def get_previews(input_file=None, output_file='spotify-preview_urls.json'):
    """main function to get preview URLs for all songs"""
    songs = load_songs(input_file)

    # validate songs format
    if not isinstance(songs, list) or len(songs) == 0:
        print('Error: No songs found. Please provide songs in the SONGS list or via INPUT_FILE.', file=sys.stderr)
        print('Format: [{"song": "Song Name", "artist": "Artist Name"}, ...]', file=sys.stderr)
        sys.exit(1)

    # validate each entry has song and artist
    for i, song in enumerate(songs):
        if not song.get("song") or not song.get("artist"):
            print(f"Error: Entry {i + 1} is missing 'song' or 'artist' property.", file=sys.stderr)
            print('Format: {"song": "Song Name", "artist": "Artist Name"}', file=sys.stderr)
            sys.exit(1)

    preview_urls = {}
    found_count = 0
    not_found_count = 0
    total_tracks = len(songs)

    print(f"Found {total_tracks} songs to process\n")
    print('=' * 60)

    # process each song
    for i, song_entry in enumerate(songs):
        song_name = song_entry["song"].strip()
        artist_name = song_entry["artist"].strip()

        print(f"\n[{i + 1}/{total_tracks}] Searching: {song_name} - {artist_name}")

        # find preview URL
        preview_url = find_preview_url(song_name, artist_name)

        if preview_url:
            # use song name as key, or song + artist for uniqueness
            key = f"{song_name} - {artist_name}"
            preview_urls[key] = preview_url
            found_count += 1
            print(f"  ✓ Found: {preview_url[:70]}...")

            # save immediately when preview URL is found
            with open(output_file, 'w', encoding='utf-8') as f:
                json.dump(preview_urls, f, indent=2, ensure_ascii=False)
            print(f"  Saved to JSON ({found_count} found so far)")
        else:
            not_found_count += 1
            print(f"  ✗ No preview URL found")

        # rate limiting between tracks
        if i < len(songs) - 1:
            time.sleep(0.5)

    # final save
    with open(output_file, 'w', encoding='utf-8') as f:
        json.dump(preview_urls, f, indent=2, ensure_ascii=False)

    # summary
    print('\n' + '=' * 60)
    print('Summary:')
    print('=' * 60)
    print(f"Preview URLs found: {found_count}")
    print(f"Preview URLs not found: {not_found_count}")
    print(f"Total tracks processed: {total_tracks}")
    print(f"\nResults saved to: {output_file}")


def main():
    parser = argparse.ArgumentParser(
        description='Get Spotify preview URLs for a list of songs and artists'
    )
    parser.add_argument(
        'input_file',
        nargs='?',
        default=None,
        help='Optional JSON input file with songs list'
    )
    parser.add_argument(
        'output_file',
        nargs='?',
        default='spotify-preview_urls.json',
        help='Output file path (default: spotify-preview_urls.json)'
    )

    args = parser.parse_args()

    try:
        get_previews(args.input_file, args.output_file)
    except Exception as error:
        print(f'Fatal error: {error}', file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()

If you’re following along and are using a proxy, you need to sign up here for the username, password, zone, etc.

Bright Data - All in One Platform for Proxies and Web Scraping

Anyway, I collected those CDN URLs for all the songs I wanted, dumped them to a JSON, and downloaded them directly from Spotify's CDN (p.scdn.co) for offline processing.

Song Fingerprinting with OpenL3

Then, each audio clip was converted into a fixed-length representation using OpenL3, a deep audio embedding model trained to capture perceptual and musical characteristics beyond simple features like tempo or key.

Here’s a minimal representation.

import numpy as np
import openl3
import librosa
import pickle
from pathlib import Path

def extract_embedding_from_mp3(filepath: Path) -> np.ndarray:
    """
    extract OpenL3 embedding from an MP3 file

    Args:
        filepath: path to MP3 file

    Returns:
        embedding vector (512-dimensional by default)
    """
    # load audio file at 48kHz (required by OpenL3)
    audio, sr = librosa.load(str(filepath), sr=48000)

    # extract embedding using OpenL3
    # returns (embeddings, timestamps) where embeddings is (n_frames, 512)
    emb, _ = openl3.get_audio_embedding(
        audio,
        sr,
        content_type='music',
        input_repr='mel256',
        embedding_size=512
    )

    # average over time frames to get a single embedding vector
    embedding = np.mean(emb, axis=0)

    return embedding

def process_mp3_directory(directory: Path) -> tuple[list, list]:
    """
    process all MP3 files in a directory and extract embeddings with metadata

    Args:
        directory: directory containing MP3 files

    Returns:
        tuple of (embeddings_list, metadata_list)
    """
    mp3_files = sorted(list(directory.glob("*.mp3")))
    embeddings = []
    metadata = []

    for filepath in mp3_files:
        # extract embedding
        embedding = extract_embedding_from_mp3(filepath)

        # parse filename for metadata (assumes format: "Track Name - Artist Name.mp3")
        if " - " in filepath.stem:
            parts = filepath.stem.rsplit(" - ", 1)
            track_name = parts[0].strip()
            artist_name = parts[1].strip()
        else:
            track_name = filepath.stem
            artist_name = "Unknown"

        # build metadata dict
        meta_dict = {
            "file_path": str(filepath),
            "track_name": track_name,
            "artist": artist_name
        }

        embeddings.append(embedding)
        metadata.append(meta_dict)

    return embeddings, metadata

def save_embeddings(embeddings: list, metadata: list, output_dir: str = "embeddings"):
    """save embeddings and metadata to disk"""
    output_path = Path(output_dir)
    output_path.mkdir(exist_ok=True)

    # convert to numpy array and save
    embeddings_array = np.array(embeddings)
    np.save(output_path / "embeddings.npy", embeddings_array)

    # save metadata
    with open(output_path / "metadata.pkl", 'wb') as f:
        pickle.dump(metadata, f)

# example usage
if __name__ == "__main__":
    mp3_dir = Path("previews")
    embeddings, metadata = process_mp3_directory(mp3_dir)
    save_embeddings(embeddings, metadata)
    print(f"extracted {len(embeddings)} embeddings")

What this does:

Loads audio from each MP3 file at 48kHz using librosa.
Creates metadata dictionaries with file path, track name, and artist (parsed from filename, saved as pickle files)
Extracts embeddings with openl3.get_audio_embedding().
Average over time frames to get a single vector, saved as a numpy array (.npy binaries)

OpenL3 produces frame-level embeddings, so I averaged them over time to obtain a single 512-dimensional vector per track.

To make cosine similarity meaningful at scale, I applied mean-centering to remove global bias, followed by L2 normalization, and indexed the historical songs using FAISS with an inner-product index.

import numpy as np
import faiss
import pickle
from pathlib import Path

def build_faiss_index(embeddings: np.ndarray, metadata: list) -> faiss.Index:
    """
    build FAISS index from embeddings for cosine similarity search

    Args:
        embeddings: numpy array of embeddings (n_samples, n_features)
        metadata: list of metadata dictionaries (must match embedding order)

    Returns:
        FAISS index
    """
    n_samples, n_features = embeddings.shape

    # normalize embeddings for cosine similarity
    # subtract global mean to improve separation
    mean_vec = np.mean(embeddings, axis=0, keepdims=True)
    embeddings_centered = embeddings - mean_vec

    # L2 normalize
    norms = np.linalg.norm(embeddings_centered, axis=1, keepdims=True)
    norms[norms == 0] = 1
    embeddings_normalized = embeddings_centered / norms

    # create inner product index (for cosine similarity on normalized vectors)
    index = faiss.IndexFlatIP(n_features)
    index.add(embeddings_normalized.astype('float32'))

    return index

def save_index(index: faiss.Index, metadata: list, output_dir: str = "faiss_index"):
    """save FAISS index and metadata to disk"""
    output_path = Path(output_dir)
    output_path.mkdir(exist_ok=True)

    # save index
    faiss.write_index(index, str(output_path / "index.faiss"))

    # save metadata (order must match FAISS IDs exactly)
    with open(output_path / "metadata.pkl", 'wb') as f:
        pickle.dump(metadata, f)

# example usage
if __name__ == "__main__":
    # load embeddings and metadata
    embeddings = np.load("embeddings/historical_embeddings.npy")
    with open("embeddings/historical_metadata.pkl", 'rb') as f:
        metadata = pickle.load(f)

    # build index
    index = build_faiss_index(embeddings, metadata)

    # save index
    save_index(index, metadata)

    print(f"built index with {index.ntotal} vectors")

What this does:

Loads embeddings (npy) and metadata (pkl) from disk
Subtracts the global mean, then L2 normalizes each vector
Builds a FAISS index: creates an inner product index (cosine similarity on normalized vectors) and adds the normalized embeddings
Writes the FAISS index and metadata pickle (order matches FAISS vector IDs)

With all of it now being in a vector database I could simply query the index for each AI generated artist for its nearest neighbors, and then aggregate results at the artist level using a top-k similarity strategy (mean of the top three matching tracks per artist, k = 400), combined across all AI tracks.

For each query song, I retrieve the top 400 nearest neighbors from the FAISS index, then convert distances to similarity scores (0.0–1.0 range, where 1 = most similar. This is not a percentage.)

Code used in this step: https://gist.github.com/sixthextinction/863140ec9d9126b142f058629ab873a1

Things to remember:

Historical embeddings are normalized when building the index. Query embeddings must be normalized the same way (using the saved mean vector) so they’re in the same space for similarity search.
Artists that appeared consistently across many queries need to be weighted more heavily, while low-confidence matches need to be filtered out.
Artists with fewer than 3 tracks are excluded (low confidence)

The result : three ranked lists of historical artists (I just took the Top 4) whose catalogs were acoustically closest — in embedding space — to each AI-generated artist.

Here’s how I’m scoring this. The final_score is:

mean_artist_score * (0.7 + 0.3 * consistency_weight)

With Range: 0.0 to 1.0 (higher = more similar)

Where:

mean_artist_score: average of the top-3 track similarities across all query songs
consistency_weight: ratio of query songs that matched this artist (capped at 1.0)

I put a 0.7 base weight on similarity and a 0.3 bonus for consistency across queries, so, for example: if an artist matches 5/10 query songs, consistency_weight = 0.5, so the multiplier is 0.7 + 0.3 × 0.5 = 0.85

Finally, I collected lyrics for all three artists from Genius and the artists’ own YouTube video descriptions, and ran a short lyrical analysis pipeline.

Genius | Song Lyrics & Knowledge

In it, rather than hand-waving about abstract “themes,” I quantified concrete lyrical properties. For each song, I measured lexical diversity using both Type–Token Ratio (TTR) and Moving-Average TTR (MATTR, more stable on short texts like lyrics), syntactic complexity (sentence length, subordination depth), sentiment and emotion distributions, and rhyme structure (rhyme density and reuse). These per-song metrics were then aggregated at the artist level to highlight patterns that persist across entire catalogs — not just individual tracks.

After that it was simply a matter of interpreting the data, and generating visualizations.

Limitations of This Study

While the historical comparison set was large — the Spotify Top 2000 dataset plus top tracks across 26 major Spotify genres — it’s still finite and biased toward commercially successful music.
My analysis also relies on short 30 second preview clips for audio embeddings and publicly available lyric transcriptions, which is good enough for a similarity analysis, but can never be a perfect comparison.

And that’s absolutely everything. Thank you for reading!

Thank you for reading! 🙌 This was the fourth in a series of data-driven deep dives I’m doing — forensic teardowns of things that are interesting, or things that shouldn’t work but do. If you want to see what else I find buried in data, follow along.

Why Google, Bing, DuckDuckGo & Yandex Show Different Results For the Same Query (2026)

Prithwish Nath — Wed, 21 Jan 2026 16:05:00 +0000

TL;DR — Search engines don’t just retrieve information; they decide what counts as “knowledge”. Depending on the engine, the same query can prioritize institutional safety, consensus, monetization, or unfiltered chaos. This investigation maps those differences.

Just how differently do the most popular search engines work?

To find out, I ran a search-engine forensics investigation using identical queries across Google, Bing, DuckDuckGo, and Yandex. Using Bright Data’s SERP infrastructure, I collected 1,630 SERP results spanning commercial, technical, YMYL (Your Money or Your Life) topics, and open-ended “wildcard” questions, and analyzed the titles and snippets users actually see. I treated each result as a data point and mapped the emergent “information signatures” of each platform — without scoring truth or intent, only structure, emphasis, and source selection.

What I found was far more than just minor ranking variations. I actually had distinct “personalities”, here. Even when user intent was identical, the answers were structurally different. It’s almost as if reality fragments at the search-engine layer — and the version of the world you encounter depends on which engine you trust.

1. Domain Diversity vs Domain Authority

How Many Voices Does Each Engine Allow?

The first signal is simple: how many distinct domains appear per query. On average, Yandex surfaced the most unique domains per query (~11.7), followed by DuckDuckGo (~9.4), Google (~8.8), and Bing (~6.2).

Average unique domains per query, across all four search engines.

This isn’t a value judgment in and of itself, but it is revealing. Higher diversity suggests a broader sampling of sources; lower diversity implies tighter editorial consolidation.

Bing’s comparatively low diversity hints at a strong preference for repeat “trusted” sources (and we’ll soon see just how deep that particular rabbit hole goes.) Yandex and DuckDuckGo, by contrast, appear structurally more exploratory — willing to surface a wider range of publishers, even if that comes at the cost of consistency.

Who Gets to Speak, and When?

Breaking sources down by domain type (.gov, .edu, .org, .com) reveals how engines encode authority differently — especially under YMYL conditions.

In commercial queries, all engines overwhelmingly favored .com domains, with minimal variation.

While the split for commercial queries is as expected, YMYL content is where the differences really start showing. Google favors institutional sources overwhelmingly, while Yandex favors .com domains more.

But for YMYL queries, big differences start showing up.

Google’s top results skews heavily toward government, academic, and nonprofit sources, with .gov/.edu/.org accounting for a majority of top-ranked content.
Bing also leaned institutional, though less aggressively.
DuckDuckGo and Yandex showed a noticeably higher tolerance for commercial and mixed-authority sources even in sensitive domains.

This suggests that “authoritativeness” is not a universal concept for search engines — it is defined + enforced differently depending on platform philosophy and risk posture.

For domain type distribution, the Technical and Wildcard categories weren’t really interesting — they’re exactly as you’d expect, and similar to Commercial. I’m skipping those.

2. The Personalities of Search Engines

By this point, it’s clear that search engines don’t just rank differently — they behave differently. Each one shows a consistent “retrieval” personality, so to speak. By that, I mean a set of preferences about authority, risk, diversity, and acceptable sources of knowledge. These “personalities” show up across categories, queries, and metrics. Let’s look into each search engine and see what kind of personality they showed.

Most frequently appearing domains across all 4 search engines.

Google: The Institutional Gatekeeper. For Better or Worse.

For YMYL queries — health, finance, safety — this is where Google behaves the least like a neutral party/index — it has clearly taken on the responsibility of only showing the highest weighted domains, no matter what.

A breakdown of Google’s top domains. There’s a healthy mix, if skewed towards institutional sources.

Across YMYL searches, 61% of Google’s results come from .gov, .edu, or .org domains, already a clear majority. But the strongest signal appears at the very top of the search results page. In the top three positions, institutional sources account for 73.33% of results, while commercial (.com) domains drop to just 16.67%.

Google doesn’t merely include institutional authority — it front-loads it, especially where perceived risk is highest.

Query: “is coffee bad for you”
Google’s Top 3:Mayo Clinic
Johns Hopkins Medicine
UT Southwestern Medical Center

In the top ten, seven results come from major medical institutions, with community or experiential sources appearing only after some sort of authoritative consensus has been established.

Needless to say, this pattern is not accidental. It reflects a deliberate policy choice on Google’s part to privilege institutional authority under YMYL conditions. From Google’s POV, obviously this is pretty rational risk management. At global scale, health and finance queries are not abstract knowledge problems; they are legal, political, and ethical liabilities for Google. Frontloading institutional sources let them externalize that responsibility, stay in line with regulatory expectations, and minimize the chance of any catastrophic harm.

But this stance comes with trade-offs.

By systematically doing this, Google collapses disagreement into consensus and emergent knowledge into settled fact. Independent researchers, patient communities, and experience-driven perspectives are not necessarily excluded — but they are consistently positionally suppressed in these search results pages, especially in the moments where users are most likely to stop scrolling.

Over time, this creates a feedback loop. Institutional voices receive disproportionate visibility and legitimacy, alternative perspectives lose reach, and institutional consensus appears even more dominant. The system always, always validates its own assumptions.

Why Google’s Approach Might Not Always Be a Good Idea.

Consider that until very recently, established institutional authorities classified transgender identity as a psychiatric disorder. For example:

The American Psychiatric Association included “Gender Identity Disorder” in earlier editions of the Diagnostic and Statistical Manual of Mental Disorders (DSM-III, DSM-IV).
Even the World Health Organization (!!!) classified “transsexualism” as a mental disorder in the ICD until 2019, when it was moved out of the mental disorders chapter in ICD-11.

Sure, those classifications are now widely regarded as wrong or harmful, including by the very institutions that once promoted them. But the correction did not originate from top-down institutional consensus. Under Google’s YMYL framework, search results during that period would have overwhelmingly surfaced:

Established medical institutions
Official diagnostic manuals
Hospital systems and government health agencies

Lived experiences from individuals, clinicians reporting mismatches between diagnosis and lived reality, early dissenting researchers — would have been algorithmically deprioritized by design.

Not censored. Just plain buried.

Anyway, according to Google, truth, in high-stakes domains, flows from recognized institutions downward. This approach is often defensible — and sometimes necessary — but it also narrows the epistemic frame.

Reality, according to Google, is what institutions say it is.

Bing: Has A Severe Monoculture Problem.

For most queries (especially technical and YMYL) Bing behaves like a consensus-finding machine instead of a search engine.

Bing’s sources have worryingly low domain diversity.

Bing has the lowest domain diversity of any engine in the dataset, averaging just 6.18 unique domains per query. That concentration becomes absurdly, comically extreme in technical searches, where 43.82% of Bing’s results come from StackOverflow alone, as well as in YMYL queries where 31.73% of results link to Mayo Clinic, with a small set of authoritative sources repeatedly resurfacing. Commercial results, similarly, always redirect to official manufacturer pages rather than any online store.

Bing is just selecting upstream arbiters and then amplifying their voices.

Where Google internally uses “Which institutions can absorb liability for this?” before answering, Bing internally considers something like:

“Which sources already function as de facto answer-oracles for this topic?”

In technical domains, that oracle is Stack Overflow. In health, it’s Mayo Clinic. In ecosystems it overlaps with, it’s Microsoft documentation — and, interestingly, even Google’s own docs when those have become canonical.

Once a source has crossed some internal threshold of:

recognizability
historical correctness
low controversy
repeat citation elsewhere

…Bing appears to treat it as an answer sink. Queries collapse into references to the same few nodes.

Like Google, this stance is understandable, if nothing else. Bing optimizes for reliability and predictability. Consensus sources are less likely to be wrong in obvious ways, less likely to trigger controversy, and easier to defend as “reasonable” choices, et cetera, et cetera. But unlike Google, which draws its authority boundary around established institutions, Bing draws it around delegated arbiters — sources that have already been socially anointed as places where “the answer” lives.

The cost is monoculture.

When a single source accounts for nearly half of all results in a category, it becomes a single point of epistemic failure. Take that StackOverflow result for example. StackOverflow is invaluable — but it is also gameable, shaped by moderator norms/community culture (read: policing), and biased toward certain voices and problem framings.

Its answers reflect StackOverflow’s community power structures just as much as technical correctness. Bing’s heavy reliance on it turns those biases into infrastructure.

Bing’s worldview is conservative and convergent. According to Bing, truth is what most experts already agree on. It is a safe engine — but again, one that rarely surprises, and rarely surfaces the odd, insightful edge case.

DuckDuckGo: Privacy Focused, But Aggregator Heavy.

DuckDuckGo presents itself as the anti-Google: private, independent, and user-first. Its information signature though, tells a more nuanced story.

DuckDuckGo’s sources, for the most part, mirror Google’s — except they have an aggregator spam problem.

In YMYL queries, ~63% of DuckDuckGo’s results come from .gov, .edu, or .org domains — lower than Google’s ~70%, but still a strong institutional majority. In the top three positions, institutional sources remain dominant, though with slightly more room for commercial and mixed-authority content than Google allows.

Where DuckDuckGo diverges is how it (frequently) tolerates aggregators. In YMYL finance queries for example, SmartAsset appears 13 times — more than any domain on any other engine. Similar patterns appear with comparison and lead-generation sites across categories. Bing loves listicles and aggregator spam.

Take this single YMYL finance query. It’s a super boring YMYL question: vague, high-stakes, and normative. There is no single correct answer — only frameworks, heuristics, and trade-offs. That ambiguity makes it a useful probe.

“best investment strategy for retirement”

On Google, this query immediately collapses into institutional authority. The top results are dominated by Fidelity, Vanguard, Merrill Lynch, the U.S. Department of Labor — large, regulated entities offering compliance-safe frameworks rather than any advice. The results page reads less like an answer to a question and more like some sort of syllabus on investment. 😅

{
"title": "Investing in Retirement: 5 Tips for Managing Your Portfolio | Merrill Lynch",
"source": "ml.com",
"description": "Merrill Lynch highlights diversification as a key strategy for retirement, noting that combining stocks and bonds helps balance long-term growth and risk protection across market conditions.",
"search_string": "best investment strategy for retirement",
"search_engine": "google"
},

{
"title": "Building Retirement Income Strategies | Fidelity",
"source": "fidelity.com",
"description": "Fidelity outlines four retirement income strategy models: interest and dividends only, investment portfolio only, portfolio plus guarantees, and short-term strategies, offering retirees flexibility based on risk tolerance and income needs.",
"search_string": "best investment strategy for retirement",
"search_engine": "google"
},
{
"title": "Guide to saving for retirement - Vanguard",
"source": "investor.vanguard.com",
"description": "Vanguard provides a step-by-step retirement preparation plan including estimating expenses, selecting accounts, investing, maximizing contributions, and adjusting strategies over time to align with life goals.",
"search_string": "best investment strategy for retirement",
"search_engine": "google"
},
{
"title": "Put your savings in different types of investments | U.S. Department of Labor",
"source": "dol.gov",
"description": "The U.S. Department of Labor emphasizes diversification as a way to reduce risk and improve returns by spreading investments across different asset types, such as stocks, bonds, and real estate.",
"search_string": "best investment strategy for retirement",
"search_engine": "google"
},
{
"title": "Learn how to secure your future with the best retirement investments | Nuveen",
"source": "nuveen.com",
"description": "Nuveen offers guidance on selecting retirement strategies through a variety of accounts and funds, helping individuals find a tailored investment approach that fits their financial goals and risk profile.",
"search_string": "best investment strategy for retirement",
"search_engine": "google"
}

Run the same query on DuckDuckGo, and now instead of institutions, we are saturated with aggregators and listicles. SmartAsset dominates the Top 5, alongside debt relief sites, comparison blogs, and SEO-optimized “top strategies” roundups. These pages are ultra-commercial, prescriptive, and aggressively optimized for conversion.

{
"title": "10 Retirement Strategies You Need to Know",
"source": "smartasset.com",
"description": "From tax-advantaged accounts to annuities, there are several retirement strategies to consider when planning. Learn more here.",
"search_string": "best investment strategy for retirement",
"search_engine": "duckduckgo"
},
{
"title": "Smart Retirement Investments: Strategies to Consider in 2025",
"source": "nationaldebtrelief.com",
"description": "Discover the best investment strategies for retirement in 2025, including top retirement investment ideas and tips to secure your financial future.",
"search_string": "best investment strategy for retirement",
"search_engine": "duckduckgo"
},
{
"title": "Top 11 Retirement Strategies",
"source": "smartasset.com",
"description": "Learn about retirement strategies including tax-advantaged accounts, annuities, and more to help plan for a secure future.",
"search_string": "best investment strategy for retirement",
"search_engine": "duckduckgo"
},
{
"title": "10 Retirement Strategies You Need to Know",
"source": "smartasset.com",
"description": "From tax-advantaged accounts to annuities, there are several retirement strategies to consider when planning. Learn more here.",
"search_string": "best investment strategy for retirement",
"search_engine": "duckduckgo"
},
{
"title": "10 Retirement Strategies You Need to Know",
"source": "smartasset.com",
"description": "From tax-advantaged accounts to annuities, there are several retirement strategies to consider when planning. Learn more here.",
"search_string": "best investment strategy for retirement",
"search_engine": "duckduckgo"
},

Yes, all those Smartasset dot com results are different articles!

This is a paradox and a half, to be honest. 😅 An engine designed to minimize tracking exposure ends up routing users through heavily commercialized middlemen. The likely explanation is probably that they rely a lot on upstream indices and ranking signals, and just…don’t have the budget/infra to throw at a Google-tier editorial suppression.

Ultimately, DuckDuckGo values privacy and avoids heavy-handed intervention — but that restraint allows monetization to leak into sensitive queries.

Yandex: “Chaotic Neutral” in Search Engine Form.

If Google curates for institutional authority and Bing for the safe picks, Yandex curates for maximum exposure. At whatever cost.

Yandex surfaces some truly out-there domains, dangerously so.

Yandex surfaces the highest domain diversity of all engines, averaging 11.69 unique domains per query, and includes 164 domains that appear on no other platform. It is the most pluralistic — and the least filtered — search engine in the dataset.

Consider YMYL-adjacent queries like:

“401k withdrawal rules”
“best running shoes”
“top rated wireless earbuds”

In each case, Yandex surfaces results hosted on opaque CloudFront subdomains — long, random-looking URLs that clearly do not represent publishers, brands, or even stable sites. These pages mimic legitimate listicles (“Tested & Rated,” “Best of 2025”) but provide no clear authorship, organization, or accountability. Like I found a CloudFront-hosted page that effectively impersonated SmartAsset content without being SmartAsset at all.

[  
  {  
    "title": "401(k) Tax Rules: Withdrawals, Deductions & More",  
    "source": "dr5dymrsxhdzh.cloudfront.net",  
    "description": "Unlike traditional 401(k) plans, Roth 401(k) accounts are funded with post-tax contributions, which means withdrawals can be taken tax-free if certain conditions are met… SmartAsset: 401(k) Tax Rules on Withdrawals, Deductions & More.",  
    "search_string": "401k withdrawal rules",  
    "search_engine": "yandex"  
  },  
  {  
    "title": "10 Best Running Shoes of 2025 | Tested & Rated",  
    "source": "d1nymbkeomeoqg.cloudfront.net",  
    "description": "A great pair of running shoes brings with it the promise of a new day, a fresh run, and a better you, no matter what is happening in the world at large.",  
    "search_string": "best running shoes",  
    "search_engine": "yandex"  
  },  
  {  
    "title": "Best Wireless Earbuds of 2025 | Tested & Rated",  
    "source": "djd1xqjx2kdnv.cloudfront.net",  
    "description": "Wireless earbuds-the ear tip seal on the vibe beam is good, allowing a more immersive…",  
    "search_string": "top rated wireless earbuds",  
    "search_engine": "yandex"  
  }  
]

Google and Bing suppress this class of result almost entirely. Yandex does not.

The same permissiveness appears throughout its YMYL handling. In health queries, Medium posts, lifestyle blogs, YouTube videos, and institutional medical sources routinely show alongside trusted authorities. In finance, LinkedIn posts from entirely random users appear alongside Forbes and Investopedia. In commerce, scammers pretending to be a totally different website, affiliate spam, thin comparison pages, and reputable reviews coexist with minimal differentiation.

This is not accidental. It’s a totally coherent — if extreme — philosophy.

Yandex treats the web as inherently chaotic and contested. It does not even try to decide which sources are legitimate, which voices are authoritative, or which domains are accountable. It’s designed to maximize exposure and variety, and leave evaluation entirely to the user. Authority is just… optional.

The upside of this approach is real. Yandex is exceptionally good at surfacing:

new emerging perspectives/early-stage knowledge
unofficial but insightful explanations
non-canonical voices (not always a good thing)

The downside is equally real, and…I’m not sure the tradeoff is worth it? By refusing to draw hard boundaries around authorship, institutional responsibility, or even basic site legitimacy, Yandex allows misinformation, impersonation, scams, and noise to blend seamlessly into the same epistemic surface as genuine expertise.

This is chaotic neutral in its purest form. 😅 Yandex does not protect the user from bad information, nor does it meaningfully privilege good information. It assumes a high-competence, high-skepticism user — and silently punishes anyone who lacks those priors.

In a search ecosystem increasingly defined by guardrails and liability management, Yandex stands apart as the wild west by design. You almost have to admire it.

3. Search Engines Vary Wildly at Figuring Out What The User Meant

Ranking differences are easy to see. Intent inference differences are harder — because they happen before ranking begins.

The clearest way to surface them is to hold the query constant and observe how each engine silently answers a different question.

Query: “should I move abroad?”

This isn’t a factual lookup, but a decision under uncertainty.

Google treats it as a decision-support problem, so you get cost-of-living comparisons, healthcare access, visa rules, and lived-experience discussions. Google assumes the user wants help weighing trade-offs.
Bing interprets the same query as an informational lookup. It surfaces immigration portals, official statistics, and formal descriptions of process. Bing assumes the user wants to understand the rules, not make the choice.
DuckDuckGo treats it as a research aggregation task, surfacing blog posts and “pros and cons” articles without strong framing. It assumes exploration without guidance.
Yandex treats it as open discourse, surfacing personal narratives, YouTube explainers, and expat forums. It assumes the user wants to see the argument, not the answer.

Same query, but with four different inferred goals. The pattern repeats across very different intents.

For “how to learn Spanish”, Google assumes the user wants a structured plan, Bing assumes understanding precedes execution, DuckDuckGo assumes tool discovery, and Yandex assumes learning happens socially and visually.

For “is intermittent fasting safe”, Google frames risk conservatively, Bing collapses onto a narrow institutional consensus, DuckDuckGo allows mixed advice, and Yandex surfaces disagreement directly.

What this reveals

Intent inference is less about correctness, more about what kind of answer the system believes the user deserves.

Before ranking begins, each engine quietly decides:

Is this a decision or a lookup?
Is subjectivity acceptable?
Should disagreement be surfaced or suppressed?
Is safety more important than exploration?

Those choices differ by platform — and they shape the user’s reality far more than ranking tweaks ever could. Even if you could somehow optimize for search engine bias, deciding what the question is for is a decision made before ranking algorithms kick in that is equally important.

4. YouTube as a Source: Yay or Nay?

How a search engine treats YouTube is also a reliable signal of how it defines knowledge itself. Across identical queries, engines don’t just rank video differently — they disagree on whether video is a legitimate way of knowing, at all.

Percentage of all 1630 search results containing youtube.com as a source, by search engine.

Take this query, for example.

Query: “react useEffect cleanup”

This is a technical + procedural question.

Google surfaces YouTube sparingly and late, after official documentation and written explanations. Video is supplementary — useful for intuition, not authority.
Bing shows a similar distribution to Google.
DuckDuckGo didn’t have any YouTube videos in the results themselves.
Yandex places YouTube front and center. Video tutorials routinely appear alongside — or ahead of — written documentation.

Same query, different epistemologies.

Query: “how to learn Spanish”

Now the task is experiential.

Google and Bing still emphasize structured programs and institutional guides, with video as an aid. DuckDuckGo shows nothing (they do contain a different Video field, though, but that’s not part of the search results themselves.) Yandex, by contrast, treats YouTube as the primary learning surface! Immersion videos, informal teachers, and community-driven instruction dominate.

Surprisingly, this continues to be a thing even in higher-risk queries.

Query: “is intermittent fasting safe”

Google and Bing absolutely suppress YouTube here in favor of institutional medical sources. Yandex though, continues to include YouTube prominently in the results, even alongside clinics and health authorities.

Again — a policy difference, not a ranking algorithm difference.

For Google/Bing, YouTube is treated as a didactic aid. But for Yandex, it’s apparently experiential knowledge.

Yandex seems to be making a call here on whether demonstration counts as explanation, whether personality and persuasion are acceptable components of understanding, and whether authority must be textual to be trusted.

Breakdown of Youtube.com listed as a source, by category of search query.

For many modern users — developers, learners, non-native speakers — YouTube is how knowledge is acquired (“visual learning” etc.) Engines that demote video implicitly privilege certain learning styles and literacies. Engines that surface more YouTube videos accept higher risk in exchange for accessibility.

Once again, the difference is not about ranking quality. It’s about what kinds of knowing are allowed to count.

5. Consensus vs Fragmentation: Just How Shared Is the Web?

One of the most striking findings in this analysis is not how differently search engines rank results — but how little they agree on which sources matter at all.

Across every query category, the overlap between Google, Bing, DuckDuckGo, and Yandex is vanishingly small. Only ~2–3% of domains appear in results from all four engines, regardless of category. In practical terms, that means fewer than three out of every hundred sources are treated as universally relevant across the modern search ecosystem.

The rest of the web is fragmented.

Depending on category, between 68.6% and 91.3% of domains appear on only one engine. Commercial: 87.68% unique, Technical: 81.05% unique, YMYL: 91.33% unique, Wildcard: 68.57% unique.

Measured by category, the percentage of domains shared across all four engines is:

Commercial queries: 1.9%
Technical queries: 1.96%
YMYL queries: 2.89%
Wildcard queries: 3.33%

Even in the best case — open-ended wildcard searches — over 96% of domains are not shared across engines. There is no category in which a meaningful consensus emerges.

YMYL queries — where one might expect the strongest agreement — are actually the most fragmented of all. While engines like Google and Bing enforce strong institutional filters, DuckDuckGo and Yandex allow a much broader — and sometimes riskier — set of sources. The result is not a shared reality with minor ranking differences, but fundamentally different epistemic environments.

This is just a core, structural property of modern search.

If there were such a thing as a canonical “top of the web,” we would expect to see convergence: a stable core of sources that every engine agrees are authoritative, especially in high-stakes domains. Instead, the data shows the opposite.

Each engine constructs its own sourcing universe, with only a thin sliver of shared ground. What qualifies as “authoritative,” “relevant,” or even “acceptable” varies dramatically depending on which engine you use.

Two users asking the same health or finance question on different engines are not being guided toward the same pool of knowledge. They are being pointed at different universes of sources.

💡 This fragmentation does not mean that search engines are broken or irrational. It just means they are optimizing for different objectives:- Legal defensibility- Risk tolerance- Diversity vs safety

Consensus vs exploration

Regional and infrastructural constraintsEach objective prunes the web differently, and our overlap statistics show that these pruning strategies rarely agree.

That’s all the results. Thank you for reading! If you want to know about my Methodology, read on.

Methodology

All observations in this analysis are based on direct SERP retrieval, not third-party summaries or clickstream data.

Queries were issued programmatically to four search engines — Google, Bing, DuckDuckGo, and Yandex — using a Node script that fetches raw results pages via a SERP API (Bright Data here, the one I had access to. Docs here.) Visualizations were generated later, via D3.js.

SERP API - SERP Scraper API - Free Trial

D3 - The JavaScript library for bespoke data visualization

The script does not simulate user interaction, personalization, or logged-in sessions. Each engine is queried with the same plain-text search string.

const path = require('path');
require('dotenv').config({ path: path.join(__dirname, '../../.env') });

const fetch = require('node-fetch');
const fs = require('fs-extra');

// you need to sign up at bright data to get these values
// https://brightdata.com/cp/setting/users
const CONFIG = {
  apiToken: process.env.BRIGHT_DATA_API_TOKEN,
  zone: process.env.BRIGHT_DATA_ZONE || 'serp_api1',
  maxResults: parseInt(process.env.MAX_RESULTS_PER_ENGINE) || 10,
  apiUrl: 'https://api.brightdata.com/request'
};

// search engine config
// as of Jan 2026, Bright data supports Google, Bing, DuckDuckGo, Yandex, Baidu, Yahoo, & Naver
const ENGINES = {
  google: {
    name: 'Google',
    buildUrl: (query) => `https://www.google.com/search?q=${encodeURIComponent(query)}`
  },
  bing: {
    name: 'Bing',
    buildUrl: (query) => `https://www.bing.com/search?q=${encodeURIComponent(query)}`
  },
  duckduckgo: {
    name: 'DuckDuckGo',
    buildUrl: (query) => `https://duckduckgo.com/?q=${encodeURIComponent(query)}`
  },
  yandex: {
    name: 'Yandex',
    buildUrl: (query) => `https://www.yandex.com/search/?text=${encodeURIComponent(query)}`
  }
};

// default query categories - all 40 queries from queries.js
const DEFAULT_QUERIES = [
  // commercial (10)
  { query: "best password manager 2025", category: "commercial" },
  { query: "buy iphone 15", category: "commercial" },
  { query: "best running shoes", category: "commercial" },
  { query: "cheapest web hosting", category: "commercial" },
  { query: "top rated mattress", category: "commercial" },
  { query: "best credit card for travel", category: "commercial" },
  { query: "cheap flights to europe", category: "commercial" },
  { query: "best laptop for programming", category: "commercial" },
  { query: "top rated wireless earbuds", category: "commercial" },
  { query: "best car insurance rates", category: "commercial" },
  // technical (10)
  { query: "react useEffect cleanup", category: "technical" },
  { query: "postgresql connection pooling", category: "technical" },
  { query: "nodejs memory leak debugging", category: "technical" },
  { query: "tailwind vs vanilla css", category: "technical" },
  { query: "docker compose volumes", category: "technical" },
  { query: "kubernetes pod restart policy", category: "technical" },
  { query: "javascript async await best practices", category: "technical" },
  { query: "git rebase vs merge", category: "technical" },
  { query: "typescript generic constraints", category: "technical" },
  { query: "redis cache invalidation strategies", category: "technical" },
  // ymyl (10)
  { query: "is coffee bad for you", category: "ymyl" },
  { query: "climate change causes", category: "ymyl" },
  { query: "covid vaccine side effects", category: "ymyl" },
  { query: "401k withdrawal rules", category: "ymyl" },
  { query: "adhd symptoms adults", category: "ymyl" },
  { query: "how to lower cholesterol naturally", category: "ymyl" },
  { query: "social security benefits calculator", category: "ymyl" },
  { query: "is intermittent fasting safe", category: "ymyl" },
  { query: "best investment strategy for retirement", category: "ymyl" },
  { query: "symptoms of diabetes type 2", category: "ymyl" },
  // wildcard (10)
  { query: "pizza restaurants chicago", category: "wildcard" },
  { query: "weather in london", category: "wildcard" },
  { query: "ukraine russia conflict timeline", category: "wildcard" },
  { query: "how to start a business", category: "wildcard" },
  { query: "best cities to live in 2025", category: "wildcard" },
  { query: "what is artificial intelligence", category: "wildcard" },
  { query: "how to learn spanish", category: "wildcard" },
  { query: "most popular video games 2025", category: "wildcard" },
  { query: "best new artists 2025", category: "wildcard" },
  { query: "top box office movies 2025", category: "wildcard" }
];

// data directory for saving results
const DATA_DIR = path.join(__dirname, 'data');
const RAW_DIR = path.join(DATA_DIR, 'raw');

// ensure data directory exists
async function setup() {
  await fs.ensureDir(RAW_DIR);
}

// fetch results from a single search engine
async function fetchEngine(engineKey, query) {
  const engine = ENGINES[engineKey];
  if (!engine) {
    throw new Error(`Unknown engine: ${engineKey}`);
  }

  const searchUrl = engine.buildUrl(query);
  const isGoogle = engineKey === 'google';

  console.log(`  [${engine.name}] Fetching: ${query}`);

  const requestBody = {
    zone: CONFIG.zone,
    url: searchUrl
  };

  if (isGoogle) {
    requestBody.format = 'json';
    requestBody.data_format = 'parsed_light';
  } else {
    requestBody.format = 'raw';
    requestBody.data_format = 'markdown';
  }

  try {
    const response = await fetch(CONFIG.apiUrl, {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${CONFIG.apiToken}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(requestBody)
    });

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}: ${response.statusText}`);
    }

    let data;
    if (isGoogle) {
      data = await response.json();
      console.log(`  [${engine.name}] Retrieved JSON data`);
    } else {
      const markdown = await response.text();
      const sizeKB = (markdown.length / 1024).toFixed(2);
      console.log(`  [${engine.name}] Retrieved markdown (${sizeKB} KB)`);
      data = markdown;
    }

    return {
      engine: engineKey,
      engineName: engine.name,
      query: query,
      data: data,
      format: isGoogle ? 'json' : 'markdown',
      timestamp: new Date().toISOString()
    };

  } catch (error) {
    console.error(`  [${engine.name}] ERROR: ${error.message}`);
    return {
      engine: engineKey,
      engineName: engine.name,
      query: query,
      data: null,
      format: null,
      error: error.message,
      timestamp: new Date().toISOString()
    };
  }
}

// fetch results from all search engines in parallel
async function fetchAllEngines(query) {
  console.log(`\n[Querying all engines for: "${query}"]`);
  console.log('-'.repeat(60));

  const engineKeys = Object.keys(ENGINES);
  const promises = engineKeys.map(engineKey => fetchEngine(engineKey, query));
  const results = await Promise.all(promises);

  const successCount = results.filter(r => r.data !== null).length;
  console.log(`\n[Successfully retrieved data from ${successCount}/${results.length} engines]`);

  return results;
}

// save results from all engines to files
async function saveResults(query, category, engineResults) {
  const timestamp = Date.now();
  const querySlug = query.replace(/[^a-z0-9]/gi, '_').toLowerCase();

  // save each engine's results to a separate file
  for (const result of engineResults) {
    if (result.data) {
      const extension = result.format === 'json' ? 'json' : 'md';
      const filename = `${result.engine}-${querySlug}-${timestamp}.${extension}`;
      const filepath = path.join(RAW_DIR, filename);

      if (result.format === 'json') {
        await fs.writeJson(filepath, result.data, { spaces: 2 });
      } else {
        await fs.writeFile(filepath, result.data, 'utf8');
      }

      console.log(`  [Saved ${result.engineName} results]`);
    }
  }

  // save metadata file with query info
  const metadataFile = `metadata-${querySlug}-${timestamp}.json`;
  const metadataPath = path.join(RAW_DIR, metadataFile);
  await fs.writeJson(metadataPath, {
    query,
    category,
    timestamp: new Date().toISOString(),
    engines: engineResults.map(r => ({
      engine: r.engine,
      engineName: r.engineName,
      hasData: r.data !== null,
      error: r.error || null
    }))
  }, { spaces: 2 });

  console.log(`  [Saved metadata]`);
}

// main function
async function main() {
  console.log('='.repeat(60));
  console.log('Search Engine Comparison Tool');
  console.log('='.repeat(60));

  // get queries from command line or use defaults
  const cliQueries = process.argv.slice(2);
  let queries;

  if (cliQueries.length > 0) {
    // use command-line queries
    queries = cliQueries.map(q => ({ query: q, category: 'custom' }));
    console.log(`Queries: ${cliQueries.join(', ')}`);
  } else {
    // use default categorized queries
    queries = DEFAULT_QUERIES;
    console.log(`Total queries: ${queries.length}`);
  }

  console.log(`Engines: Google, Bing, DuckDuckGo, Yandex`);
  console.log(`Max results per engine: ${CONFIG.maxResults}`);
  console.log('='.repeat(60));

  // check for API token
  if (!CONFIG.apiToken) {
    console.error('ERROR: BRIGHT_DATA_API_TOKEN not found in environment');
    process.exit(1);
  }

  // setup directories
  await setup();

  // process each query
  for (let i = 0; i < queries.length; i++) {
    const { query, category } = queries[i];

    console.log(`\n[${i + 1}/${queries.length}] Processing: "${query}" [${category}]`);
    console.log('-'.repeat(60));

    try {
      // fetch results from all engines
      const engineResults = await fetchAllEngines(query);

      // save results to files
      await saveResults(query, category, engineResults);

      const successCount = engineResults.filter(r => r.data !== null).length;
      console.log(`\n[Completed: ${successCount}/${engineResults.length} engines succeeded]`);

      // wait between queries (except for last one)
      if (i < queries.length - 1) {
        console.log('Waiting 2 seconds...\n');
        await new Promise(resolve => setTimeout(resolve, 2000));
      }

    } catch (error) {
      console.error(`\n[Error processing "${query}":`, error.message);
    }
  }

  console.log('\n' + '='.repeat(60));
  console.log('All queries completed!');
  console.log(`Results saved in: ${RAW_DIR}`);
  console.log('='.repeat(60));
}

// run if called directly
if (require.main === module) {
  main().catch(error => {
    console.error('FATAL ERROR:', error);
    process.exit(1);
  });
}

module.exports = { main, CONFIG };

Breaking this down, here’s what I’m doing:

1.Define a fixed set of queries, grouped into broad categories:

commercial (e.g. product comparisons)
technical (e.g. programming questions)
YMYL (health, finance, climate)
wildcard / everyday queries

2. Build native search URLs for each engine:

Google (https://www.google.com/search?q=something)
Bing (https://www.bing.com/search?q=something)
DuckDuckGo (https://duckduckgo.com/?q=something)
Yandex (https://www.yandex.com/search/?text=something)

3. Fetch SERPs directly, without rendering or post-processing:

Google results are requested in auto-parsed JSON form (via Bright Data SERP API’s parser)
Bing, DuckDuckGo, and Yandex are retrieved as raw or near-raw markup (Bing does apparently support JSON parsing, but I couldn’t get it to work)
No attempt is made to normalize rankings across engines. Way beyond the scope of what I wanted to do here.

4. Store each engine’s response separately, along with metadata. I analyzed them afterward by extracting titles, sources, search string used, and search engine used. Dataviz was pretty standard D3.js stuff, afterwards.

What this method captures — and what it doesn’t

This approach captures what each engine is willing to surface by default, given the same query and no user context. It is well-suited to studying:

domain diversity
repetition and amplification
tolerance for aggregators, blogs, and spam
relative privileging of institutions vs. individuals

It does not capture:

personalization effects
geographic fine-tuning beyond the proxy’s exit region
click behavior or downstream recommendations
subtle UI elements like knowledge panels or answer boxes

That is intentional. The goal here is not to model the entire user experience, but to compare personality defaults: what each engine treats as acceptable answers when no additional signals are provided.

In that sense, this methodology reflects the engines’ baseline worldviews — how they behave when forced to make decisions about authority, safety, and plurality on their own.

Opinion: There is No Singular “The Web” Anymore

This analysis wasn’t about correctness, bias accusations, or SEO gamesmanship.

What this data makes clear is that “the web” is no longer a single, shared informational substrate. It has fractured at the search-engine layer. If you have access to some sort of SERP infra, all of this is easily reproducible.

Search engines don’t merely reflect reality; they select which parts of reality are visible at all. When only ~2–3% of sources survive that selection process across platforms, the idea of a neutral, universal information commons stops making sense.

Instead, we live in parallel informational worlds:

Google’s institutional web
Bing’s consensus web
DuckDuckGo’s lightly curated web
Yandex’s wild west, maximalist web

The question “what does the internet say?” no longer has a single answer. It depends entirely on where you look.

Building a Fault-Tolerant Web Data Ingestion Pipeline with Effect-TS

Prithwish Nath — Tue, 20 Jan 2026 08:17:46 +0000

TL;DR — Silent failures break data pipelines. This post shows how Effect-TS enables typed errors, safe resource management, declarative retry logic, and composable pipelines to build predictable, fault-tolerant web data ingestion systems at scale.

It annoys me to no end that production web data pipelines rarely fail catastrophically. Instead, batch jobs “succeed” with incomplete data — silently corrupting downstream analytics, triggering retry storms that lead to IP bans, or letting one bad edge case crash a large nightly job.

I’m currently rebuilding the web data ingestion pipeline I’m responsible for at work: aggregation and analysis from 100+ upstream sources daily, hundreds of items per batch, with strict consistency requirements. Over time, I stopped trying to paper over failures with more logging + more retries, and started looking for a way to make system behavior explicit and easier to reason about.

That search eventually led me to Effect (formerly, Effect-TS) — a TypeScript effect system for modeling side effects, failures, and resource lifecycles directly in the type system.

Effect didn’t make my life easier in the sense of “fewer lines of code.” What it changed was how I thought about failure in TypeScript systems. Instead of treating network errors, rate limits, and partial responses as things to catch and move on from, Effect pushes you to model these failure modes explicitly and decide ahead of time how the system should respond.

Reliability engineering isn’t about building systems that never fail. It’s about building systems where failure is expected, understood, and bounded — so it doesn’t cascade into larger outages or silent data corruption.

In this post, I’ll walk through what that style of reliability engineering looks like in practice: using Effect-TS with typed errors, resource management, and declarative retries to build a fault-tolerant web data ingestion pipeline whose behavior is predictable under real-world failure.

All of the code in this post lives in a public Effect-TS web scraping and data ingestion repository on GitHub:

https://github.com/sixthextinction/effect-ts-scraping

What is Effect-TS?

At a practical level, Effect lets you describe work without running it yet.

An Effect value represents an operation that might perform I/O, might fail, and might depend on some environment…but none of that happens until you explicitly run it.

The crucial thing to realize is that Effect doesn’t just describe what the operation does. It also encodes what the operation produces on success, what it can fail with, and, optionally, what it depends on.

And ALL of that information lives in the TypeScript type system.

This might not sound like a big deal at first, but it changes when decisions get made — and turns out, that matters a lot.

In a typical TypeScript codebase, a data-fetching function looks like this:

async function fetchHtml(url: string): Promise<string> {  
  const res = await fetch(url);  

  if (!res.ok) {  
    throw new Error(`Request failed: ${res.status}`);  
  }  

  return await res.text();  
}  

// and then...  
const promise = fetchHtml("https://example.com");

What most people don’t realize is that Promises in JavaScript are eager. As soon as that line runs, the request has already started. Even if you never await the promise, the network request is already in flight, side effects have already happened, and yes — failures may already be occurring.

Now compare that to an Effect-based version:

// first define the error  
class NetworkError extends Data.TaggedError('NetworkError')<{  
  url: string;  
}> {}  

// then, do this.  
const fetchHtml = (url: string): Effect.Effect<string, NetworkError, never> =>  
  Effect.tryPromise({  
    try: async () => {  
      const res = await fetch(url);  
      if (!res.ok) {  
        throw new Error();  
      }  
      return await res.text();  
    },  
    catch: () => new NetworkError({ url }),  
  });

Effect is lazy, not eager. With Effect, just doing const effect = fetchHtml(“https://example.com"); does nothing. It’s simply data, returning a description of a computation. Nothing runs until you explicitly say so, by calling a runner like this:

Effect.runPromise(fetchHtml("https://example.com"));

Because the work doesn’t start until you run that, turns out you can still alter how it should behave — retries, timeouts, cancellations and more, attached before execution, not bolted on afterward.

Instead of discovering failure modes at runtime (or more likely, encoding them in comments/conventions) you’re forced to confront them at design time.

const program = pipe(  
  fetchHtml("https://example.com"),  
  Effect.retry(retryPolicy), // add retry logic  
  Effect.timeout("10 seconds") // add exponential backoff  
 // anything else  
);  

// still nothing has run  

// until….  
Effect.runPromise(program);

That’s why Effect works so well for hostile I/O like data ingestion. You’re deciding ahead of time how the system behaves when failures do happen. And with it, cross-cutting concerns (retries, rate limits, cleanup) can go on top without refactoring core logic.

Also, look at the Effect version’s type — Effect<string, NetworkError> — this is a machine-checkable contract that tells you, precisely:

this operation performs effects
it produces a string on success
it can fail with NetworkError
it CANNOT fail with any other expected error

Compare that to the vanilla TS type signature ((url: string) => Promise<string>), you cannot tell:

what errors might be thrown
whether they’re retryable
whether this is safe to call multiple times
whether this does I/O or just compute

All of that information exists only in comments, conventions, or someone’s head (or you only find out by running it and reacting.)

All this is why Effect feels like the TypeScript framework that you didn’t know you needed.

How Effect Changed How I Design for Failure

When the mental model of Effect clicked for me, I knew that if I can describe behavior before anything runs, then I’m not just deciding what happens on success, I’m deciding how the operation behaves under every condition. That includes failures obviously, but it also includes retries, slowdowns, and backpressure.

That’s where my thinking about data ingestion started to change. Most failures in a data ingestion pipeline are expected. None of them behave like typical fix-and-forget bugs:

networks are slow or unavailable
upstream APIs rate-limit you
data formats change without notice
some batches succeed while others fail

What’s different about these failures is that they’re maddeningly partial, and often. A job can succeed just enough to look healthy while quietly producing incomplete/stale data.

That’s not a correctness problem so much as a reliability problem. Once I started using Effect more deliberately, I noticed that it actually pushed me away from reacting to failures after the fact, and toward making those decisions up front.
So instead of adding another retry or another catch, while designing I had to decide:

What kinds of failures do I expect to see in production?
Which of these should be retried, and which shouldn’t?
When should the pipeline slow down instead of pushing harder?
When is failing fast the correct behavior?*

*This one is slightly debatable, but lets throw it in there because it’s an adjacent problem anyway

Because these questions were now part of the TypeScript type system itself, those decisions end up close to the code that triggers them. There’s less room for “we’ll handle it later” logic that never quite materializes because Effect forces the conversation.

Designing a Web Data Pipeline with Effect

The first concrete step was obvious: I needed to enumerate what actually breaks in my pipeline, and decide how each case should behave.

So I sat down and reduced my Puppeteer-based ingestion pipeline down to its real failure modes:

Network timeouts. Transient. These should be retried with backoff.
Rate limits. Expected. These require slowing down.
IP blocks. Fatal without proxy rotation; but with the right infrastructure (as was my case), just another retryable case.
CAPTCHAs. Not a logic problem. For me, this is handled entirely by the proxy layer, and is also retryable without any code on my part.
Schema changes. The site changed and selectors broke. This isn’t transient — it’s a logic error and should fail fast.

Traditional error handling lumps all of these into “something went wrong, throw an exception.” Effect lets you model them as distinct failure types, which means you can build infrastructure that handles them systematically. And that’s exactly where we’re going to start.

For reference, here’s the code for the full pipeline: https://github.com/sixthextinction/effect-ts-scraping/blob/main/full-pipeline.ts

Failure as a First-Class Concept (Tagged Errors)

The first thing I do is write down every failure mode I expect to see, and give each one a name. Each of these errors represents something meaningfully different from an operational perspective.

class NetworkError extends Data.TaggedError('NetworkError')<{  
  message: string;  
  url: string;  
  cause?: unknown;  
}> {}  

class TimeoutError extends Data.TaggedError('TimeoutError')<{  
  message: string;  
  url: string;  
  timeout: number;  
}> {}  

class RateLimitError extends Data.TaggedError('RateLimitError')<{  
  message: string;  
  url: string;  
  retryAfter?: number;  
}> {}  

class IPBlockError extends Data.TaggedError('IPBlockError')<{  
  message: string;  
  url: string;  
  proxyId?: string;  
}> {}  

class ParseError extends Data.TaggedError('ParseError')<{  
  message: string;  
  cause?: unknown;  
}> {}

What’s this Data.TaggedError? That’s something Effect provides us. Basically, it’s a premade error class that automatically gets a _tag field — a string literal that acts as a discriminant.

This _tag field gives us type-safe error handling. TypeScript can distinguish between different error types at compile time, and you can use functions like Effect.catchTag to handle specific errors without losing type information.

You technically can do this with vanilla TypeScript (discriminated unions), but it’ll be a pain. Yes, you can catch generic Error objects and use instanceof checks — but TypeScript can’t always narrow them correctly. Effect’s tagged errors give you precise type narrowing. When you catch a NetworkError, for example, TypeScript 100% knows it has a url property. When you catch a RateLimitError, TypeScript knows it might have a retryAfter property. This makes error handling both type-safe and composable (not to mention 500% less annoying to write code for. 😅)

Anyway, when modeling complex pipelines, this has to be your first step because once you have typed errors, you can decide which ones are retryable and which aren’t:

const retryableErrors = [  
  'NetworkError',  
  'TimeoutError',   
  'RateLimitError',  
  'IPBlockError',  
  'BrowserError',  
] as const;  

const isRetryableError = (error: ScrapingError): boolean =>  
  retryableErrors.includes(error._tag as any);

So a ParseError means your HTML selectors broke. That’s not a network problem so retrying won’t help. But a TimeoutError is when you retry with backoff.

Browser Logic — Side Effects Without the Pain

My pipeline uses Puppeteer to handle dynamic/JS based websites.

For this, we’ll use the Effect interface. (this Effect is a type within the Effect-TS/effect library we’re working with) Instead of letting Puppeteer leak browser state all over the codebase, everything related to it lives in these Effects.

The Effect interface is the quintessential part of the Effect-TS library — a description of a workflow or operation that is lazily executed. Here's what it looks like:

Effect<Success, Error, Requirements>

Where Success represents the type of what is returned on a success, Error represents the same for an error, and Requirements represents the type of required dependencies that you need to pass.

We’ve talked about the main difference before — unlike Promises, Effects are lazy. They don’t run until executed. This opens up a lot of opportunities for us for composition, cancellation, and resource management.

Using Effect, the lowest-level operation we need is…simply launching a browser. Makes sense that this should be our Step 1:

// STEP 1: Actually launch the browser  

// tryPromise converts a Promise-returning function into an Effect  
// Errors are caught and converted to typed errors (BrowserError)  
const launchBrowser = (  
  proxyConfig: ProxyConfig  
): Effect.Effect<Browser, BrowserError, never> =>  
  Effect.tryPromise({  
    try: async () => {  
      process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0'; // disable SSL validation for Bright Data proxy  
      return await puppeteer.launch({  
        headless: true,  
        ignoreHTTPSErrors: true, // ignore SSL certificate errors  
        args: [  
          '--no-sandbox',  
          '--disable-setuid-sandbox',  
          `--proxy-server=${proxyConfig.host}:${proxyConfig.port}`, // proxy host:port (credentials set via page.authenticate later)  
        ],  
      });  
    },  
    catch: (error: unknown) =>  
      new BrowserError({  
        message: 'Failed to launch browser with Bright Data proxy',  
        cause: error,  
      }),  
  });

The never in the Requirements position means the effect doesn’t require any external dependencies or context to run.

Effect.tryPromise will convert a Promise-returning function into an Effect (here, puppeteer.launch). Any thrown error gets mapped into a typed failure — since I explicitly use a catch function here, it will explicitly map it to an error of type BrowserError.

Your proxy config can live in a separate object like so. Using a proxy is technically optional, but I already had access to residential proxies and that handles the messy parts for me — fingerprinting, CAPTCHA solving, IP rotation, and geo-targeting — so in my pipeline, my Puppeteer instances behave like a real user instead of getting blocked immediately, with no extra code on my part.

Bright Data - All in One Platform for Proxies and Web Scraping

// Bright Data HTTP Proxy configuration (from env vars or .env file)  
// You'll get these values from your dashboard when you sign up  
const BRIGHT_DATA_CONFIG = {  
  customerId: process.env.BRIGHT_DATA_CUSTOMER_ID,  
  zone: process.env.BRIGHT_DATA_ZONE,  
  password: process.env.BRIGHT_DATA_PASSWORD,  
  proxyHost: 'brd.superproxy.io',  
  proxyPort: 33335,  
};  

// Validate configuration  
if (!BRIGHT_DATA_CONFIG.customerId || !BRIGHT_DATA_CONFIG.zone || !BRIGHT_DATA_CONFIG.password) {  
  throw new Error(  
    'Bright Data configuration missing. Set BRIGHT_DATA_CUSTOMER_ID, BRIGHT_DATA_ZONE, and BRIGHT_DATA_PASSWORD environment variables or add them to .env file'  
  );  
}  

interface ProxyConfig {  
  host: string;  
  port: number;  
  username: string;  
  password: string;  
}  

const buildProxyConfig = (): ProxyConfig => {  
  const username = `brd-customer-${BRIGHT_DATA_CONFIG.customerId}-zone-${BRIGHT_DATA_CONFIG.zone}`;  
  return {  
    host: BRIGHT_DATA_CONFIG.proxyHost,  
    port: BRIGHT_DATA_CONFIG.proxyPort,  
    username,  
    password: BRIGHT_DATA_CONFIG.password!,  
  };  
};

Those proxy config values are just the credentials you get when you set up a proxy to use.

With Puppeteer, you use proxies via page.authenticate(), which is where we’ll use this in the next step.

Alright, so as of now, we have a Puppeteer instance up and running. Next, we need navigation and content extraction. We’ll use Effect.acquireUseRelease to do this:

// STEP 2: Go to page, extract content.   

const navigatePageAndGetContent = ( browser: Browser, // this was returned as a result of what we did in Step 1  
  url: string, // the URL to go to  
  proxyConfig: ProxyConfig, // we already set this up earlier  
  timeout: number // use your own values in ms) =>  
  Effect.acquireUseRelease(  
    // acquire: create the page  
    // use: navigate and get content  
    // release: always close the page, even on error  
  );

This acquireUseRelease is Effect’s version of try / finally. You use it when describing real-world operations where you have to work with external resources (database connections, network stuff, etc.) that must be acquired, used properly, and released when no longer needed (even an error occurs).

It always involves a 3-step process. For us, this will involve:

Acquire: open a page in Puppeteer using a proxy that we authenticate
Use: navigate, check status codes, return HTML
Release: close the page, even if something failed

You don’t have to explicitly remember to do cleanup — the structure enforces it.

Let’s look at all of those steps in detail.

// STEP 2: Go to page, extract content.   

// Effect.acquireUseRelease manages resource lifecycle: acquire, use, and release  
// Ensures cleanup happens even if errors occur (like try/finally)  
// See: https://effect.website/docs/resource-management/introduction  
const navigatePageAndGetContent = (  
  browser: Browser,  
  url: string,  
  proxyConfig: ProxyConfig,   
  timeout: number = 10000   
): Effect.Effect<string, BrowserError | TimeoutError | IPBlockError | RateLimitError | NetworkError> =>  
  Effect.acquireUseRelease(  
    // STEP 2.1: acquire: create the page  
    Effect.tryPromise({  
      try: async () => {  
        const page = await browser.newPage();  
        await page.authenticate({ username: proxyConfig.username, password: proxyConfig.password }); // authenticate with Bright Data proxy  
        return page;  
      },  
      catch: (error: unknown) =>  
        new BrowserError({  
          message: 'Failed to create page or authenticate',  
          cause: error,  
        }),  
    }),  
    // STEP 2.2: use: navigate and get content  
    (page) =>  
      Effect.tryPromise({  
        try: async () => {  
          const response = await page.goto(url, {  
            waitUntil: 'networkidle2', // use 'load' if 'networkidle2' fails - proxies can have background requests that never stop  
            timeout: timeout    
          });  

          // check for HTTP errors that indicate blocks/rate limits  
          if (response) {  
            const status = response.status();  
            if (status === 429) {  
              throw new RateLimitError({  
                message: `Rate limited: ${url}`,  
                url,  
              });  
            }  
            if (status === 403) {  
              throw new IPBlockError({  
                message: `IP blocked: ${url}`,  
                url,  
              });  
            }  
            if (status >= 400) {  
              throw new NetworkError({  
                message: `HTTP error ${status}: ${url}`,  
                url,  
              });  
            }  
          }  

          return await page.content();  
        },  
        catch: (error: unknown) => {  
          if (error instanceof RateLimitError || error instanceof IPBlockError) {  
            return error;  
          }  
          if (error instanceof NetworkError) {  
            return error;  
          }  
          if (error instanceof Error && error.message.includes('timeout')) {  
            return new TimeoutError({  
              message: `Navigation timeout after ${timeout}ms`,  
              url,  
              timeout,  
            });  
          }  
          return new BrowserError({  
            message: 'Failed to navigate or get content',  
            cause: error,  
          });  
        },  
      }),  
    // STEP 2.3: release: always close the page, even on error  
    (page) =>  
      pipe(  
        Effect.tryPromise({  
          try: async () => await page.close(),  
          catch: () => new Error('Failed to close page'),  
        }),  
        Effect.catchAll(() => Effect.void) // ignore close errors  
      )  
  );

Effect's pipe (seen here in Step 2.3) composes functions left-to-right, passing the output of one as input to the next. It makes Effect operations readable instead of nested. So while reading, you start with Effect.tryPromise, then apply Effect.catchAll

Great, we now have a Puppeteer instance, and we can use it to go visit a page, extract the content we want, and close the page. Now we just have to bring it all together i.e. manage the browser lifecycle.

This should also be on an acquire → use → release cycle, this time on a browser level rather than a page level.

const scrapeUrl = (  
 url: string,  
 options?: { timeout?: number }  
 ): Effect.Effect<  
 string,  
 BrowserError | TimeoutError | IPBlockError | RateLimitError | NetworkError, never>  
=> {  
 const proxyConfig = buildProxyConfig();  
 const timeout = options?.timeout || 10000;  
// Bright Data automatically rotates IPs on each request,  
 // so retrying after an IP block gets a fresh IP  
 return Effect.acquireUseRelease(  
 // STEP 1: ACQUIRE -- launch browser  
 launchBrowser(proxyConfig),  

     // STEP 2: USE -- navigate and extract HTML  
(browser) =>  
  navigatePageAndGetContent(browser, url, proxyConfig, timeout),  

// STEP 3: RELEASE -- always clean up the browser  
(browser) =>  
  pipe(  
    Effect.tryPromise({  
      try: async () => await browser.close(),  
      catch: () => new Error('Failed to close browser'),  
    }),  
    Effect.catchAll(() => Effect.void)  
  )  
 );  
};

💡 In production usage you will usually also want to separate our fetch → parse → exit cycle into fetch → persist raw → parse → persist parsed, so you can debug raw HTML later, or parallelize parsing.

Again, each step is expressed as a function returning an Effect, and cleanup is guaranteed, even if navigation or retries fail.

At the end of this stage we have the basic Puppeteer loop: visiting a dynamic page, extracting its HTML, and cleaning up after ourselves.

But there’s more to do — namely, making all of the above work with parsing (business logic), and retry behavior + rate limiting (cross-cutting concerns.)

Retries That Understand Why Something Failed

Effect makes retry behavior declarative via its Schedule API.

// remember we defined which errors were retryable in step 1  
// here, first, we define HOW we should schedule retries…  

const retryPolicy = pipe(  
  Schedule.exponential(Duration.seconds(1)),  
  Schedule.intersect(Schedule.recurs(3))  
);

This says: “retry with exponential backoff starting at 1 second, up to 3 times, but only if the error is retryable.”

But the schedule alone isn’t enough. The system also needs to know which failures deserve a retry. Luckily, we already know those.

//… and then actually retry the ones which are retryable  
// Effect<A, E, R> is just the Effect ecosystem’s convention/shorthand for Effect<Success, Error, Requirements>  

const retryIfRetryable = <A, E extends ScrapingError, R>( effect: Effect.Effect<A, E, R>) =>  
  Effect.retry(effect, {  
    schedule: retryPolicy,  
    until: (error) => isRetryableError(error),  
  });

The until predicate is the key. Instead of retrying blindly, the system checks the error type and only retries when it makes sense. If you hit a ParseError (not in our list of retryable errors), the pipeline fails immediately — which makes sense, no point in hammering a broken CSS selector.

This is where our tagged errors from Step 1 pay off. The retry logic doesn’t inspect strings or guess intent. It operates entirely on types.

We’ll use retryIfRetryable at the very end, when we’re bringing all parts of our pipeline together.

Rate Limiting as a Declarative Policy

Rate limiting is what we call backpressure, rather than proper error handling. That is — you don’t want to wait until you get rate-limited to slow down, you want to prevent it in the first place.

For this tutorial, we can keep rate limiting intentionally boring here. Because even our rate limiting is an Effect (Effect.Effect), it composes cleanly with retries and resource management above.

const withSimpleRateLimit = <A, E, R>( effect: Effect.Effect<A, E, R>) =>  
  pipe(  
    Effect.sleep(Duration.millis(100)),  
    Effect.flatMap(() => effect)  
  );

This simply introduces a delay before the effect runs. Just like retries, we’ll use withSimpleRateLimit at the very end when composing the pipeline.

Of course, if you wanted to, you could go all out — you can build production grade rate limiting because Effect provides primitives like Ref (keep track of some form of state), Queue (lightweight in-memory queue), and Schedule (we used this in just the previous step.)

💡 I’m not going to go into detail on building a full rate limiter with Effect because that’s way too much cognitive load for just a blogpost. The point isn’t to build perfect throttling — it’s to show that backpressure can be a first-class part of the pipeline in Effect.

Parsing as Its Own Failure Domain

Finally, remember that while parsing logic is synchronous, it can still fail. We haven’t accounted for those yet.

This is our HTML parsing step to get the data we need. Instead of letting ParseError’s throw, it’s better if we wrap it in Effect.try — this is similar to Effect.tryPromise from earlier, but ONLY for synchronous functions that may throw (like cheerio parsing here.)

// This is just your scraping logic with selectors for the data you want  
// For this one we just get h1’s and spans.  

interface ScrapingResult {  
  title: string;  
  spans: string[];  
  url: string;  
}  

// Effect.try wraps synchronous logic that may throw  
// Errors are caught and converted to typed errors (ParseError)  
const parseHtml = (html: string): Effect.Effect<ScrapingResult, ParseError, never> =>  
  Effect.try({  
    try: () => {  
      const $ = cheerio.load(html);  
      const title = $('h1').text().trim();  
      const spans = $('span')  
        .map((_i: number, el: any) => $(el).text().trim())  
        .get()  
        .filter((s: string) => s.length > 0);  

      return {  
        title,  
        spans,  
        url: TARGET_URL,  
      };  
    },  
    catch: (error: unknown) =>  
      new ParseError({  
        message: 'Failed to parse HTML',  
        cause: error,  
      }),  
  });

This completes our error modeling — now parsing failures are distinct from network failures and both are handled properly. If parsing breaks, all our retries stop. That should be intentional — and so we made it so.

Composing the Pipeline

Up to this point, we’ve built individual pieces in isolation:

scrapeUrl knows how to fetch HTML safely
retryIfRetryable knows when to retry
withSimpleRateLimit enforces basic throttling
parseHtml turns raw HTML into structured data as per our domain logic (we provide the selectors we need)

Now we compose them into a single pipeline.

const scrapeWithRetry = (): Effect.Effect<  
  ScrapingResult,  
  ScrapingError  
> =>  
  pipe(  
    // Step 2: fetch HTML  
    scrapeUrl(TARGET_URL, { timeout: 30000 }),  

    // Step 4: apply rate limiting  
    withSimpleRateLimit,  

    // Step 3: retry transient failures  
    retryIfRetryable,  

    // Step 5: parse HTML  
    Effect.flatMap(parseHtml)  
  );

Read this top to bottom.

We start by fetching HTML with scrapeUrl (which instantiates and uses Puppeteer)
That operation is rate-limited
If it fails with a retryable error, it’s retried with backoff
If it succeeds, we move on to parsing
If parsing fails, the whole Effect fails immediately (as it should)

There are no callbacks here, no try/catch, and no manual error propagation. Control flow is handled by the Effect runtime.

Crucially, this function does not run anything yet. It only describes what should happen.

💡 I’ve skipped observability for this blog post, but in production, you should add more detailed logging, save retry metrics/failures, tracing per URL or proxy etc. Effect makes this very easy with its Logging APIs.

Executing the Pipeline

So far, we’ve built a description of a workflow. To actually execute it, we need to define what happens at the edges of the system (and this is where we finally use scrapeWithRetry.)

That’s what the final program does. This is again a pipe, so these happen sequentially:

const program = pipe(  
  scrapeWithRetry(),  

  // Log success  
  Effect.tap((result: ScrapingResult) =>  
    pipe(  
      Console.log('Scraping successful!'),  
      Effect.flatMap(() =>  
        Console.log(JSON.stringify(result, null, 2))  
      )  
    )  
  ),  

  // Handle all failures in one place  
  Effect.catchAll((error: ScrapingError) =>  
    pipe(  
      Console.error('Pipeline failed:', error),  
      Effect.flatMap(() =>  
        Effect.sync(() => process.exit(1))  
      )  
    )  
  )  
);

Finally, here’s how you run this.

// This is the entry point that ACTUALLY kicks off the entire pipeline  
Effect.runPromise(program).catch((error: unknown) => {  
  console.error('Unhandled error:', error);  
  process.exit(1);  
});

The .catch() wrapper here handles any truly unexpected errors that escape the Effect system (really shouldn’t happen, but it’s defensive programming).

This is the moment where everything becomes “real” — the browser is launched, requests are made, retries happen, resources are acquired and released, and logs are written.

Until this line runs, nothing has executed.

That mental separation — describing a workflow first, then running it explicitly — is one of the key reasons Effect works well for systems like this. You can reason about behavior before anything touches the network.

Why This Matters for Production Data Pipelines

So what did we build? Our pipeline has a few important properties:

Our errors are part of the type system and MUST be handled or intentionally propagated, and structured errors make any debugging or observability WAY easier
All of our resource lifecycles are enforced by construction
Our retry behavior is declarative, composable, and constrained by error types. In general, all cross-cutting concerns (retries, rate limits, cleanup) compose without refactoring core logic
All failure modes are explicit and discoverable at compile time
Any concurrency is safer by default, especially around shared resources

That structure is what makes this production pipeline evolvable. You can add ingestion sources, tune retries, adjust rate limits, or add observability without turning the code into a pile of special cases.

This system will always, always fail predictably, with enough context to debug what went wrong and why.

This kind of setup pays off when you’re scraping at scale, your failures have real business impact, and you need debuggability and auditability — something a team will maintain.

Most scraping tutorials stop at “how to fetch a lot of HTML without getting blocked.” That’s not the hard part. The hard part is building something you don’t have to babysit. Effect.ts gives you a way to model failure honestly, and a lot of readymade, first-class APIs to handle the parts application code built from scratch never should + you can add proxies to handle CAPTCHA/general unblocking. It’s a solid foundation to work off of.

It’s way more difficult, absolutely — Effect’s learning curve is more like a cliff wall — but it’s also way more reliable. And when a system runs unattended in production, that reliability is what actually matters.

Full source code: https://github.com/sixthextinction/effect-ts-scraping

A Beginner's Guide To Building Data Pipelines with Apache Arrow

Prithwish Nath — Tue, 20 Jan 2026 07:26:18 +0000

If you’ve ever built data pipelines for analytics, feature extraction, or model training, you’ve probably noticed a pattern: scraping or ingestion is rarely your bottleneck. It’s that the pipeline technically works, but compute and memory usage spike, and scaling the system becomes more expensive overall.

I’m betting the issue is not network I/O or parsing itself. It’s what happens after your data arrives.

Your pipeline probably relies on JSON as the interchange format between stages. Over time, data is parsed, transformed, cached, reloaded, and exported multiple times. Each step looks reasonable in isolation but taken together, they add up to a large (and unnecessary) cost.

Let’s talk about that cost, explain where it comes from, and show how Apache Arrow can be used to avoid it. We’ll build a small end-to-end pipeline, benchmark it against a traditional JSON approach, and look at the hard numbers directly.

🔥 Spoilers: With Arrow, on average: ~2.6x faster processing, ~84% less memory usage, ~98–99% lower storage and I/O costs — all adding up to a ~60% reduction in compute cost without even considering storage/bandwidth. Read on to know more.

What is Apache Arrow?

Apache Arrow is a columnar, in-memory data format that is open-source. Instead of representing data as a collection of rows (say, a list of dictionaries), it stores each column as a contiguous buffer of typed values.

Turns out, that design has a few important consequences:

Operations can work on entire columns at once
Memory layouts are predictable and cache-friendly
Many transformations can reuse existing buffers
Serialization to formats like Parquet avoids intermediate language-specific objects

The key point I’m trying to make is not that Arrow is “faster” in isolation, but that it changes the execution model. Once data is in Arrow format, most transformations no longer involve Python objects at all.

💡 FYI “Zero-copy” here doesn’t mean no memory is ever allocated. It means you avoid repeated parse/encode cycles and Python-level object creation — the dominant cost in traditional pipelines.

The Problem Isn’t “JSON vs. Arrow”

The JSON you get back from your API call is fundamentally just…text.

# This is what you actually get from the API  
api_response = '{"organic": [{"title": "Example", "position": 1}, ...]}'  
# Then you parse it, and...  
data = json.loads(api_response) # ...now it's a Python dict, not JSON anymore!  
# Your data is now:  
# {  
#  "organic": [  
#    {"title": "Example", "position": 1}, # Each result is a Python dict object  
#    {"title": "Another", "position": 2},  
#    # hundreds more  
#  ]  
# }

The moment you parse it, you’re no longer working with JSON, but with language-native object graphs. In Python, that means dictionaries, lists, strings, and integers allocated on the heap. From then on, every transformation operates on that object graph. Want to access the organic.title field? That’s actually a hashmap lookup operation under the hood.

💡 Python is uniquely bad at this (because dicts are hashtables with high overhead, every int/string/bool is a heap object, pointers everywhere, no JIT etc.), but this isn’t a problem limited to Python. Node.js (V8, with JIT) is much faster at object-heavy workloads for example, but once JSON is parsed, the data still becomes arrays of JavaScript objects processed one row at a time + each filter, sort, or map still allocates new arrays and performs property lookups. V8 makes this faster so you hit the wall much later, but no JIT or interpreter can escape this fundamental shape.

Instead, you should think about the root problem as Row-oriented object graphs vs Columnar memory structures.

With the former (how Python does it normally), each row is its own “container”, and each field is a separate object allocated on the heap (spread out randomly across system memory) connected by pointers. If your CPU wants to process that data, the runtime will need to enter a loop and repeatedly traverse back-and-forth to manipulate objects.

Apache Arrow simply sidesteps this problem. Instead of grouping values by row, it groups values by whole columns and stores them in typed, contiguous buffers — laid out sequentially in system memory. Each column buffer, directly, is the “container” now. Filtering, sorting, and aggregation directly operate on these system memory buffers.

So you’re moving computation out of Python’s object model entirely. Operating on a lower level in memory entirely vs. the Python abstraction is why we get the speed + storage gains we do.

Here are the typical workflow problems you solve by switching to Apache Arrow:

Problem 1: Per-Element Execution

A typical filtering op in Python will probably look like this:

filtered = [r for r in organic if r.get('position', 0) <= 10]

Super easy to understand, very straightforward and idiomatic. It is also 100% row-oriented, fundamentally:

One loop iteration per result
One dictionary lookup per row
One conditional branch per row
A new Python list allocated for the result

As inputs grow into the hundreds or thousands or millions, it will always become the dominant cost.

In contrast, Arrow expresses the same operation at the column level:

import pyarrow.compute as pc  

# Filtering with pyarrow - Python bindings for Arrow  

filtered = table.filter(pc.less_equal(table['position'], 10))

What Arrow does:

Actually runs optimized C++ code under the hood (via pyarrow, its Python library) to operate system memory directly
Operates on the entire column at once (vectorized)
Avoids Python’s interpreter entirely in the hot path
Returns a new table referencing existing buffers — you reuse memory automatically whenever possible.

Here, the comparison runs inside optimized native code (and the optimizations are ones you could never make operating solely at the Python level), operating on a contiguous buffer of values. Python is not involved in the inner loop. Ever.

Problem 2: You Convert More Than You Think

In real systems, data is rarely parsed once and discarded. You’re probably converting constantly between each stage of your pipeline without even realizing it:

# 1. Fetch from API = gives you a JSON string  
response = requests.get(url).text  
# 2. Parse to Python objects (CONVERSION #1)  
data = json.loads(response)  
# 3. Process data (Python dicts)  
filtered = [r for r in data if …]  
# 4. Save to cache or disk (CONVERSION #2)  
with open('cache.json', 'w') as f:  
json.dump(filtered, f)  
# 5. Later, read from cache (CONVERSION #3)  
with open('cache.json', 'r') as f:  
cached = json.load(f)  
# 6. Export to CSV or another format (CONVERSION #4)

Every json.loads() and json.dumps():

Parses or serializes the entire dataset
Allocates new Python objects
Goes back and visits every value again

This overhead compounds with batch size and iteration count. You’re paying that same Problem #1 cost repeatedly.

Problem 3: Memory Overhead

I’ve already said how JSON becomes Python objects once parsed. Just how bad is the problem in terms of space? Consider a simple row:

result = {  
  "title": "Example",  
  "position": 1,  
  "link": "https://..."  
}

In Python’s memory, this roughly becomes:

A dictionary with hash table overhead: ~240 bytes
Separate string objects for keys (title, position, link): ~150 bytes
Separate objects for each value: ~100+ bytes each
Pointers connecting everything together

Total: ~500–600 bytes per result.

Now, exact sizes will of course vary by Python version and workload, but overall, row-oriented object graphs are memory-dense and cache-unfriendly.

In Arrow, this row does not exist as a standalone object. There is no dictionary, no per-row container, and no per-field Python object:

One typed value in an integer buffer (position): ~4–8 bytes
One entry in a string offsets buffer per string field (title and link): ~4–8 bytes each
UTF-8 string bytes stored contiguously (amortized, no object headers)
Optional validity bits: ~1 bit per column

Total: ~150–200 bytes per result (depending mostly on string length)

The difference shows up quickly when you scale beyond toy data sizes.

What We’re Building

To demonstrate how Arrow serves our needs better, we’ll build a simple data pipeline that:

Fetches ~100 results in JSON from an external API (use anything you want that gives you data at scale; I’m going with Google SERP results)
Converts the JSON response directly to Arrow tables (one-time conversion cost)
Simulates a real production pipeline load (filtering, sorting, and aggregation) using Arrow-native operations
Exports to Parquet or CSV with minimal overhead

We’ll compare this against a JSON-based version that performs the same logical work, and measure both runtime and memory usage.

If you already have some data as structured JSON from whatever source, just start at Step 2.

Setting Up the Project

Install dependencies:

First of all, we’ll need PyArrow. That’s the official Python interface to the Apache Arrow columnar memory format + ecosystem.

The rest should be self explanatory.

pip install pyarrow requests python-dotenv

If you aren’t skipping Step 1, I’m using Bright Data’s SERP API to get JSON data at scale for this demo quick. For this, you’ll need to sign up, get these credentials, and put them in an .env file:

BRIGHT_DATA_CUSTOMER_ID=your_customer_id  
BRIGHT_DATA_ZONE=your_zone  
BRIGHT_DATA_PASSWORD=your_password

Recommended project structure (also optional, really):

project/  
├── src/  
│ ├── api_client.py # SERP API client  
│ ├── arrow_builder.py # JSON → Arrow conversion  
│ └── transformations.py # Arrow-native operations  
├── benchmarks/  
│ └── json_vs_arrow.py # Performance comparison

We’ll start by building the things we’ll need to eventually run the benchmark, starting with the API client.

Step 1: API Client to Fetch JSON Data

As mentioned before, skip this step if you already have some JSON.

Our API client will use our SERP API to fetch Google search results. Replace with your own API/implementation. All that matters is that you have a way of getting a lot of clean, structured JSON at scale.

SERP APIs are just ideal here because we’ll be ramping this up from ~100 to 1,000, 5,000, and even 10,000 results for the benchmark.

# src/api_client.py  

import os  
import requests  
from dotenv import load_dotenv  

load_dotenv()  


class  BrightDataClient:  
def  __init__(self):  
self.api_key = os.getenv("BRIGHT_DATA_API_KEY")  
self.zone = os.getenv("BRIGHT_DATA_ZONE")  

if  not self.api_key or  not self.zone:  
raise ValueError(  
"Missing BRIGHT_DATA_API_KEY or BRIGHT_DATA_ZONE. "  
"Set these in your .env file."  
)  

self.session = requests.Session()  
self.session.headers.update({  
'Authorization': f'Bearer {self.api_key}'  
})  
self.api_endpoint = "https://api.brightdata.com/request"  

def  search(self, query: str, num_results: int = 10):  
search_url = (  
f"https://www.google.com/search"  
f"?q={requests.utils.quote(query)}"  
f"&num={num_results}"  
f"&brd_json=1"  # returns Google search data as structured JSON instead of HTML  
)  

payload = {  
'zone': self.zone,  
'url': search_url,  
'format': 'json'  
}  

response = self.session.post(self.api_endpoint, json=payload, timeout=30)  
response.raise_for_status()  

result = response.json()  
# handle SERP API response format  
if  isinstance(result, dict) and  'body'  in result:  
body = result['body']  
if  isinstance(body, str):  
import json  
body = json.loads(body)  
return body if  isinstance(body, dict) else result  

return result

Right now this does nothing. We’ll instantiate and use this client to fetch data when we’re benchmarking the pipeline.

Step 2: Convert JSON to Arrow Tables

This is where you pay the one-time conversion cost.

Note that Arrow requires schemas. This is a feature, not a limitation. Schemas force you to be explicit about data shape, meaning better optimization + you catch bugs early.

# src/arrow_builder.py  

import pyarrow as pa  

def  serp_to_arrow(serp_data: dict):  
schema = pa.schema([  
pa.field('title', pa.string()),  
pa.field('link', pa.string()),  
pa.field('snippet', pa.string()),  
pa.field('position', pa.int32()),  
pa.field('display_link', pa.string()),  
pa.field('date', pa.string(), nullable=True),  
])  

organic_results = serp_data.get('organic', [])  

if  not organic_results:  
return pa.Table.from_pylist([], schema=schema)  

rows = []  
for idx, result in  enumerate(organic_results):  
row = {  
'title': result.get('title', ''),  
'link': result.get('link', ''),  
'snippet': result.get('snippet', ''),  
'position': result.get('position', idx + 1),  
'display_link': result.get('display_link', ''),  
'date': result.get('date', None),  
}  
rows.append(row)  

return pa.Table.from_pylist(rows, schema=schema)

Step 3: Arrow-Native Transformations

Now we can work with the data without re-serializing it. You can get as creative as you want here (and if you do, the Arrow cookbook has you covered) but to simulate a basic production workflow I’m considering three broad operations — filtering, sorting, and aggregation. That should cover most real-world use cases.

These all make heavy use of PyArrow’s Table class, and Compute functions.

# src/transformations.py  
# Simulates a typical production workflow with Arrow native transformations  
import pyarrow as pa  
import pyarrow.compute as pc  
from typing import  Optional, List  
from urllib.parse import urlparse  


# 1. filters SERP results by position (zero-copy operation)  
# Returns a filtered Arrow table  
def  filter_by_position(table: pa.Table, max_position: int = 10) -> pa.Table:  
if  'position'  not  in table.column_names:  
return table  

mask = pc.less_equal(table['position'], max_position)  
return table.filter(mask)  

# 2. Sort table by position (zero-copy operation)  
# Returns a sorted arrow table  
def  sort_by_position(table: pa.Table, ascending: bool = True) -> pa.Table:  
if  'position'  not  in table.column_names:  
return table  

sort_keys = [('position', 'ascending'  if ascending else  'descending')]  
return table.sort_by(sort_keys)  

# 3. Select specific columns (zero-copy operation)  
# Returns a table containing ONLY the selected columns  
def  select_columns(table: pa.Table, columns: List[str]) -> pa.Table:  
available_columns = [col for col in columns if col in table.column_names]  
if  not available_columns:  
return table  

return table.select(available_columns)  

# 4. Aggregate SERP results by domain using Arrow-native group_by operations  
# Returns an aggregated Arrow table with domain stats  
# This uses Arrow's native group_by which is zero-copy and much faster than Python loops for large datasets.  
def  aggregate_by_domain(table: pa.Table) -> pa.Table:  
if  'display_link'  not  in table.column_names and  'link'  not  in table.column_names:  
return table  

# extract domain using Arrow compute functions  
# we need to extract domains first, then group by them  
link_column = 'display_link'  if  'display_link'  in table.column_names else  'link'  

# extract domains - we still need Python for URL parsing, but minimize it  
# by using Arrow compute for string operations where possible  
domains_list = []  
positions_list = []  

# extract domains efficiently  
link_array = table[link_column]  
position_array = table['position'] if  'position'  in table.column_names else  None  

for i in  range(len(table)):  
url_str = link_array[i].as_py()  
if  not url_str or  not  isinstance(url_str, str) or  not url_str.startswith('http'):  
continue  

try:  
parsed = urlparse(url_str)  
domain = parsed.netloc  
if domain.startswith('www.'):  
domain = domain[4:]  
except Exception:  
if  '//'  in url_str:  
domain = url_str.split('//')[1].split('/')[0]  
if domain.startswith('www.'):  
domain = domain[4:]  
else:  
continue  

if domain:  
domains_list.append(domain)  
if position_array is  not  None:  
positions_list.append(position_array[i].as_py())  
else:  
positions_list.append(0)  

if  not domains_list:  
return pa.Table.from_pylist([])  

# create Arrow table with domains and positions  
domain_table = pa.Table.from_pydict({  
'domain': domains_list,  
'position': positions_list  
})  

# use Arrow's native group_by for aggregation (zero-copy)  
# aggregate returns a table with domain + aggregated columns  
# columns are returned in order: domain, position_count, position_mean, position_min, position_max  
grouped = domain_table.group_by('domain').aggregate([  
('position', 'count'),  
('position', 'mean'),  
('position', 'min'),  
('position', 'max')  
])  

# rename columns to match expected output format  
# group_by returns: domain, position_count, position_mean, position_min, position_max  
aggregated = grouped.rename_columns([  
'domain',  
'result_count',  
'avg_position',  
'min_position',  
'max_position'  
])  

return aggregated  

# 5. (Optional) If you want you can filter SERP results by domain name just as easily  
# And it's also a zero-copy operation.  
# This returns a filtered Arrow table:  
def  filter_by_domain(table: pa.Table, domains: List[str]) -> pa.Table:  
if  'link'  not  in table.column_names and  'serp_link'  not  in table.column_names:  
return table  

link_column = 'link'  if  'link'  in table.column_names else  'serp_link'  

# create mask for matching domains  
masks = []  
for domain in domains:  
domain_mask = pc.match_substring(table[link_column], domain)  
masks.append(domain_mask)  

# combine masks with OR  
if masks:  
combined_mask = masks[0]  
for mask in masks[1:]:  
combined_mask = pc.or_(combined_mask, mask)  
return table.filter(combined_mask)  

return table

All done, let’s put it all together, run it, and benchmark the difference.

Step 4: Benchmarking the Complete Pipeline

We’ll simulate a realistic workflow: fetch data from an API, filter it, sort it, and serialize it for storage — then deserialize and compute on it. This mirrors what happens when you cache intermediate results or pass data between workers.

We’re measuring the cost of repeatedly materializing, transforming, serializing, and re-parsing row-oriented Python objects vs. keeping data columnar (Arrow) and operating in native code (via pyarrow).

We’ll run 500 iterations to simulate batch processing:

import sys  
import time  
import json  
import tracemalloc  
import tempfile  
import os  
import argparse  
from pathlib import Path  
from datetime import datetime  

sys.path.insert(0, str(Path(__file__).parent.parent))  

from src.api_client import BrightDataClient  
from src.arrow_builder import serp_to_arrow  
from src.transformations import (  
filter_by_position,  
sort_by_position,  
aggregate_by_domain  
)  
import pyarrow as pa  
import pyarrow.compute as pc  
import pyarrow.parquet as pq  


def  json_processing(serp_data: dict, iterations: int = 100):  
"""traditional JSON-based processing pipeline  

We're simulating a typical data pipeline/workflow: filter -> sort -> cache (serialize) -> read (deserialize) -> compute  
Many real pipelines serialize data for caching, storage, or transmission between services  
"""  
organic = serp_data.get('organic', [])  

start = time.time() # to measure runtime  
tracemalloc.start() # to measure Python heap usage  

for _ in  range(iterations):  
filtered = [r for r in organic if r.get('position', 0) <= 10]  
sorted_data = sorted(filtered, key=lambda x: x.get('position', 0))  
# simulate caching/storage: serialize to JSON (expensive but common in real pipelines)  
json_str = json.dumps(sorted_data)  
# simulate reading cached data: deserialize back to Python objects (expensive but common)  
parsed = json.loads(json_str)  
total_positions = sum(item.get('position', 0) for item in parsed)  

elapsed = time.time() - start  
current, peak = tracemalloc.get_traced_memory()  
tracemalloc.stop()  

return elapsed, len(organic), peak / 1024 / 1024  


def  arrow_processing(serp_data: dict, iterations: int = 100):  
"""Arrow-based zero-copy processing pipeline  

We'll simulate the same operations as JSON version, but no serialization needed:  
Filter -> Sort -> Compute here ALL work directly with Arrow data  
If you want persistence, just export to Parquet (native via pyarrow, but not needed here for in-memory ops)  
"""  
# convert JSON to Arrow table - one-time cost, amortized across iterations  
table = serp_to_arrow(serp_data)  

start = time.time() # to measure runtime  
tracemalloc.start() # to measure Python heap usage  

for _ in  range(iterations):  
# zero-copy: filter directly on Arrow data (no Python conversion)  
filtered = filter_by_position(table, max_position=10)  
# zero-copy: sort directly on Arrow data (no Python conversion)  
sorted_table = sort_by_position(filtered, ascending=True)  
# zero-copy: compute directly on Arrow column (no serialization needed for in-memory ops)  
total_positions = pc.sum(sorted_table['position']).as_py()  

elapsed = time.time() - start  
current, peak = tracemalloc.get_traced_memory()  
tracemalloc.stop()  

return elapsed, len(table), peak / 1024 / 1024  


def  main():  
cache_dir = Path(__file__).parent / "cache"  
cache_dir.mkdir(exist_ok=True)  
cache_file = cache_dir / "benchmark_data.json"  

parser = argparse.ArgumentParser(description='Run JSON vs Arrow benchmark')  
parser.add_argument('--refresh', action='store_true', help='Force refresh cached data')  
args = parser.parse_args()  

serp_data = None  
num_results = 0  

# try to load cached API data to avoid re-fetching on every run  
if  not args.refresh and cache_file.exists():  
try:  
with  open(cache_file, 'r') as f:  
cached_data = json.load(f)  
serp_data = cached_data.get('serp_data')  
num_results = len(serp_data.get('organic', [])) if serp_data else  0  
except Exception:  
serp_data = None  

# fetch data if cache doesn't exist or is too small (< 100)  
if serp_data is  None  or num_results < 100:  
client = BrightDataClient()  

# if you want more results, increase results per query or just...run more queries!  
num_results_per_query = 10  
target_results = 100  

queries = [  
"Python data processing",  
"distributed computing",  
"data engineering",  
"ETL pipeline design",  
# add more as needed  
]  

all_results = []  
successful_queries = 0  

for query in queries:  
if  len(all_results) >= target_results:  
break  

try:  
serp_data = client.search(query, num_results=num_results_per_query)  
query_results = serp_data.get('organic', [])  
all_results.extend(query_results)  
successful_queries += 1  
time.sleep(0.5) # just in case, for rate limits  
except Exception:  
pass  

if all_results:  
serp_data = {'organic': all_results}  
num_results = len(all_results)  

# cache results for future runs  
try:  
cache_data = {  
'serp_data': serp_data,  
'timestamp': datetime.now().isoformat(),  
'num_results': num_results,  
'successful_queries': successful_queries  
}  
with  open(cache_file, 'w') as f:  
json.dump(cache_data, f, indent=2)  
except Exception:  
pass  
else:  
serp_data = client.search("Python data processing", num_results=100)  
num_results = len(serp_data.get('organic', []))  

# run benchmarks: simulate processing multiple batches (here, 500)  
iterations = 500  
json_time, json_rows, json_memory = json_processing(serp_data, iterations)  
arrow_time, arrow_rows, arrow_memory = arrow_processing(serp_data, iterations)  

# calculate performance improvements  
speedup = json_time / arrow_time if arrow_time > 0  else  0  
memory_reduction = ((json_memory - arrow_memory) / json_memory * 100) if json_memory > 0  else  0  

# compare file sizes: JSON vs Parquet  
table = serp_to_arrow(serp_data)  

with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as json_file:  
json.dump(serp_data.get('organic', []), json_file, indent=2)  
json_filepath = json_file.name  

json_size = os.path.getsize(json_filepath)  

with tempfile.NamedTemporaryFile(suffix='.parquet', delete=False) as parquet_file:  
parquet_filepath = parquet_file.name  

# Parquet is columnar and compressed, typically much smaller than JSON  
pq.write_table(table, parquet_filepath, compression='snappy')  
parquet_size = os.path.getsize(parquet_filepath)  

# Arrow's in-memory representation is also more efficient  
arrow_memory_size = table.nbytes  

size_reduction = ((json_size - parquet_size) / json_size * 100) if json_size > 0  else  0  
memory_reduction_vs_json = ((json_size - arrow_memory_size) / json_size * 100) if json_size > 0  else  0  

# Print out the results  
print(f"\n{'Metric':<25} {'JSON':<20} {'Arrow':<20} {'Improvement':<15}")  
print("-" * 70)  
print(f"{'Processing Time':<25} {json_time:.4f}s{'':<15} {arrow_time:.4f}s{'':<15} {speedup:.2f}x faster")  
print(f"{'Throughput':<25} {iterations/json_time:.1f} ops/s{'':<10} {iterations/arrow_time:.1f} ops/s{'':<10} {speedup:.2f}x more")  
print(f"{'Peak Memory':<25} {json_memory:.2f} MB{'':<13} {arrow_memory:.2f} MB{'':<13} {memory_reduction:.1f}% less")  
print(f"{'Data Size (JSON file)':<25} {json_size/1024:.2f} KB")  
print(f"{'Data Size (Parquet file)':<25} {parquet_size/1024:.2f} KB{'':<13} {size_reduction:.1f}% smaller")  
print(f"{'Data Size (Arrow in-memory)':<25} {arrow_memory_size/1024:.2f} KB{'':<13} {memory_reduction_vs_json:.1f}% smaller")  

try:  
os.unlink(json_filepath)  
os.unlink(parquet_filepath)  
except Exception:  
pass  

# All done, lets save benchmark results to disk  
results_dir = Path(__file__).parent.parent / "benchmarks" / "results"  
results_dir.mkdir(parents=True, exist_ok=True)  

timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")  
results_file = results_dir / f"benchmark_{timestamp}.json"  

results_data = {  
"timestamp": datetime.now().isoformat(),  
"dataset_size": num_results,  
"iterations": iterations,  
"json": {  
"time_seconds": json_time,  
"throughput_ops_per_sec": iterations / json_time,  
"peak_memory_mb": json_memory,  
"data_size_kb": json_size / 1024  
},  
"arrow": {  
"time_seconds": arrow_time,  
"throughput_ops_per_sec": iterations / arrow_time,  
"peak_memory_mb": arrow_memory,  
"data_size_kb": parquet_size / 1024,  
"in_memory_size_kb": arrow_memory_size / 1024  
},  
"improvements": {  
"speedup": speedup,  
"memory_reduction_percent": memory_reduction,  
"size_reduction_percent": size_reduction,  
"parquet_vs_json_reduction_percent": size_reduction  
}  
}  

with  open(results_file, 'w') as f:  
json.dump(results_data, f, indent=2)  


if __name__ == "__main__":  
main()

Optional: You can persist Arrow data as Parquet easily with pyarrow. Parquet is Arrow’s natural output format — both are columnar.

import pyarrow.parquet as pq  
from pathlib import Path  
def  export_to_parquet(table, filepath: str, compression: str = 'snappy'):  
    Path(filepath).parent.mkdir(parents=True, exist_ok=True)  
    pq.write_table(table, filepath, compression=compression)

Benchmark Results

So I ran that benchmark across dataset sizes from ~100 to 10,000 SERP results by altering num_results_per_query and target_results, then executed each logical pipeline 500 times per run to simulate repeated batch processing in a real data pipeline. How do the results scale?

Processing Time

On average (1,000–10,000 rows), Arrow is ~2.6x faster in processing time.

Press enter or click to view image in full size

Processing Time (in seconds) vs. Dataset size (number of results). Arrow remains a constant low latency throughout, but using JSON at each stage makes processing time scale with dataset size.

We can ignore the data for very small dataset sizes (that n=100 spike), because then the JSON pipeline appears MUCH slower than it really is due to fixed overheads in the JSON pipeline: repeated json.dumps() / json.loads() calls and Python object allocation dominate runtime.

Arrow’s processing time remains nearly constant (~0.085–0.091 seconds) throughout, so here’s how the Arrow speedup grows with dataset size:

~1.6x at 1,000 rows
~2–3x between 2,000 and 8,000 rows
~3.6x at 10,000 rows

This pattern is expected. The JSON pipeline scales processing time linearly with row count, The Arrow pipeline does not: filtering, sorting, and aggregation run inside native kernels over columnar buffers, so Python never enters the per-element hot path.

Throughput

On average (1,000–10,000 rows), Arrow delivers ~2.6x higher throughput.

Press enter or click to view image in full size

Throughput (operations/second) vs. Dataset size (number of results). Arrow maintains at least 5,000+ ops/sec across all dataset sizes.

Throughput mirrors processing time exactly.

Arrow maintains a relatively constant throughput of ~5,500–6,000 ops/sec across all dataset sizes. The JSON pipeline’s throughput degrades steadily as row count increases, dropping from ~3,600 ops/sec @ 1,000 rows to ~1,500 ops/sec @ 10,000 rows.

Memory Usage (Python Heap)

On average (1,000–10,000 rows), Arrow uses ~84% less Python heap memory.

Press enter or click to view image in full size

Peak Memory Usage (in MB). Pipelines that use Arrow use 80–100% less memory across all dataset sizes.

Peak memory usage, measured via tracemalloc, shows a consistent and substantial reduction when using Arrow (again, we can ignore the test case with n=103.) Bear in mind we’re capturing Python heap allocations only.

As dataset size increases, JSON memory usage remains tied to object churn, while Arrow’s Python-level memory stays low and stable — most data lives in native buffers outside the Python object model.

File Size (JSON vs Parquet)

On average (1,000–10,000 rows), Parquet files are ~98.7% smaller than JSON.

Press enter or click to view image in full size

File Size (in KB) vs. Dataset size (number of results). Arrow based pipelines compress persisted results dramatically better than pipelines which use JSON.

The most dramatic — but also the most boringly predictable difference.

Parquet files are consistently 97–99% smaller than their JSON equivalents at realistic dataset sizes. No surprises there, that’s exactly what columnar formats like Parquet were designed to do.

How Much Money Does This Save You?

Breaking this down:

~2.6x faster processing (average) = ~1/2.6th the compute time
~84% less Python heap memory = smaller instance sizes, less GC pressure
~98–99% smaller files = lower storage and I/O costs

So for a pipeline that processes, say, 10,000 queries/day, a JSON approach would use ~10,000 seconds of compute time while an Arrow approach would use ~3,800 seconds of compute time

Assuming $0.10/hour for compute, that’s ~$0.28/day vs ~$0.11/day — roughly a 60% reduction in compute cost, before even accounting for memory and storage savings. I’ll take that any day.

When Should You Use Apache Arrow?

Here’s the big caveat — you can’t just use Apache Arrow for everything. It is an architectural choice for production data pipelines rather than some general optimization hack.

Use an Arrow-based pipeline when:

You process dozens to thousands of rows per batch
Your pipeline applies multiple transformations (e.g. filter → sort → aggregate → export)
You’re building analytics, feature extraction, or training pipelines
Memory usage, throughput, or storage size matter
You might want to persist data in Parquet for downstream systems

Honestly though, from experience, most production data pipelines should find using Arrow a net improvement. In these cases, avoiding that repeated Python object materialization + serialization tax will dramatically improve how your pipeline scales (and how much it costs to operate.)

Keep your existing pipeline when:

You’re running one-off scripts on very small datasets
The end of your pipeline has to return JSON directly to another API consumer
Your data can’t be described by a schema
You aren’t transforming the data at all (the conversion cost won’t amortize)

Adopting Apache Arrow just won’t be worth the effort (or rewrite) in those cases.