How I tell a poisoned SERP from a ranking drop with plain Python

TLDR Google sometimes answers a search with a poisoned SERP, a second version of the results page that's on topic but holds a different set of sites. On the same set of non-local keywords over three days, every one of seven rank tracker APIs got it now and then, and so did two Chrome windows. The tracker rule that catches it is one comparison of two host sets, and its threshold comes from measuring how far two sources disagree on an ordinary day, with the seven APIs on the same 20 keywords next to two Chrome windows opened in the same minute. The responses and the comparison code are in a public repo, and everything below runs on the standard library.

The full comparison prices the APIs and walks through each one. This post is the measurement side, so the numbers can be rerun or argued with. HasData is our own product and one of the seven, and the only claim I make for it here is the row it gets in each table.

The rule that flags a poisoned page

I sent the same non-local keywords on three days to the seven APIs and to both Chrome windows, and counted a response as poisoned when a site in the API's top 3 was missing from the top 10 of both browsers. The poisoned page is on topic, its sites repeat from day to day, and it moves between providers, so a keyword that came back poisoned from one API in the morning matched Chrome an hour later. For "what is the tallest mountain in the world" two APIs opened their top three with a trekking company's page while neither Chrome window had it in its top 10, so that time the browsers held the other version. One page on its own doesn't show it. The previous run for the same keyword does.

The tracker in the full comparison stores the sorted hosts of every page next to the position, and on the next run compares the two sets. When they share fewer than half of the smaller set, the row is stored as suspect and gets one manual search before it goes into a report.

before = set(prev_hosts.split())
if len(hosts & before) * 2 < min(len(hosts), len(before)):
    status = "suspect"  # shares under half of the last list, recheck before reporting

Half is a measured threshold. The rest of this post is the measurement, which also says how far two trackers disagree on an ordinary day, so a flagged run is further from its own previous run than two different providers were from each other.

What the data looks like

Each response is one JSON file at data/raw/<pass>/<vendor>/<keyword>.json in the benchmark repo. The pass is country, austin or mobile, the vendor is one of hasdata, serpapi, brightdata, searchapi, scrapingdog, serper and scraperapi, and the two Chrome windows are stored as real_clean (signed out) and real_account (signed in to Google). The norm.organic list holds each organic result with pos, title, url and snippet, normalized to the same shape so the rest of the code doesn't care which API it came from.

import json
from pathlib import Path

RAW = Path("data/raw")
KEYWORDS = [f"k{i:02d}" for i in range(1, 21)]

def organic(block, vendor, kw):
    f = RAW / block / vendor / f"{kw}.json"
    if not f.exists():
        return []
    rec = json.loads(f.read_text(encoding="utf-8"))
    return (rec.get("norm") or {}).get("organic") or []

An empty list stands for a failed call, an empty page or a keyword the vendor didn't get, and every function below skips it instead of counting it as zero.

One list per page

Positions compare by site, so each URL is reduced to its host with www. dropped. A site that ranks twice on one page gets two entries, the second tagged with an occurrence number, so a page with two Yelp results doesn't collapse into one.

from collections import Counter
from urllib.parse import urlsplit

def host(o):
    h = o.get("host") or urlsplit(o.get("url", "")).netloc
    return h.lower().removeprefix("www.")

def keys(items, n=10):
    seen, out = Counter(), []
    for o in sorted(items, key=lambda o: o.get("pos") or 99)[:n]:
        h = host(o)
        if h:
            out.append((h, seen[h]))
            seen[h] += 1
    return out

The result for one page is a list of up to ten (host, occurrence) pairs in rank order, and that list is the only thing the comparisons below look at.

Did the city take

The first question for a local tracker is whether the location parameter changed anything. For the three local keywords, "pizza near me", "dentist" and "home depot hours", I counted how many of the top 10 results named Austin in the title, URL or snippet.

LOCAL = {"k11": "pizza near me", "k12": "dentist", "k17": "home depot hours"}

def austin_hits(items):
    top = items[:10]
    if not top:
        return None
    text = lambda o: " ".join(str(o.get(f, "")) for f in ("title", "url", "snippet")).lower()
    return sum("austin" in text(o) for o in top), len(top)

for vendor in ["hasdata", "serpapi", "brightdata", "searchapi", "serper", "scraperapi"]:
    row = [austin_hits(organic("austin", vendor, kw)) for kw in LOCAL]
    print(f"{vendor:<11}", "  ".join("empty" if r is None else f"{r[0]} of {r[1]}" for r in row))

For "dentist", HasData, SerpApi and Bright Data each returned nine results, and all nine named Austin. Serper and ScraperAPI returned ten and nine results, and none of them named Austin. Their pages were for Frederick, Maryland, and Beardstown, Illinois, which is what a US-wide request routed through a proxy in those towns looks like. SearchAPI applied the city on two of the three keywords. The same check on the country pass gives zero for every vendor, so the count measures the location parameter and nothing else.

How far a position moves between two APIs

Two trackers rarely agree, and the question is by how much. For each pair of APIs that applied the city I took every site in one API's Austin top 10, looked up its position at the other API for the same keyword, in both directions, and binned the difference.

BINS = ["same position", "off by 1", "off by 2 or more", "not in the other top 10"]

def moves(block, a, b, keywords=KEYWORDS):
    count = dict.fromkeys(BINS, 0)
    for kw in keywords:
        for x, y in ((a, b), (b, a)):
            t, u = keys(organic(block, x, kw)), keys(organic(block, y, kw))
            if not t or not u:
                continue
            for p, site in enumerate(t):
                if site not in u:
                    count[BINS[3]] += 1
                else:
                    count[BINS[min(abs(u.index(site) - p), 2)]] += 1
    total = sum(count.values())
    return {k: v / total for k, v in count.items()}, total

shares, sites = moves("austin", "brightdata", "hasdata")
print(sites, {k: f"{v:.0%}" for k, v in shares.items()})

Bright Data and HasData came out closest. Of 281 sites checked, 74% held the same position at both, 7% were off by one, 4% off by two or more and 15% were missing from the other top 10. HasData and SerpApi agreed on 54%, and the lowest pair, SearchAPI and SerpApi, on 21%, because SearchAPI's Austin pages mixed in other cities. Even that pair kept more than half of each other's top 10, which is where the tracker's threshold comes from. Run the same function on the country pass with real_account and real_clean, and the two Chrome windows agree on 16%, with 29% of sites off by one place. The browsers differ by a Google account and nothing else, so a move of one place between two days is inside what two live searches show at the same moment.

How much a top 10 keeps when the city or the device changes

The same keys() lists answer how stable one API is against itself. The overlap below is the share of a top 10 that survives in another top 10 for the same keyword and vendor, so a vendor's US-wide list against its own Austin list, or its desktop list against its mobile one. Vendors with fewer than five keywords in both passes are left out.

from statistics import mean

def overlap(t, x, k=10):
    t, x = t[:k], x[:k]
    return len(set(t) & set(x)) / min(k, len(t)) if t and x else None

def kept(vendor, a, b):
    vals = [overlap(keys(organic(a, vendor, kw)), keys(organic(b, vendor, kw))) for kw in KEYWORDS]
    vals = [v for v in vals if v is not None]
    return mean(vals) if len(vals) >= 5 else None

for vendor in ["hasdata", "brightdata", "serpapi", "searchapi", "serper", "scraperapi"]:
    city, mob = kept(vendor, "country", "austin"), kept(vendor, "country", "mobile")
    print(f"{vendor:<11} austin {city:.0%}  mobile {'no option' if mob is None else f'{mob:.0%}'}")

Bright Data kept 82% of its US-wide top 10 once Austin was set and HasData 74%, while SerpApi and SearchAPI replaced about half. Serper kept 85%, and that row reads well until the city check above shows both of its runs were US-wide pages, so it's an hour of plain drift with no location change. On mobile the five APIs with a device option kept between 38% (SearchAPI) and 78% (Bright Data) of their desktop top 10, and the mobile page also held fewer organic results, a median of eight to eight and a half against nine on desktop.

Running it yourself

The repo has scripts/summarize.py, which is these functions in one file and prints every table above from the raw responses, plus the request parameters each vendor got, with the keys removed. Clone it, run the script, and change the keyword list or the vendor list to see what moves. The article adds the response times, the plans that cover 1,000 keywords a day and the tracker script in full, which also stores the parsed count next to the position, since a short page and a lost ranking look the same without it.