Same product, two behaviours
Two search endpoints, same provider, different pagination:
- Google web search takes
page=1,2,3… - DuckDuckGo search returns a
cursoryou pass back for the next slice
Both cost 1 credit per request, and both count each page as a separate call. The difference is not pricing — it is what happens to your data when you paginate.
I have shipped pipelines on both, and the page-number one taught me a lesson about deduplication that cost me a week.
Page numbers assume the results are standing still
page=2 means "give me results 11 to 20 for this query". It does not mean "continue from where the first call ended".
Between call 1 and call 2, the results page can change. A result that was #8 can move to #4, another can drop out of the top 10. When you stitch page 1 and page 2 together, you get gaps and duplicates.
This is not a bug in the API. It is what pagination means when the underlying result set is live.
The cursor does not fix it, but it changes the failure mode
DuckDuckGo's cursor is an opaque token. You send it back and get the next slice.
You still get drift, because the index moves. What changes is the failure mode: with a cursor, the server decides where the next slice starts, so you are less likely to fetch overlapping ranges. You are more likely to get a consistent sequence, with occasional duplicates across slices.
The practical difference: page numbers can produce holes, cursors mostly produce repeats. Repeats you can dedupe. Holes you cannot.
A unified interface
If you use both, wrap them behind one function that returns rows plus a token:
import os
import requests
BASE = "https://api.serpbase.dev"
HEADERS = {
"Content-Type": "application/json",
"X-API-Key": os.getenv("SERPBASE_API_KEY", ""),
}
def fetch_page(engine: str, q: str, token=None, gl="us", hl="en"):
"""Returns (rows, next_token). token is a page number or a cursor."""
if engine == "google":
payload = {"q": q, "gl": gl, "hl": hl, "device": "pc", "page": token or 1}
path = "/google/search"
else:
payload = {"q": q, "gl": gl, "hl": hl}
if token:
payload["cursor"] = token
path = "/duckduckgo/search"
data = requests.post(f"{BASE}{path}", headers=HEADERS, json=payload, timeout=(5, 30)).json()
if data.get("status") != 0:
raise RuntimeError(f"status={data.get('status')} req={data.get('request_id')}")
rows = data.get("organic") or []
if engine == "google":
nxt = (token or 1) + 1
else:
nxt = data.get("cursor")
return rows, nxt
The caller does not care which model is underneath. It loops until next_token is empty, or until a page cap.
Dedupe on the URL, never on rank
The week I lost was spent deduping by rank.
Ranks shift between pages, so the same result showed up with two different ranks and survived the dedupe. Meanwhile two genuinely different pages that happened to land on the same rank in different slices got collapsed into one.
The key has to be the canonical URL: hostname + path, lowercase, query string stripped.
from urllib.parse import urlsplit
def key_of(link: str) -> str:
u = urlsplit(link or "")
return (u.hostname or "").lower() + u.path.rstrip("/")
Strip tracking parameters but keep the path. Different paths are different pages.
Both models get worse as you go deeper. Results past page 5 are rarely worth the credits: the intent match drops, and the drift grows.
I default to three pages for discovery, one page for daily monitoring. At 1 credit per request, a 200-keyword daily run at one page is 200 credits; at three pages it is 600, for very little extra signal.
New accounts come with 500 free credits and no card, which is enough to compare both pagination models on the same 20 queries before you commit to either.
Endpoint parameters and cursor behaviour are documented in the SerpBase docs; the catalog currently lists 49 endpoints, 44 available.
Top comments (0)