Everyone building RAG or an LLM agent eventually wants the same thing: live search results, fed into the model. And everyone hits the same wall — the Google-only SERP scrapers keep dying on the /sorry CAPTCHA page the moment they run from a cloud IP.
I tested it directly: from datacenter and even residential cloud IPs, Google's CAPTCHA wall is a coin-flip. So the honest answer isn't "beat Google" — it's use an engine that actually returns data every run.
Bing is the pragmatic default
Bing returns organic results comparable to Google, is far more tolerant of automated access, and exposes the same fields you need: title, URL, snippet, position, related searches. So the SERP Scraper defaults to Bing, with DuckDuckGo as a second reliable engine and Google as best-effort (with an honest disclaimer, not a false promise).
One gotcha: residential IP personalization
Early on, a query for "best web scraping tools" returned… adult Reddit threads. The cause: residential proxy IPs are real users' connections, and their Bing session/personalization leaks into your results. The fix:
- clear cookies before each query,
- force
mkt(market) + SafeSearch, - don't carry any session cookie.
After that, results are clean and query-driven every time.
Input
{
"queries": ["apify web scraping platform"],
"engine": "bing",
"maxResults": 20,
"country": "us",
"language": "en"
}
You get one row per organic result:
{ "query": "apify web scraping platform", "engine": "bing", "position": 1,
"title": "Apify: The largest marketplace of trusted tools for AI", "url": "https://apify.com", "domain": "apify.com" }
Use it for
- RAG / LLM grounding — pair it with a URL-to-Markdown reader to build a full "search + read" web browser for your model.
- SEO / rank tracking — per country and language.
- Keyword research — harvest Related Searches.
Try it: SERP Scraper on Apify. No API key; public search-results data only.
Top comments (0)