DEV Community

XSron Hou
XSron Hou

Posted on Originally published at scrapio.dev

How to Scrape Google Search Results with an API

#webscraping #api #ai #showdev

Originally posted on the Scrapio blog — sharing here too.

Google Search is one of the most valuable data sources on the internet, and one of the hardest to scrape reliably. Google detects and blocks bots aggressively — rate limits, CAPTCHAs, and IP bans make direct scraping fragile.

Scrapio routes search requests through a specialized proxy layer that mirrors real browser behavior. You get clean, structured results without any of the blocking.

What you'll need

  • A Scrapio API key on a Pro plan or higher (Google Search isn't available on Free or Starter)
  • curl or Python

Basic search request

The Google Search endpoint is a GET request — pass parameters as query strings:

curl "https://api.scrapio.dev/v1/google/search?search=best+web+scraping+api+2026" \
  -H "Authorization: Bearer sk-..."
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "organic_results": [
    {
      "position": 1,
      "title": "Top 10 Web Scraping APIs Compared (2026)",
      "url": "https://example.com/web-scraping-apis",
      "description": "We tested 10 web scraping APIs on speed, reliability, and price...",
      "domain": "example.com"
    }
  ],
  "questions": [
    "Is web scraping legal?",
    "What is the difference between scraping and crawling?",
    "How do I scrape Google without getting blocked?"
  ],
  "related_searches": [...]
}
Enter fullscreen mode Exit fullscreen mode

Paginate results 

import httpx

API_KEY = "sk-..."

def get_serp_page(query: str, page: int = 1) -> dict:
    return httpx.get(
        "https://api.scrapio.dev/v1/google/search",
        headers={"Authorization": f"Bearer {API_KEY}"},
        params={
            "search": query,
            "page": page,
        },
        timeout=30,
    ).json()

# Collect first 3 pages of results
all_organic = []
for page in range(1, 4):
    results = get_serp_page("web scraping api", page)
    all_organic.extend(results.get("organic_results", []))

print(f"Collected {len(all_organic)} results")
Enter fullscreen mode Exit fullscreen mode

Localize results

Pass country_code (ISO 3166-1 alpha-2) and language to get localized SERPs:

httpx.get(
    "https://api.scrapio.dev/v1/google/search",
    headers={"Authorization": f"Bearer {API_KEY}"},
    params={
        "search": "web scraping api",
        "country_code": "de",
        "language": "de",
    },
    timeout=30,
).json()
Enter fullscreen mode Exit fullscreen mode

Search type

Use search_type to query different Google verticals:

Value Description
classic Standard web search (default)
news Google News results
images Google Images
lens Google Lens (visual search)
ai_mode Google AI Mode results
ads Google Ads results

shopping and maps aren't supported — Google blocks them on every provider path this API has access to, so they're rejected at validation rather than silently failing.

params={"search": "wireless headphones", "search_type": "images"}
Enter fullscreen mode Exit fullscreen mode

Use cases

  • SEO rank tracking — monitor your keyword positions daily
  • Competitor research — see what's ranking for your target keywords
  • Market research — extract PAA boxes to understand user intent
  • Content strategy — find gaps in top-ranking pages

Next steps

Top comments (0)