How to Collect SERP Data: A Complete Guide to Scraping Google Search Results

For SEO rank tracking, competitor analysis, keyword research, or localized search analysis, you often need to collect Google search results at scale. Manual searches are inefficient, which is where SERP data collection comes in. This guide covers the key types of SERP data, common collection methods, and how to properly configure proxies and request parameters for reliable, efficient data collection at scale.

I. What Is SERP Data Collection?

SERP (Search Engine Results Page) data collection uses automated programs to extract search result data returned by Google and other search engines for specific keywords. SERP scraping lets you quickly obtain standardized Google search data for analysis across multiple dimensions.

• Data you can collect includes:
• Organic search results and rankings
• Page titles, URLs, and meta descriptions
• Featured Snippets
• People Also Ask (PAA)
• Related Searches
• Image/video results
• Special results such as Google Shopping
• SERP data from different countries and regions

Common use cases:

• SEO keyword rank tracking: Monitor target keyword rankings and changes at scale.
• Competitor SERP analysis: Identify which websites consistently rank at the top for target keywords.
• Keyword and content research: Discover new search demand from Related Searches, PAA, and other SERP features.
• Regional SERP analysis: Compare Google search results across markets such as the US, Japan, and the UK.
• Market and sentiment monitoring: Track the search visibility of specific brands, products, or industry trends.
• AI & LLM data feeds: Provide up-to-date external search context for RAG (Retrieval-Augmented Generation) systems or industry-specific LLMs.

II.What Are the Methods for SERP Data Collection?

The right method depends on your team’s technical capabilities and the scale of data collection. The table below provides a quick comparison:

Manual collection works only for checking a few keywords occasionally. SERP APIs eliminate the need to maintain proxies or parse pages, but costs increase with query volume. Python scrapers and browser automation are better suited to teams that need deeper customization and long-term collection. Whichever method you choose, proxies are essential once you start sending requests at scale.

III. How to Collect Google SERP Data

Using a self-built collection workflow as an example, here are five key steps.

Step 1: Define the SERP Data You Need to Collect

Before writing any code, decide which fields you need. A clear data schema makes storage and analysis much easier later. A common SERP data structure looks like this:

For example: Keyword → Rank → Title → URL → Description → SERP Features → Location → Date

In addition to organic results, you can store Featured Snippets, People Also Ask, and other SERP modules separately for later search intent and content gap analysis.

Step 2: Configure Anti-Blocking Proxies and Request Headers

This step is critical to successful SERP collection. Google is highly effective at detecting automated requests, and the following patterns can easily trigger risk controls:

• A single IP sending a large number of requests in a short period
• Abnormal request frequency
• A single IP querying many different keywords
• Geographic location not matching the target SERP
• Improper Session / Cookie management
• Automated request patterns that differ significantly from normal browsing behavior

Once triggered, you may receive HTTP 429 responses, CAPTCHAs, failed requests, or incomplete search results. Proxies help distribute requests across different IPs while also enabling SERP collection from different countries and regions.

High-quality residential proxies are the foundation of a stable setup. For example, by configuring IPFoxy rotating or static proxies in your scripts, you can build a high-quality pool of real IPs and distribute requests across hundreds of thousands of real nodes worldwide, reducing the risk of a single IP being blocked due to excessive request volume.

A simple example of configuring a proxy in Python:

import requests

proxies = {
    "http": "http://user:[e-mail]:8888",
    "https": "http://user:[e-mail]:8888"

}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...",
    "Accept-Language": "en-US,en;q=0.9",

}

response = requests.get("https://www.google.com/search?q=keyword", proxies=proxies, headers=headers)

In addition to proxies, keep these two points in mind:

• Geolocation testing: IPFoxy country- and city-level nodes let you collect SERP results for specific locations more accurately. For example, to track rankings in Japan, select a Japan node and set gl=jp.
• Request Header spoofing: Make sure each request uses realistic, randomized fingerprint information such as User-Agent, Accept-Language, and Sec-Ch-Ua, rather than sending identical headers every time.

Step 3: Build the Google Search Request

Google search parameters directly affect the returned results. Common parameters include:

• q: keyword
• gl: country/region (e.g., us, jp, uk)
• hl: language (e.g., en, ja)
• num: number of results per page
• start: pagination offset
• device: device type (desktop or mobile can be simulated through the User-Agent) For paginated collection, set start to 10 or 20 for the second page, depending on the num setting. Note that Google also limits pagination depth, and deeper pages may return duplicate or incomplete results.

Step 4: Fetch and Parse Search Results

You can fetch pages with Python Requests or browser automation tools, then parse the fields you need. Because Google now relies heavily on JavaScript rendering, tools such as Playwright are recommended; Requests may return a page without complete results.

Below is a simple Python SERP collection example using Playwright + a proxy:

import asyncio
import json
import random
from playwright.async_api import async_playwright, TimeoutError as PWTimeoutError

PROXY = {
    "server": "http://proxy.ipfoxy.com:8888",
    "username": "user",
    "password": "pass",
}

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    ...
]

SEARCH_URL = "https://www.google.com/search?q={q}&gl=us&hl=en&num=20"

EXTRACT_JS = """
() => {
  const items = [];
  document.querySelectorAll('div#search div.g, div#rso > div[data-hveid]').forEach(el => {
    const titleEl = el.querySelector('h3');
    if (!titleEl) return;
    const a = titleEl.closest('a');
    const link = a ? a.href : (el.querySelector('a[href^="http"]')?.href ?? '');
    const snippet = el.querySelector('div[data-sncf], div.VwiC3b, span.aCOpRe')?.innerText ?? '';
    items.push({title: titleEl.innerText.trim(), link, snippet: snippet.trim()});
  });
  return items;
}
"""

Adjust the selectors based on the actual DOM structure. If you use Requests, keep in mind that Google may return a page without complete results.

Step 5: Save and Organize SERP Data**
**
After collection, store the data in a structured format:

Lightweight analysis: Export regularly to CSV or Excel for easy comparison.

Long-term monitoring: Store data in MySQL, PostgreSQL, MongoDB, or another database. Create an index based on Keyword + Date + Location to build ranking trend charts.

**

IV. Common SERP Data Collection Issues

**

1. What should you do if SERP collection triggers a CAPTCHA?
Reduce the request rate, switch proxies, and make sure the Request Headers match a real browser. Residential proxies can significantly reduce the likelihood of CAPTCHAs. If a particular IP has already triggered one, do not keep forcing requests through it—switch to another IP.

2. What should you do if SERP collection returns a 429 error?
A 429 response means too many requests. Reduce concurrency, increase the interval between requests, and rotate IPs. Avoid querying many keywords from the same IP within a short period. You can also set a request quota for each IP and rotate automatically once the limit is reached.

3. Why do Google rankings differ across locations?
Google personalizes search results based on factors such as location, language, and device. To get rankings for a specific market, use a proxy in that region and set the corresponding gl and hl parameters. Otherwise, the rankings may reflect your local search perspective rather than the target market.

Conclusion

SERP data can be collected through SERP APIs, Python scrapers, or tools such as Playwright/Puppeteer. When collecting data across multiple keywords and regions, you also need to account for request rate limits, IP blocking, Session management, and geolocation.

If you need SERP data from different countries and regions, you can select IPFoxy proxy resources in the target markets to build an efficient and stable automated collection infrastructure.