DEV Community

Cover image for Scrape google search results python github: Resilient 2026 guide
SerpApi.Org
SerpApi.Org

Posted on Originally published at serpapi.org

Scrape google search results python github: Resilient 2026 guide

Having spent years maintaining data extraction pipelines, I’ve learned a hard truth: if your web scraper relies on class names to parse search engine results, it is already broken. Google dynamically updates and randomizes its HTML layout selectors. If you clone an open-source Python scraper from GitHub, chances are it will throw null-pointer errors within weeks of deployment.

To build a truly resilient pipeline, you need to shift from scraping visual layers to targeting underlying structured data.

Bypassing CSS with JSON-LD Extraction

Instead of targeting fragile CSS selectors, target the raw structured data embedded directly within the page source. Google relies on structured JSON-LD schemas to populate its rich snippets and knowledge graphs. This metadata resides inside script tags of type application/ld+json.

Because this data feeds search engine engines, the keys stay consistent even when CSS classes change daily. Here is how I structure selector-free extraction in Python:

import json
from bs4 import BeautifulSoup

def extract_search_metadata(html_content):
    soup = BeautifulSoup(html_content, 'html.parser')
    scripts = soup.find_all('script', type='application/ld+json')
    extracted_data = []

    for script in scripts:
        try:
            data = json.loads(script.string)
            # Traverse the schema dictionary to find search result payloads
            if "itemListElement" in data:
                extracted_data.append(data["itemListElement"])
        except (ValueError, TypeError):
            continue

    return extracted_data
Enter fullscreen mode Exit fullscreen mode

This approach shifts your code's maintenance cycle from weekly emergency hotfixes to simple, bi-annual structure audits.

Simulating Human Behavior with Playwright

When dynamic search elements (like interactive maps or local listings) are required, static HTML parses are insufficient. I use Playwright to run headless browser sessions. However, modern anti-bot systems check for automated signatures.

To execute successfully:

  1. Remove Automation Indicators: Disable the navigator.webdriver flag inside your browser context.
  2. Implement Micro-Scrolls: Simulate human reading patterns by scrolling the page down in small, randomized pixel increments.
  3. De-synchronize Events: Introduce randomized delays between 800ms and 2400ms before clicks or viewport changes.

Overcoming IP Bans and CAPTCHAs

Running automation from standard hosting servers or GitHub Actions default runner IPs will trigger immediate blockages. Datacenter IP ranges are heavily monitored.

  • Utilize Rotating Residential Proxies: These route requests through real consumer ISPs, maintaining a 98%+ success rate.
  • Rotate User-Agents Dynamically: Match your rotated IPs with updated browser headers representing modern, active operating systems.
  • Build Exponential Backoff: If your pipeline encounters a 429 rate limit, dynamically double the execution delay before retrying with a fresh IP address.

Build vs. Buy: The Technical Trade-off

Building custom scraping infrastructure is a rewarding challenge, but scale changes the financial equation. When you calculate developer wages spent on maintaining code, premium residential proxy bandwidth, and running dynamic headless browsers, self-hosting can become incredibly expensive.

For mission-critical production data, leveraging a managed JSON endpoint like SerpApi is often the more efficient engineering decision. It completely abstracts browser automation, IP pool rotation, and DOM changes, allowing you to focus on processing data rather than acquiring it.

Originally published at Scrape google search results python github: Resilient 2026 guide

Top comments (0)