I Built a Self-Healing Web Scraper That Refuses to Break

Dharmendra Kushwaha — Wed, 12 Aug 2026 13:59:40 +0000

Every developer who has worked with web scraping knows the pain: a website updates its UI, a CSS class name changes, and suddenly your entire data pipeline breaks.
To solve this, I’ve been building a Self-Healing Web Scraper in Python. Instead of failing silently when a website redesigns, it automatically adapts, heals itself, and continues extracting data.
--> The Problem
"Normal scrapers are incredibly fragile. They rely on rigid HTML structures and CSS selectors. The moment a website changes its layout, your scraper dies".
The Solution: A 5-Layer Fallback Pipeline
I designed an extraction engine that tries five different strategies in sequence. If one fails, it automatically falls back to the next:
1) JSON-LD Parsing: Looks for structured data (the most resilient method).
2) CSS Selectors: Traditional fast extraction if classes haven't changed.
3) Regex Fallback: Pattern matching on raw HTML when CSS fails.
4) Fuzzy Self-Healing: Scores nearby text blocks to find price symbols and product data as a last resort.
5) LLM / AI Self-Healing: The newest addition! If all standard methods fail, it uses an LLM (Google Gemini or offline Ollama) to "read" the broken HTML and extract the products anyway.

Self-Learning Memory
The scraper doesn't just survive; it learns. It remembers which extraction strategy worked best for a specific website and automatically prioritizes that method on the next run, saving execution time and API calls.
|--| Key Features

Multi-Site Price Comparison: Tested across Flipkart, Amazon, Robu.in, and more.
AI-Powered Resilience: Uses Google Gemini or 100% offline Ollama for intelligent data extraction.
Smart Garbage Filtering: Rejects junk data like menu items, ads, and accessories.
Auto Currency Conversion: Automatically normalizes prices to ₹ INR.
Dual Fetching & Stealth: Uses Playwright with stealth mode to bypass basic anti-bot detection.
Persistent Storage: Saves data to an SQLite database and exports clean, timestamped CSV files.

-->Watch it in Action
I recorded a quick demo showing how the scraper handles failures and triggers the AI fallback:
-->Watch the Demo Video Here
--> Check out the Code
I am open-sourcing this project and would love to get feedback from the community on architecture, edge cases, or feature requests!
🔗 GitHub Repository: https://github.com/uyg7x/Self-Healing-Web-Scraper
If you find this useful or interesting, I would highly appreciate a ⭐ on the repo and your thoughts in the comments below! How do you currently handle website redesigns in your scraping pipelines?

DEV Community: Dharmendra Kushwaha

I Built a Self-Healing Web Scraper That Refuses to Break