Signal: A Scraper that heals itself

SIGNAL: I Built a Web Scraper That Fixes Itself
Because nobody wants to manually debug broken scrapers at 2 AM.

The Problem Nobody Talks About
Web scraping is easy. Web scraping that stays working is hard.
Every developer who's built a scraper knows the drill: you write selectors, they work for a week, then a site updates its HTML and your scraper returns empty objects. Suddenly you're debugging $('.article-title') vs $('.post-title') at midnight, and your data pipeline is broken.

I wanted to solve this. Not by writing better selectors, but by building a system that writes its own fixes.

What I Built
SIGNAL: is a self-healing web scraper with RAG-powered Q&A. It does three things:

1. Scrapes documentation sites using Bright Data Scraper Studio
2. Indexes the content into a searchable knowledge base (local RAG, no paid APIs)
3. Self-heals when scrapers break — automatically detects degradation, triggers AI to fix the collector, and verifies the fix worked The whole system runs on a single Railway container. No GPU, no OpenAI key, no vector database service.

How I Built It: Cursor AI as a Pair Programmer
I want to be upfront about how this got built: I used Cursor AI throughout development. Not as a replacement for engineering judgment, but as a pair programmer that kept me moving when I would have otherwise gotten stuck.

Here's what that looked like in practice:
Debugging in real-time. When the self-healing loop was stuck in an infinite retry cycle, I pasted the logs into Cursor and asked "why is this looping?" It traced the issue to a missing stuck_since timestamp reset — a bug I'd been staring at for an hour.
Architecture decisions. I'd sketch out a module (like the normalizer that handles 8+ different collector schemas) and ask Cursor to review the approach. It caught that my initial field-name mapping would break on Wikipedia's HTML content and suggested the HTML-stripping fallback.
Rapid prototyping. The entire heal pipeline — diagnose → trigger → poll → approve → re-scrape — went from concept to working code in an afternoon. Cursor helped me write the Bright Data API client, the health metrics scoring, and the auto-approval logic without getting bogged down in API documentation.

The boring stuff. Pydantic models, FastAPI route boilerplate, config loading from .env — Cursor handled all of it so I could focus on the interesting parts: the self-healing logic and the RAG pipeline.
What it didn't replace: I still made the design decisions. I chose Bright Data over Scrapy, ChromaDB over Pinecone, sentence-transformers over OpenAI embeddings. I decided the heal loop should auto-approve rather than require manual review. I wrote the demo script and the architecture explanation. Cursor wrote code; I made it make sense together.
The honest truth: without Cursor, this project would have taken 3-4x longer. Not because the code is complex, but because the integration surface is huge — Bright Data APIs, ChromaDB, Groq, FastAPI, React, Railway deployment. Cursor kept me from getting lost in documentation and boilerplate.

Architecture: Three Layers
Layer 1: Scraping
I use Bright Data Scraper Studio for the heavy lifting — anti-bot protection, JavaScript rendering, proxy rotation. For simple public pages, there's a direct HTTP fallback using httpx.
Each scrape produces a normalized JSON with url, title, content, and metadata. The normalizer handles 8+ different collector schemas (news articles, docs, job listings, Wikipedia, GitHub READMEs) in a single function.
Layer 2: RAG Pipeline
Content gets chunked into 800-character pieces with 150-character overlap (sentence-aware splitting). Each chunk is embedded locally using sentence-transformers/all-MiniLM-L6-v2 — a 80MB model that runs on CPU.
Embeddings go into ChromaDB (persistent, local, free). Queries use cosine similarity to find the top 5 relevant chunks, which get passed as context to Groq's LLM for answer generation.
Key numbers:

• 858 chunks indexed across 12 documentation sites
• 0.68 max cosine distance threshold for relevance filtering
• 5500 character context window for LLM
• <2 second average query response time Layer 3: Self-Healing (The Cool Part) This is where it gets interesting. Every scraper has health metrics:
• Success rate: % of records with both title AND body
• Empty title %: % of records missing titles
• Empty body %: % of records missing content

When these metrics drop below thresholds (success < 60% or empty fields > 40%), the system automatically:

1. Diagnoses — runs a scrape to measure current extraction health
2. Triggers Bright Data AI — sends the collector to Bright Code Fixer, which analyzes the site's HTML changes and proposes a collector modification
3. Auto-approves — validates the fix preview looks reasonable (has title-like and body-like fields, no junk values)
4. Saves to production — publishes the fix to the same collector ID
5. Re-scrapes — runs a fresh scrape to measure "after" metrics
6. Compares — shows before/after improvement The whole loop runs in 7-8 minutes. If the site changes again next week, the system catches it and heals again. Real Example I triggered a heal on a broken React documentation scraper: BEFORE HEAL: Empty Title: 100% Empty Body: 100% Success Rate: 0%

AFTER HEAL:
Empty Title: 0%
Empty Body: 0%
Success Rate: 100%

Bright Data's AI analyzed the React docs HTML, found that the old selectors were targeting deprecated class names, and rewrote the extraction logic. I didn't touch a single line of code.

Total cost: $0 per query. Everything runs locally except the LLM (Groq free tier) and scraping (Bright Data free tier: 5,000 credits/month).

The Interesting Parts

1. Schema-Aware Normalizer The normalizer handles 8+ different collector schemas in a single function. News articles have headline + article_contet. Documentation sites have main_content + section_headings + code_examples. Wikipedia has article_content (HTML) + short_description. Each path extracts title and content from different field names, strips HTML, and produces a uniform NormalizedDoc.
2. Interleaved Multi-Query Retrieval The retriever doesn't just embed the question and search. It generates expanded queries (strips site names, adds alternative phrasings), runs them in parallel, and interleaves results so specialized searches aren't buried by generic matches.
3. Memory-Safe Ingestion Railway gives you 1-2GB RAM. The ingestion pipeline processes embeddings in slices of 4 documents with explicit gc.collect() between batches to stay within limits. Each batch chunks, embeds, and upserts to ChromaDB without OOM.
4. Smart Query Routing The pipeline auto-routes queries:
5. Weather questions → Open-Meteo API (live, free, no scrape needed)
6. Wikipedia questions → Wikipedia REST API + Groq summarization
7. Corpus questions → ChromaDB retrieval + Groq generation
8. Out-of-scope → Refuses to answer, lists indexed domains

What I'd Do Differently

1. Make the heal loop event-driven instead of polling. Right now it polls Bright Data every 2 seconds, which works but isn't elegant.
2. Add a feedback loop — if the same collector heals repeatedly, something is structurally wrong with the selectors.
3. Support more sites — the 13 collectors are a good start, but the normalizer could handle more schemas. Try It API: https://healscrape-production.up.railway.app/docs # Query the knowledge base curl -X POST https://healscrape-production.up.railway.app/query \ -H "Content-Type: application/json" \ -d '{"query": "What are React server components?"}'

Check health metrics

curl https://healscrape-production.up.railway.app/knowledge
Frontend: The React dashboard shows the full pipeline — radar visualization of indexed sources, query console with source filtering, and the heal dashboard with before/after metrics.
Conclusion
The hardest part of web scraping isn't writing the first scraper — it's keeping it working. SIGNAL solves this by closing the loop: scrape → detect degradation → auto-fix → verify. No manual intervention, no midnight debugging sessions.
And honestly? I couldn't have built it this fast without Cursor AI. It didn't write the architecture or make the design decisions, but it kept me out of the weeds so I could focus on what matters: making the system work end-to-end.
If you're building data pipelines that depend on web scraping, self-healing isn't a nice-to-have. It's the only way to stay sane.

Built for the Bright Data Hackathon 2026. Developed with Cursor AI. Source code available on GitHub.