DEV Community: Chandan Garg

Signal: A Scraper that heals itself

Chandan Garg — Sun, 23 Aug 2026 06:11:14 +0000

SIGNAL: I Built a Web Scraper That Fixes Itself
Because nobody wants to manually debug broken scrapers at 2 AM.

The Problem Nobody Talks About
Web scraping is easy. Web scraping that stays working is hard.
Every developer who's built a scraper knows the drill: you write selectors, they work for a week, then a site updates its HTML and your scraper returns empty objects. Suddenly you're debugging $('.article-title') vs $('.post-title') at midnight, and your data pipeline is broken.

I wanted to solve this. Not by writing better selectors, but by building a system that writes its own fixes.

What I Built
SIGNAL: is a self-healing web scraper with RAG-powered Q&A. It does three things:

Scrapes documentation sites using Bright Data Scraper Studio
Indexes the content into a searchable knowledge base (local RAG, no paid APIs)
Self-heals when scrapers break — automatically detects degradation, triggers AI to fix the collector, and verifies the fix worked The whole system runs on a single Railway container. No GPU, no OpenAI key, no vector database service.

How I Built It: Cursor AI as a Pair Programmer
I want to be upfront about how this got built: I used Cursor AI throughout development. Not as a replacement for engineering judgment, but as a pair programmer that kept me moving when I would have otherwise gotten stuck.

Here's what that looked like in practice:
Debugging in real-time. When the self-healing loop was stuck in an infinite retry cycle, I pasted the logs into Cursor and asked "why is this looping?" It traced the issue to a missing stuck_since timestamp reset — a bug I'd been staring at for an hour.
Architecture decisions. I'd sketch out a module (like the normalizer that handles 8+ different collector schemas) and ask Cursor to review the approach. It caught that my initial field-name mapping would break on Wikipedia's HTML content and suggested the HTML-stripping fallback.
Rapid prototyping. The entire heal pipeline — diagnose → trigger → poll → approve → re-scrape — went from concept to working code in an afternoon. Cursor helped me write the Bright Data API client, the health metrics scoring, and the auto-approval logic without getting bogged down in API documentation.

The boring stuff. Pydantic models, FastAPI route boilerplate, config loading from .env — Cursor handled all of it so I could focus on the interesting parts: the self-healing logic and the RAG pipeline.
What it didn't replace: I still made the design decisions. I chose Bright Data over Scrapy, ChromaDB over Pinecone, sentence-transformers over OpenAI embeddings. I decided the heal loop should auto-approve rather than require manual review. I wrote the demo script and the architecture explanation. Cursor wrote code; I made it make sense together.
The honest truth: without Cursor, this project would have taken 3-4x longer. Not because the code is complex, but because the integration surface is huge — Bright Data APIs, ChromaDB, Groq, FastAPI, React, Railway deployment. Cursor kept me from getting lost in documentation and boilerplate.

Architecture: Three Layers
Layer 1: Scraping
I use Bright Data Scraper Studio for the heavy lifting — anti-bot protection, JavaScript rendering, proxy rotation. For simple public pages, there's a direct HTTP fallback using httpx.
Each scrape produces a normalized JSON with url, title, content, and metadata. The normalizer handles 8+ different collector schemas (news articles, docs, job listings, Wikipedia, GitHub READMEs) in a single function.
Layer 2: RAG Pipeline
Content gets chunked into 800-character pieces with 150-character overlap (sentence-aware splitting). Each chunk is embedded locally using sentence-transformers/all-MiniLM-L6-v2 — a 80MB model that runs on CPU.
Embeddings go into ChromaDB (persistent, local, free). Queries use cosine similarity to find the top 5 relevant chunks, which get passed as context to Groq's LLM for answer generation.
Key numbers:

858 chunks indexed across 12 documentation sites
0.68 max cosine distance threshold for relevance filtering
5500 character context window for LLM
<2 second average query response time Layer 3: Self-Healing (The Cool Part) This is where it gets interesting. Every scraper has health metrics:
Success rate: % of records with both title AND body
Empty title %: % of records missing titles
Empty body %: % of records missing content

When these metrics drop below thresholds (success < 60% or empty fields > 40%), the system automatically:

Diagnoses — runs a scrape to measure current extraction health
Triggers Bright Data AI — sends the collector to Bright Code Fixer, which analyzes the site's HTML changes and proposes a collector modification
Auto-approves — validates the fix preview looks reasonable (has title-like and body-like fields, no junk values)
Saves to production — publishes the fix to the same collector ID
Re-scrapes — runs a fresh scrape to measure "after" metrics
Compares — shows before/after improvement The whole loop runs in 7-8 minutes. If the site changes again next week, the system catches it and heals again. Real Example I triggered a heal on a broken React documentation scraper: BEFORE HEAL: Empty Title: 100% Empty Body: 100% Success Rate: 0%

AFTER HEAL:
Empty Title: 0%
Empty Body: 0%
Success Rate: 100%

Bright Data's AI analyzed the React docs HTML, found that the old selectors were targeting deprecated class names, and rewrote the extraction logic. I didn't touch a single line of code.

Total cost: $0 per query. Everything runs locally except the LLM (Groq free tier) and scraping (Bright Data free tier: 5,000 credits/month).

The Interesting Parts

Schema-Aware Normalizer The normalizer handles 8+ different collector schemas in a single function. News articles have headline + article_contet. Documentation sites have main_content + section_headings + code_examples. Wikipedia has article_content (HTML) + short_description. Each path extracts title and content from different field names, strips HTML, and produces a uniform NormalizedDoc.
Interleaved Multi-Query Retrieval The retriever doesn't just embed the question and search. It generates expanded queries (strips site names, adds alternative phrasings), runs them in parallel, and interleaves results so specialized searches aren't buried by generic matches.
Memory-Safe Ingestion Railway gives you 1-2GB RAM. The ingestion pipeline processes embeddings in slices of 4 documents with explicit gc.collect() between batches to stay within limits. Each batch chunks, embeds, and upserts to ChromaDB without OOM.
Smart Query Routing The pipeline auto-routes queries:
Weather questions → Open-Meteo API (live, free, no scrape needed)
Wikipedia questions → Wikipedia REST API + Groq summarization
Corpus questions → ChromaDB retrieval + Groq generation
Out-of-scope → Refuses to answer, lists indexed domains

What I'd Do Differently

Make the heal loop event-driven instead of polling. Right now it polls Bright Data every 2 seconds, which works but isn't elegant.
Add a feedback loop — if the same collector heals repeatedly, something is structurally wrong with the selectors.
Support more sites — the 13 collectors are a good start, but the normalizer could handle more schemas. Try It API: https://healscrape-production.up.railway.app/docs # Query the knowledge base curl -X POST https://healscrape-production.up.railway.app/query \ -H "Content-Type: application/json" \ -d '{"query": "What are React server components?"}'

Check health metrics

curl https://healscrape-production.up.railway.app/knowledge
Frontend: The React dashboard shows the full pipeline — radar visualization of indexed sources, query console with source filtering, and the heal dashboard with before/after metrics.
Conclusion
The hardest part of web scraping isn't writing the first scraper — it's keeping it working. SIGNAL solves this by closing the loop: scrape → detect degradation → auto-fix → verify. No manual intervention, no midnight debugging sessions.
And honestly? I couldn't have built it this fast without Cursor AI. It didn't write the architecture or make the design decisions, but it kept me out of the weeds so I could focus on what matters: making the system work end-to-end.
If you're building data pipelines that depend on web scraping, self-healing isn't a nice-to-have. It's the only way to stay sane.

Built for the Bright Data Hackathon 2026. Developed with Cursor AI. Source code available on GitHub.

Beyond the Prompt: How AI Agent Harnesses Actually Manage Memory

Chandan Garg — Sun, 16 Aug 2026 13:46:43 +0000

Most tutorials make building an AI agent look deceptively simple: take a user prompt, append the last five chat messages, hand it to an LLM, and parse a JSON tool call.

In production, this naive pattern fails immediately. Context windows fill with polite conversational filler, token costs skyrocket, latency degrades, and the model forgets critical preferences established three turns earlier.

To build an agent that operates reliably over weeks or months, you need two systems working together:
A Cognitive Harness: An execution environment that structures memory into distinct cognitive tiers (procedural, episodic, and semantic) and enforces safety guardrails around tool execution.
A Closed-Loop LLMOps Pipeline: An evaluation and observability flywheel that diagnoses failures and continuously tunes prompts and retrieval parameters.

The Anatomy of an Agent Harness The Harness is the runtime engine wrapping the model. It handles context assembly, coordinates memory retrieval, executes tools, and validates outputs before returning a response to the user.

The Working Memory Layer
Working memory is the dynamic prompt assembled just before inference. It combines:
The Static System Prompt: Core identity, high-level behavioral constraints, and format contracts.
The Current Turn Context: The user’s latest query alongside immediate conversational context.
Dynamic Injections: Retrieved knowledge pulled on demand from external memory layers.

The Three Cognitive Memory Tiers Treating all memory as a single append-only text log creates noisy retrievals and bloated context windows. Production architectures adopt a tripartite model borrowed from cognitive neuroscience:

A. Procedural Memory: Knowing How
Procedural memory represents an agent's operational muscle memory. It encodes the rules, workflows, and execution policies that dictate how tasks should be performed without requiring the model to deduce them from scratch.
Implementation: Modular markdown files (e.g., skills.md), standardized tool schemas (OpenAPI specs), and few-shot exemplars demonstrating correct parameter construction.
Role in Runtime: Injected deterministically into the system prompt based on active task classification.

B. Episodic Memory: Knowing What Happened
Episodic memory captures timestamped records of past experiences, interactions, and raw conversation turns.
Implementation: Append-only message databases (Postgres, DynamoDB) keyed by session_id and timestamp.
Role in Runtime: Queried via time-based lookups or semantic similarity when answering questions about past sessions (e.g., "What did we discuss last Tuesday?").

C. Semantic Memory: Knowing What Is True
Semantic memory stores distilled, timeless facts, concepts, and user preferences detached from chronological session transcripts.
Implementation: Vector databases (pgvector, Pinecone, Qdrant) or Knowledge Graphs (Neo4j).
Role in Runtime: Retrieved via RAG to ground the model in persistent domain truths (e.g., User prefers Python over TypeScript, Budget ceiling is $5,000).

The Memory Consolidation Subsystem Storing raw chat turns indefinitely causes vector retrieval to degrade because search queries end up matching casual banter instead of key facts. To maintain an accurate semantic store, production harnesses use active background consolidation.

Episodic Capture: Every turn is written directly to the episodic message store.
Periodic Extraction: Every N interactions, an asynchronous background worker (the Summarizer Agent) processes recent turns.
Fact Distillation: The Summarizer identifies atomic facts, strips out conversational filler, and resolves conflicting updates (e.g., updating a previously recorded preference).
Semantic Upsert: Distilled facts are embedded and stored in the semantic database, ready for low-latency retrieval.

The Agentic Tool Loop & Guardrails Once working memory is assembled, the LLM enters an execution loop:

Tool Invocation: The model emits a structured tool call (e.g., reading a calendar, querying a database, or invoking a payment gateway).
Harness Execution: The harness runs the tool in a sandbox, captures output data or error traces, appends the result to the working context, and re-invokes the model.
Guardrail Interception: Once the model generates a final response, it passes through output guardrails before reaching the user:
Schema Validation: Verifies structural compliance of JSON/typed outputs.
Safety & Privacy: Filters PII leaks, system prompt leakage, and policy violations.
Hallucination Checks: Cross-references citations and tool returns against the final claims.

Closing the Loop: The LLMOps Flywheel Deploying an agent without telemetry turns it into an untraceable black box. The right-hand side of a mature architecture creates an automated feedback loop.

Tracing (1 Trace per Run)
Every interaction generates an end-to-end trace logging the full call tree: retrieved memory chunks, raw prompts, intermediate tool calls, execution latencies, and token consumption.

The Two-Pronged Evaluation Pipeline
LLM-as-a-Judge: A separate, highly capable model inspects the full trace against standardized evaluation rubrics:
Faithfulness: Did the model hallucinate beyond retrieved context?
Tool Correctness: Were the correct tools selected with valid parameters?
Task Completion: Did the response directly resolve the user’s intent?
System Observability: Tracks infrastructure metrics including p95 latency, token burn rates, error distributions, and rate-limit headroom.
The Diagnostic Gate
The evaluation output routes through a gate:
Eval Failed: Triggers diagnostic alerts. Engineers analyze whether the failure stemmed from missing RAG context, ambiguous tool definitions, or prompt drift, then patch the pipeline.
Eval Passed: Confirmed high-quality traces feed back into the system to optimize system prompts, refine few-shot exemplars, and tune retrieval parameters (such as similarity thresholds and top-k selection).