DEV Community

Sagar Maurya
Sagar Maurya

Posted on

How I Built a Self-Healing AI Tracker That Fixes Its Own Scrapers

#wemakedevs #brightdatachallenge #hackathon

WeMakeDevs and Bright Data ran a week-long hackathon called Into the Scrape-Verse — the challenge was to build a self-healing web scraper and turn live web data into something real.

I had one week. I built Crawlr — an autonomous AI model and dataset tracker that watches four research sources in real time, detects when a scraper breaks, heals it automatically, and logs every event to a live audit trail. No human in the loop.

Crawlr landing page

The Problem I Wanted to Solve

Scrapers break silently. A site changes a class name, a layout shifts, a field moves — and your scraper returns nothing. You find out days later when someone notices the data is stale.

The usual fix is manual: inspect the HTML, find the new selector, redeploy. It works until the next time.

I wanted to build something that detects the break, figures out what went wrong, and fixes itself — without any human involvement.

What Crawlr Tracks

The AI research world moves fast. New models drop on EleutherAI. Together AI publishes blog posts. Papers With Code indexes the latest benchmarks. OpenRouter lists every available model with real pricing data.

Crawlr watches all four:

Source What it scrapes
EleutherAI Model and dataset releases
Together AI Blog posts and announcements
Papers With Code Latest research papers
OpenRouter Live model catalog with context length and pricing

Every pipeline run scrapes all four, normalises the data into a unified format, and serves it through a real-time dashboard. 78 records across 4 sources, updated on every run.

Dashboard

How Bright Data Scraper Studio Powers Everything

Every scraper in Crawlr was built using Bright Data's Scraper Studio — an AI-powered platform that builds, runs, and self-heals custom web scrapers from your terminal.

The entire workflow is four commands:

# Login once
npx -p @brightdata/cli bdata login

# Describe the data you want — AI builds the scraper
bdata scraper create https://eleuther.ai/releases \
  "model name, description, release date, type (model/dataset/library)"

# Run it — returns clean JSON
bdata scraper run c_mt4lee1f5n8ouckku https://eleuther.ai/releases --pretty

# When the site changes — heal it with a plain-English description
bdata scraper heal c_mt4lee1f5n8ouckku \
  "description field is empty in 6 out of 8 records, selector may have changed"
Enter fullscreen mode Exit fullscreen mode

Terminal showing bdata scraper run — clean JSON output

The c_* Collector ID you get back is a live production API endpoint. No deployment step. No proxy rotation, retries, or unblocking to manage — Bright Data handles all of that. You just describe what you want.

The Moment That Made It Click

While building the OpenRouter collector, the initial scraper was returning records but the context_length and pricing_per_token fields were all null. I ran:

bdata scraper heal c_mt4m2cql181bpcgals \
  "context_length and pricing fields returning null. 
   Model names contain provider prefix like 'Meta: Llama 3.1'. 
   Extract provider from model name string."
Enter fullscreen mode Exit fullscreen mode

Two minutes later, the collector was returning this:

{
  "title": "Meta: Muse Spark 1.2 Contributor",
  "provider": "Meta",
  "context_length": "1.05M context",
  "pricing_per_token": "$0.10 /M input tokens"
}
Enter fullscreen mode Exit fullscreen mode

Same Collector ID. Nothing downstream changed. The scraper just started working correctly. That's when I understood what made Scraper Studio genuinely different.

The Architecture

Crawlr has three components working together in a closed loop.

1. The Pipeline

scrapers/pipeline.js runs all four collectors in sequence, normalises the raw JSON into a unified schema, and saves the output to the data/ folder.

Each source returns completely different data shapes — OpenRouter looks nothing like EleutherAI. Every source has its own unify* function that maps it to the same schema:

{
  id: "eleuther-0-trlX",
  source: "eleuther",
  title: "trlX",
  description: "A repo for distributed training of language models...",
  url: "https://github.com/CarperAI/trlx",
  type: "Library",
  date: "Dec 9, 2023"
}
Enter fullscreen mode Exit fullscreen mode

2. Sentry — The Health Agent

scrapers/sentry.js is an autonomous health checker that runs after every pipeline and validates each source's output for three failure modes:

  • Schema Break — payload is empty or not an array
  • Volume Drop — record count fell more than 50% vs the baseline
  • Null Creep — required fields are empty in more than 20% of records

When it detects a failure, it auto-generates a diagnosis from the actual failure data — field names, failure percentages, a sample broken record — and calls bdata scraper heal:

function buildDiagnosis(sourceId, data, issues) {
    const sample = JSON.stringify(data[0]).slice(0, 200);
    return `Self-healing prompt for '${sourceId}': 
            ${issues.join(' | ')} 
            Sample record: ${sample}
            Action: locate updated selectors for the listed fields.`;
}
Enter fullscreen mode Exit fullscreen mode

After healing, Sentry re-runs the collector to verify recovery. Every step is logged with timestamps. Here's what a normal healthy run looks like — no anomalies this time, all four sources verified clean. (The full heal cycle, triggered live, is a bit further down in Chaos Mode.)

Sentry health check — all 4 sources verified healthy

3. The Dashboard

A pure HTML/CSS/JS frontend — no framework, no build step. It reads directly from the JSON files the pipeline writes.

Dashboard with search active, filtering cards

Features:

  • 4 source tabs with live record counts
  • Instant search across title, description, and organization
  • Live Audit Log — color-coded pills for each event type (HEALTHY, ANOMALY, HEALING, ERROR)
  • Chaos Mode — simulates a scraper break so you can see the full heal cycle live

Chaos Mode — log showing the full simulated break and recovery

What a Full Run Looks Like

Pipeline:

✅ eleuther:   8 items saved
✅ togetherai: 25 items saved
✅ pwc:        25 items saved
✅ openrouter: 20 items saved
✅ Pipeline complete.
Enter fullscreen mode Exit fullscreen mode

Sentry health check:

✅ [eleuther]   Healthy. 8 records returned.
✅ [togetherai] Healthy. 31 records returned.
✅ [pwc]        Healthy. 50 records returned.
✅ [openrouter] Healthy. 20 records returned.
✅ All sources checked.
Enter fullscreen mode Exit fullscreen mode

Terminal showing full pipeline.js run — all 4 sources scraped and saved

What I Learned

bdata scraper heal is the real unlock. You describe what's broken in plain English, Bright Data's AI rewrites the extraction logic, and the same Collector ID starts returning clean data. I watched it fix broken selectors on live sources multiple times during the week.

Bright Data handles the hard parts of scraping. Proxy rotation, retries, unblocking, rate limiting — none of that is your problem. You describe what you want and get clean JSON back.

The Collector ID is a production API endpoint. Trigger it with POST /dca/trigger from any language or scheduler. No deployment step. This made wiring it into the pipeline and dashboard trivial.

Self-healing is not a gimmick. Going into this I wasn't sure how well automated healing would actually work on real sites. After a week of watching it fix real extraction failures, I'm convinced it's the right direction for production scrapers.

Try It

GitHub: github.com/mauryasagar/crawlr

Clone it, add your .env with four Bright Data Collector IDs, and run:

npm run pipeline   # Scrape all 4 sources
npm run sentry     # Health check + auto-heal
npm run serve      # Open the dashboard at localhost:3000
Enter fullscreen mode Exit fullscreen mode

Built for the Into the Scrape-Verse hackathon — WeMakeDevs × Bright Data, August 2026

#webdev #javascript #hackathon #brightdata #scraping #wemakedevs #opensource

Top comments (0)