Scraper Factory took first place at the WeMakeDevs Zero Downtime Hackathon in San Francisco. On the surface it is a small product. It scrapes about twelve days of TV guide data for Philo, YouTube TV and Sling, ranks the top ten upcoming shows and the top ten upcoming movies, and gives you one click to record each title on Philo.
The interesting part is underneath. I wanted scrapers that notice when they break, attempt their own repair, and only accept that repair once it is proven correct. Three tools carried the project, each doing what it is best at: Bright Data for scraping and self-healing, Port as the control plane, and SigNoz for the audit trail. This post walks through how they fit together, with the code and config that make each piece work, so you can follow along or run it yourself.
Try the product: scraper-factory-top20.vercel.app
Source code: github.com/Nitishmane/scraper-factory
The product: the top 10 shows and top 10 movies, with a one-click Record on Philo button where a direct link exists.
The problem: scrapers rot without anyone noticing
Every scraper I have built eventually fails. A site ships a redesign, a selector stops matching, and the data either stops arriving or, worse, keeps arriving slightly wrong. Times land in the wrong column, titles swap with descriptions, rows go missing. You usually find out a week later when something downstream looks odd.
Most teams treat scrapers as scripts. Scripts don't have owners, health states or an on-call rotation. I wanted each scraper treated like any other production service: state you can see, automation that reacts to that state, and a clear record of every repair.
The one rule that ties the project together: judgment goes to the agents, proof stays deterministic. A model decides what broke and how to repair it. Plain code, diffing against a frozen fixture, decides whether the repair is real.
How the pieces fit
Here is the production data path. Port triggers the scrape, Bright Data extracts the guide, the pipeline ranks titles with TMDB data, and the results land back in Port's Context Lake, where the webapp reads them.
The data pipeline, from Port's scheduled trigger to the published top 10 lists.
In production this runs weekly without a laptop. Port's wf_scheduled_scrape workflow fires every Monday at 08:00 UTC and dispatches a GitHub Actions job that runs scrape, heal (if needed), catalog refresh, rank and publish, then exits. The same job can be started on demand from a Port button.
Bright Data: scrapers that repair themselves
The usual approach to scraping is structural: you find that the airtime lives in div.card > span:nth-child(3) and write that down. That selector says where the data sits, not what it means. Rename a CSS class and it breaks. Reorder the cards and it breaks.
Bright Data's Scraper Studio builds a collector from a natural language description of the data instead of selectors. A description of meaning survives a redesign far better than a selector does. And the prompt becomes the thing you heal from: when a scraper drifts, the scraper heal command reasons over that same description plus a plain language report of what went wrong.
A prompt about meaning
This is the full prompt behind the channel schedule collector, from src/factory/pipeline.py:
PROMPT = (
"This page lists a TV channel's upcoming schedule as program cards. Extract every "
"program card. For each: the program title as title, the episode title if shown as "
"episode_title, the airing date exactly as displayed (like Sat, Aug 22) as date_raw, "
"the start time exactly as displayed (like 2:00 PM) as start_raw, the time range text "
"if shown (like Sat, Aug 22 2:00 PM - 2:30 PM) as time_range_raw, the category text "
"as category. Include every card."
)
There are no class names, positions or DOM paths in it. Notice also date_raw and start_raw. I always capture the string exactly as displayed next to any parsed value. If start_raw says "8:00 PM" but the parsed UTC time is null, that is a drift signal, and one of the detection rules checks for exactly that.
Everything goes through the Bright Data CLI, which made it easy to script from Python. Creating a collector is one command:
brightdata login --api-key $BRIGHTDATA_API_KEY
brightdata scraper create "$FIXTURE_BASE_URL/channel.html" "<PROMPT>"
brightdata scraper run <collector_id> --urls "<channel page url>" --pretty
One collector, three providers
The guide data comes from a third party site where every channel has its own page: roughly 455 program cards covering about twelve days. Because every channel page has the same structure, one shared collector serves all three providers (137 Philo channels, 175 YouTube TV channels and 119 Sling channels). A second collector reads Philo's browse pages, about 739 shows and movies, so the ranked list can link straight to a record button.
One practical tip: generate the collector against a small, clean sample. I trimmed a real channel page down to 30 program cards, removed links the generator might try to follow, served it through a tunnel, and created the collector from that. That same trimmed page later became my frozen test fixture.
Splitting fetch and extract
Bright Data's usage policy does not allow collector runs directly against streaming media domains. After checking with their support team, I split the job across two Bright Data products. The Web Unlocker fetches the live page and archives it. The Scraper Studio collector then extracts from that archived copy, served through a tunnel.
Fetch with the Web Unlocker, extract with a Scraper Studio collector.
The code still tries a direct collector run first, so if the policy ever changes the relay step drops away on its own. A side benefit: every scrape's input HTML is archived, so any drift investigation can be replayed byte for byte.
The two Scraper Studio collectors: channel-schedule-v3 for every guide page, and philo-catalog for record links.
The repair loop
When drift is detected, the healer calls scraper heal with the collector ID and a plain language symptom description. Bright Data returns a proposal in an awaiting_approval state, with a preview_result of what the repaired scraper would extract. Nothing changes until my code decides. Here is the core of src/factory/agents/healer.py, trimmed for length:
proposal = brightdata.heal(collector_id, drift_description)
# CLI: brightdata scraper heal <collector_id> "<symptom>"
# -> {"status": "awaiting_approval", "preview_result": [...]}
verdict = verifier.verify("channel", proposal.preview_result) # no model
if verdict.passed:
brightdata.approve(collector_id) # scraper approve <id>
port.set_scraper_health(collector_id, "healthy", verify_status="pass")
else:
brightdata.approve(collector_id, reject=True) # approve <id> --reject
port.set_scraper_health(collector_id, "broken", verify_status="fail")
# heal_event recorded as rolled_back; a person takes it from here
The awaiting_approval state is what makes this safe. The model proposes, my code checks, and only then does anything change.
Because every run goes through the CLI, the Bright Data control panel doubles as a run log. Each weekly scrape shows up as a batch of collector runs with inputs, records and success rate:
Run history for the channel schedule collector. The Monday batches are the weekly production scrape.
The gate: detection and verification in plain code
Two parts of the system never involve a model: deciding that something is wrong, and deciding that a fix is right.
Seven drift rules
Every run passes through seven deterministic checks in src/factory/detect.py: rows per page out of band, required field null rate, raw time present but parsed time null, channel collapse, empty page rate, a change in the schema hash of returned keys, and lineup disappearance. The thresholds are plain constants:
REQUIRED_FIELDS = ("channel", "title", "start_utc")
NULL_RATE_LIMIT = 0.10
EMPTY_PAGE_LIMIT = 0.20 # >20% of channel pages returning nothing is breakage
CHANNEL_COLLAPSE = 0.80 # <80% of requested channels yielding rows is breakage
LINEUP_DELTA_LIMIT = 0.10 # >10% channel-set churn needs two confirming runs
PAGE_ROW_BANDS = (15, 800)
A broken scraper and a real change at the source look identical in one sample. If 15 percent of channels vanish, that could be breakage or a real carriage change. So lineup churn needs two consecutive runs agreeing before it is accepted, and drift state is keyed by collector:provider so one provider's lineup is never compared against another's.
The verifier
The verifier normalises the heal preview and the hand-checked expected output for the frozen fixture, then diffs them. It is the only thing allowed to promote a repair:
COMPARE_FIELDS = ("title", "date_raw", "start_raw")
def verify(provider: str, rows: list[dict]) -> Verdict:
want, got = _normalise(expected(provider)), _normalise(rows)
if want == got:
return Verdict(True, [])
diff = []
missing = [w for w in want if w not in got]
extra = [g for g in got if g not in want]
if len(want) != len(got):
diff.append(f"row count {len(got)}, expected {len(want)}")
for item in missing[:3]:
diff.append(f"missing {item}")
for item in extra[:3]:
diff.append(f"unexpected {item}")
return Verdict(False, diff)
The fixture is three files: a real channel page trimmed to 30 cards, its hand-checked expected output, and a mutated copy with every structural handle moved (classes renamed, cards reordered, airtimes wrapped in new elements). The mutated copy lets me inject drift on demand with make demo-drift.
The self-heal loop: three ways to detect a problem, one repair path, and a deterministic gate before anything is promoted.
Port: the control plane
Scrapers needed to be entities with state, not scripts. Port's data model made that possible. Seven blueprints model the system: data_source, scraper, scrape_run, heal_event, ranked_title, requirement and factory_deployment. Each scraper has a health property that moves between healthy, drifting and broken. Every run is a scrape_run entity. Every repair attempt is a heal_event recording its trigger, the drift description, the outcome and a link to the trace.
Both scrapers as Port entities, with collector ID, verify status and health.
Automations that react to state
The fastest repair path is a Port automation on the scraper blueprint. A JQ condition on the updated entity means it only fires when health flips to drifting:
{
"identifier": "invoke_healer_on_drift",
"trigger": {
"type": "automation",
"event": { "type": "ENTITY_UPDATED", "blueprintIdentifier": "scraper" },
"condition": {
"type": "JQ",
"expressions": [".diff.after.properties.health == \"drifting\""],
"combinator": "and"
}
},
"invocationMethod": { "type": "WEBHOOK", "url": "<factory>/heal?token=..." },
"publish": true
}
Workflows with an AI agent in the middle
When a run fails outright, the wf_heal_on_bad_run workflow takes over. It is triggered when a scrape_run is created with status error or with drift detected. The middle node is Port's Triage agent, which reads the scraper's run and heal history and writes a plain language symptom report. The last node passes that report to the heal endpoint:
trigger -> EVENT_TRIGGER on scrape_run created
(.status == "error") or (.drift_detected == true)
triage -> AI_AGENT "triage"
"Describe the observed SYMPTOM (what changed in the data),
never a guess at the fix. Under 400 characters."
heal -> WEBHOOK POST /heal
{ provider, collector_id, drift_description: {{ .triage.output }} }
The wf_heal_on_bad_run workflow in Port: the trigger, the Triage agent, then the call to the healer.
The agent's job stays narrow on purpose. Triage writes the symptom report but never approves its own fix. That decision stays with the verifier.
The other workflows are simpler. wf_scheduled_scrape runs on a cron (0 8 * * 1) and dispatches the GitHub Actions job. wf_publish_on_success publishes the rankings whenever a clean run lands. Three self-service actions, run_scrape_now, refresh_catalog_now and publish_now, let an operator force a cycle from the Port UI.
Agents, scorecards and the Context Lake
- Four AI agents. Triage (approval required), Heal Explainer (explains why a repair was approved or rejected), On-Call Assistant (answers operator questions) and Catalog Coverage (reports how many ranked titles have a direct record link).
-
A reliability scorecard.
scraper_reliabilitygrades each scraper Basic, Bronze (verified today), Silver (verification passing) or Gold (healthy and verified). -
The Context Lake as the data layer. The 20
ranked_titleentities are the product. The Next.js webapp has no database of its own; it reads them straight from Port. Publishing is just an upsert.
The Factory Operator dashboard in Port: scrapers by health, the architecture note and recent heal events, including the rolled back repairs.
Everything above is provisioned through Port's API by one idempotent script, scripts/bootstrap_port.py, so the whole setup can be rebuilt at any time.
SigNoz: the audit trail
Any system that changes itself needs a clear record of what it did and why. SigNoz carried every trace, metric and log, and its alerting acted as the safety net for the repair loop. It runs self-hosted with Docker Compose, so there was no API key or trial to manage during the hackathon.
Auto-instrumentation first
The FastAPI service runs under OpenTelemetry auto-instrumentation and exports OTLP to SigNoz:
git clone -b main https://github.com/SigNoz/signoz.git ../signoz
docker compose -f ../signoz/deploy/docker/docker-compose.yaml up -d
# the factory service (this is what `make serve` runs)
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
opentelemetry-instrument python -m uvicorn factory.app:app --host 0.0.0.0 --port 8000
Because the web layer is instrumented, even Port's automation calling /heal shows up as a span. On top of that I added custom spans for each pipeline step, with the attributes I would want during an incident:
with tracer().start_as_current_span("scrape.heal.reprompt") as span:
span.set_attribute("target.provider", provider)
span.set_attribute("scraper.id", collector_id)
span.set_attribute("heal.preview_rows", len(proposal.preview_result))
log().info("heal requested for %s: %s", provider, drift_description)
...
span.set_attribute("verify.status", "pass" if verdict.passed else "fail")
A scrape produces one scrape.run trace with a scrape.fetch span per channel page (tagged with provider, channel ID, row count and fetch mode), then scrape.extract, scrape.validate, and when a repair runs, scrape.heal.reprompt and scrape.verify.fixture. Logs carry trace and span IDs, and the drift description, heal prompt and verifier diff are all logged inside the relevant span.
Two detection paths, one endpoint
The inline detector marks the scraper as drifting in Port within seconds. SigNoz is the safety net: an alert rule on scraper.drift.detected > 0 over a five minute window, evaluated every minute, calls the same /heal endpoint through a webhook channel. Whichever arrives first does the work; the second finds the scraper already healthy and exits. A second rule fires when data stops arriving altogether, which is the only signal that catches a pipeline that died before it could run its own detector.
Pulling in what isn't pushed
Port's action and workflow runs and Bright Data's remaining budget are not emitted as telemetry by default. A small background watcher polls both and re-emits them as metrics (port.action.runs, port.workflow.runs and brightdata.budget.remaining), de-duplicated in SQLite so a restart never double counts. The dashboard, the alert and the webhook channel are all created by scripts/bootstrap_signoz.py, and the dashboard is embedded in Port's operator view.
The three correlated signals sent to SigNoz, plus the dashboard and alert.
The unhappy path, on purpose
The clearest proof came from a fault I injected during the hackathon. A scrape run landed with an error, the heal workflow fired, the Triage agent wrote up the symptom, and the repair endpoint ran a real cycle against Bright Data. The proposal came back with a preview of 2 rows where the fixture expected 30. My verifier rejected it. Port recorded the heal event as rolled back, set the scraper to broken, and left it for a person. Nothing bad reached the product. The next clean scheduled run moved health back to healthy on its own.
Opening the SigNoz trace told the whole story: the fetch spans with their row counts, the heal span holding the symptom description, and the verify span whose log read "row count 2, expected 30". No guessing, no digging through separate log files.
If I had let a model approve its own repair, that two row extraction would have been promoted and the product would have shown a nearly empty guide with no alarm going off.
Run it yourself
You need Python 3.11+, Node 20+, Docker, a Bright Data API key, a Port account and a SigNoz instance. A free TMDB key improves the ranking.
git clone https://github.com/Nitishmane/scraper-factory && cd scraper-factory
cp .env.example .env # fill in credentials
make setup # dependencies + OpenTelemetry instrumentation
make bootstrap # Port: blueprints, workflows, agents
python3 scripts/bootstrap_signoz.py # SigNoz: dashboard, drift alert, webhook channel
make serve # factory API + demo site on :8000
make run && make catalog && make rank && make publish
make demo-drift # inject drift and watch the repair loop
make verify # the fixture gate; non-zero exit on mismatch
Bright Data's collectors run in the cloud and can't reach localhost, so the fixtures and the factory's webhooks are exposed through cloudflared tunnels during development. The README covers the full setup.
What I would pass on
- Write prompts about meaning, not layout. The prompt is what the healer works from.
- Capture the raw string next to every parsed value. Raw present but parsed null turned out to be one of my most reliable drift signals.
- Treat a heal preview as a candidate. Check it against known good output before approving it.
- Model the unusual things. Treating scrapers, runs and repairs as entities is what made every other feature useful.
- Keep the agent's job narrow. Triage writes the symptom report; it does not approve its own fix.
- Make the repair endpoint idempotent. With more than one trigger path, they will race.
- Alert on a metric, not a log line. And add an alert for silence.
- Keep the setup in scripts. Re-running one idempotent bootstrap beats clicking through settings at 2 AM during a hackathon.
About the project
Scraper Factory was built for the WeMakeDevs Zero Downtime Hackathon, where it won first place and an NVIDIA DGX Spark. Thanks to Kunal Kushwaha and the WeMakeDevs team for putting on a great event, and to Port, Bright Data and SigNoz for the tools that made it possible.
Live webapp: scraper-factory-top20.vercel.app
Source code: github.com/Nitishmane/scraper-factory
The Scraper Factory
What should I record this week? The Scraper Factory scrapes ~12 days of TV-guide data per channel for Philo, YouTube TV, and Sling, ranks the top 10 upcoming shows and top 10 upcoming movies by TMDB popularity, and gives you one click to record each title on Philo.
That is the product. The engineering thesis underneath it is the real point: web scrapers rot. A guide site ships a redesign, selectors stop matching, and the data either stops arriving or — worse — arrives subtly wrong. This project treats that as a reliability problem with an on-call rotation. Scrapers are catalogued services with health states. Every run is traced. Drift is an alert. Repair is an automated loop that must prove itself against a frozen fixture before its output is trusted, and escalates to a human when it can't.
The same doctrine now runs the whole factory…
Top comments (0)