DEV Community

Cover image for Building a Change & Alert Engine That Webhooks Any Diff on Any URL
Oaida Adrian
Oaida Adrian

Posted on

Building a Change & Alert Engine That Webhooks Any Diff on Any URL

#python #automation #api #webscraping

Most data jobs re-pull the whole source every run and waste time and money on data that didn't move. A change-detection engine wakes up, checks what actually changed, and pushes only the diff. I built a generic one: point it at any URL — a JSON feed, an RSS/Atom feed, or a plain HTML page — and it emits only the changes, with before and after values, to the dataset and/or your webhook.

How it works

  1. Poll — fetch sourceUrl once per run (the interval is your schedule; the actor is a single-shot poll)
  2. Parse — content is sniffed automatically: JSON arrays used directly; object feeds auto-detect common list keys (items, results, data, records…) or take an itemsPath dot path; RSS/Atom parsed with stable ids (guid or link); HTML becomes one monitored item keyed on a content hash
  3. Diff — each item gets a stable item_id (from idField, natural keys like guid/link/url/title, or a content-hash fallback) and a content hash. New id → added; same id, different hash → modified with full before/after; id missing → removed (only with includeRemovals: true)
  4. Emit — changed items go to the dataset, one per item, plus one batched webhook POST per run that has changes. Nothing is emitted on a clean poll.

The delivery guarantee

At-least-once is the detail I care most about: state is persisted only after a successful webhook POST (or immediately when no webhook is configured). If the POST fails, the run FAILS and the next run re-emits the same changes — an alert is never silently lost. Within a run, duplicate item ids collapse to the last occurrence, so a feed that repeats entries won't double-notify.

The persistence trap

Same lesson as the sibling price monitor: Apify's default key-value store is per-run for API-started runs, so state kept there vanishes between runs and the second run re-emits everything as added. The fix is a named store — Actor.open_key_value_store(name='change-alert-state') (keyword-only name= in SDK 3.4, account-scoped, persists across runs).

The smoke test

  • BBC News RSS: run 1 → 41 added (real titles and links); run 2 → 0 items, zero changes ✓
  • Controlled changed source (an Apify dataset with a pre-signed public URL — gist raw CDN served stale content to datacenter IPs, so a dataset is the deterministic route): run A → 2 added; append a modified g2 and a new g3 → run B → modified g2 (price 20 → 25) + added g3; run C → 0 ✓

The honest bits

  • The engine compares current fetch vs last persisted fetch — items that leave the feed are only reported with includeRemovals: true (off by default, because first-page rotation is often noise).
  • JSON items without any natural id fall back to a content hash; any edit then looks like an add+remove pair — set idField for clean modified detection.
  • Feeds with per-item volatile timestamps can look "modified" every poll — list those fields in ignoreFields to hash only what matters.

Try it

👉 Change & Alert Engine on Apify Store

More from me

While you're here, these might be worth a read:

Top comments (0)