DEV Community

Daniel Root
Daniel Root

Posted on Originally published at pingwhen.app

How to get notified when a web page changes (without writing a scraper)

You want to know when a page changes: a "registration opens soon" notice, a competitor's pricing table, a changelog with no RSS, a government page that says "updates will be posted here". Most guides jump straight to a tool. It's worth spending five minutes first, because the right URL matters more than the right tool.

Step 0: check whether a feed already exists

A feed is the most reliable change notification there is, and it's free.

  • Look in the page source for <link rel="alternate" type="application/rss+xml"> or application/atom+xml.
  • Try the usual paths: /feed, /rss, /atom.xml, /index.xml.
  • GitHub has Atom feeds for releases (https://github.com/OWNER/REPO/releases.atom), tags (/tags.atom) and commits (/commits/main.atom).
  • Subreddits have one at https://www.reddit.com/r/NAME/.rss.

If there's a feed, put it in any feed reader (or an automation tool that emails you new items) and stop here.

Step 1: the view-source test

Open the page, then open its source: Ctrl+U / Cmd+Option+U, or prefix the address with view-source:. Search for the words you care about.

  • The text is in the source. The page is server-rendered. Any text-based monitor will see what you see, and a simple, cheap tool is enough.
  • The source is mostly <script> tags and an empty <div id="root">. The page is rendered by JavaScript in the browser. A plain HTTP fetch sees an empty shell.

For JavaScript pages you have two options:

  1. Find the data URL. Open DevTools, go to the Network tab, filter by Fetch/XHR, and reload. The page is usually loading JSON from an API. If that JSON URL works when you open it in a new tab (no login cookie needed), watch the JSON instead of the page. It's smaller, it changes only when the data changes, and many monitors handle JSON well.
  2. Use a monitor that runs a real browser (Visualping, PageCrawl, or self-hosted changedetection.io with its Playwright fetcher). These are heavier and usually cost more per check, but they work on single-page apps.

Step 2: pick the narrowest URL you can

Whatever tool you use, noise usually comes from the page, not the tool. Before you set anything up:

  • Prefer the specific page (one product, one notice, one job posting) over a homepage or listing.
  • Look for a print view, a "text only" version, or a filtered URL (?category=...) that shows only what you need.
  • Watch out for text that changes on every load: "posted 5 minutes ago", rotating testimonials, view counters, "people are looking at this now". A monitor that compares the whole page will alert on every one of those. Either pick a different URL or use a tool that lets you select a region.
  • Markup-only churn (cache-busting file names, CSRF tokens in attributes, analytics snippets) is harmless for monitors that compare visible text. It's a problem for ones that compare raw HTML.

Step 3: choose how you want to be alerted

There are three broad families. None is "best"; they trade money for effort differently.

Approach Examples Good at Watch out for
Browser extension, local checks Distill (local monitors) Free, can see pages you're logged into Only checks while your browser is running
Self-hosted urlwatch, changedetection.io Free and flexible (selectors, filters, JS rendering) You run and update it, and it's down when your box is
Hosted service Visualping, PageCrawl, ChangeTower, PingWhen Checks while your laptop is closed, no maintenance Monthly cost; free tiers are usually small

If you're comfortable with a terminal, the smallest possible self-hosted version is a cron job. This one hashes the page's text and emails you when the hash changes:

#!/usr/bin/env bash
# watch.sh URL: run from cron, e.g. */15 * * * * /home/me/watch.sh https://example.com/notice
set -euo pipefail
url="$1"
state="$HOME/.pagewatch/$(printf '%s' "$url" | sha256sum | cut -c1-16)"
mkdir -p "$(dirname "$state")"

new=$(curl -fsSL --max-time 20 -A 'pagewatch/1.0' "$url" \
  | sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' \
  | tr -s '[:space:]' ' ' \
  | sha256sum | cut -d' ' -f1)

old=$(cat "$state" 2>/dev/null || true)
if [ -n "$old" ] && [ "$new" != "$old" ]; then
  echo "$url changed" | mail -s "Page changed: $url" you@example.com
fi
echo "$new" > "$state"
Enter fullscreen mode Exit fullscreen mode

Honest caveats: the sed is crude (a <script> block that spans several lines survives it), it needs a working mail command, and because of curl -f it silently skips a run when the site is down instead of telling you. It's a fine starting point for one or two pages. Past that, a real tool pays for itself in time.

Step 4: expect a baseline, then tune

Every monitor has to store a first copy before it can tell you anything changed, so the first check is silent. If you then get alerts that don't matter, go back to Step 2 before you blame the tool. A narrower URL fixes most noise.

Where PingWhen fits

We built PingWhen for the simple case: a public, server-rendered page (or a JSON URL) where you just want an email or a webhook when the text changes, plus a "down" and "back up" alert on the same URL. It compares the page's visible text (scripts and styles removed, ISO timestamps masked, JSON keys sorted) and sends one alert with a short excerpt of what's new. It doesn't render JavaScript or let you pick a CSS selector, and there's no free plan: it's $9/month for 10 pages checked every 15 minutes.

Not sure whether a page is a good candidate? Our free page check fetches it once, the way our watcher would, and tells you whether it's up and whether we can see its text.


Originally published on the PingWhen blog. Written with AI assistance for PingWhen, the page-change and uptime alert tool we make.

Top comments (1)

Collapse
 
devsupport profile image
Dev Support •

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‌‍