DEV Community: Daniel Root

Monitor your robots.txt and sitemap for accidental changes

Daniel Root — Sat, 26 Sep 2026 20:21:15 +0000

Two SEO accidents come up again and again:

A staging robots.txt with Disallow: / gets deployed to production.
A migration or plugin update changes the sitemap generator, and whole sections disappear from sitemap.xml.

Neither breaks the site for visitors, so nobody notices. Search engines do notice, just slowly: Google says it generally caches robots.txt for up to 24 hours, and the effect on indexing and traffic can take much longer to show up in reports. By the time someone asks "why is organic down?", the cause is weeks old.

The fix is boring: watch the few files that control crawling, and add a smoke test to your deploys.

What to watch

/robots.txt: plain text, small, and should almost never change. Any change is worth a human look.
Your sitemap index and the child sitemaps that matter. If your sitemap is split (sitemap-pages.xml, sitemap-products.xml...), watch the index plus the ones for your most important sections.
The visible text of 3-5 money pages (home, pricing, top landing pages), which catches a template change that wipes copy.
Other plain-text control files you rely on, such as /ads.txt or /.well-known/security.txt.

What a text-change monitor will not catch

Most change monitors, ours included, compare a page's visible text. That's great for robots.txt and sitemaps, but several indexing controls live in markup or headers, not in text:

<meta name="robots" content="noindex"> (an attribute, not visible text)
the X-Robots-Tag: noindex HTTP header
rel="canonical" and hreflang changes
tags injected by JavaScript after load

For those, use Search Console's indexing reports, a scheduled crawl (Screaming Frog, Sitebulb and similar tools can run on a schedule), or better, a check that runs on every deploy.

A post-deploy SEO smoke test

Drop this into your CI after deploy (or run it from cron). It exits non-zero if production looks de-indexed:

#!/usr/bin/env bash
# seo-smoke.sh: fail loudly if production is telling crawlers to go away
set -uo pipefail
site="https://example.com"
pages=(/ /pricing /blog)
fail=0

robots=$(curl -fsS "$site/robots.txt") || { echo "robots.txt not reachable"; fail=1; }
if grep -Eiq '^[[:space:]]*disallow:[[:space:]]*/[[:space:]]*$' <<<"$robots"; then
  echo "robots.txt contains 'Disallow: /'. Check which user-agent group it's in"
  fail=1
fi

for path in "${pages[@]}"; do
  headers=$(curl -fsSI "$site$path") || { echo "$path: not reachable"; fail=1; continue; }
  body=$(curl -fsS "$site$path")
  if grep -qi '^x-robots-tag:.*noindex' <<<"$headers"; then
    echo "$path: X-Robots-Tag noindex"; fail=1
  fi
  if grep -Eiq '<meta[^>]*name=.?robots[^>]*noindex' <<<"$body"; then
    echo "$path: meta robots noindex"; fail=1
  fi
done

count=$(curl -fsS "$site/sitemap.xml" | grep -o '<loc>' | wc -l)
echo "sitemap.xml lists $count URLs"
if [ "$count" -lt 10 ]; then echo "sitemap looks too small"; fail=1; fi

exit $fail

Adjust before trusting it:

The robots check flags Disallow: / anywhere in the file, including groups for specific bots you meant to block. Treat a hit as "look at this", not proof.
The meta check assumes name comes before content, which is the common order but not guaranteed.
Set the sitemap threshold to something meaningful for your site, e.g. 80% of last week's count. If you use a sitemap index, count the child sitemaps instead.

Between deploys: change alerts

The smoke test covers your deploys. It doesn't cover a CMS plugin update at 2am, a teammate editing robots.txt in an admin panel, or a hosting platform "helpfully" changing defaults. That's where a change monitor on the files above earns its keep.

Some practical notes for sitemaps specifically:

If your generator rewrites <lastmod> on every build, a text monitor will alert on every deploy. Full ISO timestamps like 2026-09-26T10:00:00+00:00 are ignored by monitors that mask timestamps (PingWhen does), but date-only values like 2026-09-26 are not. Watch a child sitemap that changes rarely, or accept a deploy-time alert.
Very large sitemaps (tens of thousands of URLs) are better handled by the count check above than by a diff.

With PingWhen

PingWhen watches public URLs and emails you, or POSTs a signed JSON webhook, when their text changes, when they go down, and when they come back. Because robots.txt is plain text, every edit produces an alert with a short excerpt of the added text. Point the webhook at your CI (anything that can take a JSON POST) and you can re-run the smoke test automatically whenever robots.txt or a sitemap changes.

It's $9/month for 10 URLs checked every 15 minutes, and there's no free plan. You can check whether your files are reachable and readable with our free page check.

Originally published on the PingWhen blog. Written with AI assistance for PingWhen, the page-change and uptime alert tool we make.

How to get notified when a web page changes (without writing a scraper)

Daniel Root — Sat, 26 Sep 2026 20:21:11 +0000

You want to know when a page changes: a "registration opens soon" notice, a competitor's pricing table, a changelog with no RSS, a government page that says "updates will be posted here". Most guides jump straight to a tool. It's worth spending five minutes first, because the right URL matters more than the right tool.

Step 0: check whether a feed already exists

A feed is the most reliable change notification there is, and it's free.

Look in the page source for <link rel="alternate" type="application/rss+xml"> or application/atom+xml.
Try the usual paths: /feed, /rss, /atom.xml, /index.xml.
GitHub has Atom feeds for releases (https://github.com/OWNER/REPO/releases.atom), tags (/tags.atom) and commits (/commits/main.atom).
Subreddits have one at https://www.reddit.com/r/NAME/.rss.

If there's a feed, put it in any feed reader (or an automation tool that emails you new items) and stop here.

Step 1: the view-source test

Open the page, then open its source: Ctrl+U / Cmd+Option+U, or prefix the address with view-source:. Search for the words you care about.

The text is in the source. The page is server-rendered. Any text-based monitor will see what you see, and a simple, cheap tool is enough.
The source is mostly <script> tags and an empty <div id="root">. The page is rendered by JavaScript in the browser. A plain HTTP fetch sees an empty shell.

For JavaScript pages you have two options:

Find the data URL. Open DevTools, go to the Network tab, filter by Fetch/XHR, and reload. The page is usually loading JSON from an API. If that JSON URL works when you open it in a new tab (no login cookie needed), watch the JSON instead of the page. It's smaller, it changes only when the data changes, and many monitors handle JSON well.
Use a monitor that runs a real browser (Visualping, PageCrawl, or self-hosted changedetection.io with its Playwright fetcher). These are heavier and usually cost more per check, but they work on single-page apps.

Step 2: pick the narrowest URL you can

Whatever tool you use, noise usually comes from the page, not the tool. Before you set anything up:

Prefer the specific page (one product, one notice, one job posting) over a homepage or listing.
Look for a print view, a "text only" version, or a filtered URL (?category=...) that shows only what you need.
Watch out for text that changes on every load: "posted 5 minutes ago", rotating testimonials, view counters, "people are looking at this now". A monitor that compares the whole page will alert on every one of those. Either pick a different URL or use a tool that lets you select a region.
Markup-only churn (cache-busting file names, CSRF tokens in attributes, analytics snippets) is harmless for monitors that compare visible text. It's a problem for ones that compare raw HTML.

Step 3: choose how you want to be alerted

There are three broad families. None is "best"; they trade money for effort differently.

Approach	Examples	Good at	Watch out for
Browser extension, local checks	Distill (local monitors)	Free, can see pages you're logged into	Only checks while your browser is running
Self-hosted	urlwatch, changedetection.io	Free and flexible (selectors, filters, JS rendering)	You run and update it, and it's down when your box is
Hosted service	Visualping, PageCrawl, ChangeTower, PingWhen	Checks while your laptop is closed, no maintenance	Monthly cost; free tiers are usually small

If you're comfortable with a terminal, the smallest possible self-hosted version is a cron job. This one hashes the page's text and emails you when the hash changes:

#!/usr/bin/env bash
# watch.sh URL: run from cron, e.g. */15 * * * * /home/me/watch.sh https://example.com/notice
set -euo pipefail
url="$1"
state="$HOME/.pagewatch/$(printf '%s' "$url" | sha256sum | cut -c1-16)"
mkdir -p "$(dirname "$state")"

new=$(curl -fsSL --max-time 20 -A 'pagewatch/1.0' "$url" \
  | sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' \
  | tr -s '[:space:]' ' ' \
  | sha256sum | cut -d' ' -f1)

old=$(cat "$state" 2>/dev/null || true)
if [ -n "$old" ] && [ "$new" != "$old" ]; then
  echo "$url changed" | mail -s "Page changed: $url" you@example.com
fi
echo "$new" > "$state"

Honest caveats: the sed is crude (a <script> block that spans several lines survives it), it needs a working mail command, and because of curl -f it silently skips a run when the site is down instead of telling you. It's a fine starting point for one or two pages. Past that, a real tool pays for itself in time.

Step 4: expect a baseline, then tune

Every monitor has to store a first copy before it can tell you anything changed, so the first check is silent. If you then get alerts that don't matter, go back to Step 2 before you blame the tool. A narrower URL fixes most noise.

Where PingWhen fits

We built PingWhen for the simple case: a public, server-rendered page (or a JSON URL) where you just want an email or a webhook when the text changes, plus a "down" and "back up" alert on the same URL. It compares the page's visible text (scripts and styles removed, ISO timestamps masked, JSON keys sorted) and sends one alert with a short excerpt of what's new. It doesn't render JavaScript or let you pick a CSS selector, and there's no free plan: it's $9/month for 10 pages checked every 15 minutes.

Not sure whether a page is a good candidate? Our free page check fetches it once, the way our watcher would, and tells you whether it's up and whether we can see its text.

Originally published on the PingWhen blog. Written with AI assistance for PingWhen, the page-change and uptime alert tool we make.