DEV Community

Daniel Root
Daniel Root

Posted on Originally published at pingwhen.app

Monitor your robots.txt and sitemap for accidental changes

#seo #webdev #monitoring #devops

Two SEO accidents come up again and again:

  1. A staging robots.txt with Disallow: / gets deployed to production.
  2. A migration or plugin update changes the sitemap generator, and whole sections disappear from sitemap.xml.

Neither breaks the site for visitors, so nobody notices. Search engines do notice, just slowly: Google says it generally caches robots.txt for up to 24 hours, and the effect on indexing and traffic can take much longer to show up in reports. By the time someone asks "why is organic down?", the cause is weeks old.

The fix is boring: watch the few files that control crawling, and add a smoke test to your deploys.

What to watch

  • /robots.txt: plain text, small, and should almost never change. Any change is worth a human look.
  • Your sitemap index and the child sitemaps that matter. If your sitemap is split (sitemap-pages.xml, sitemap-products.xml...), watch the index plus the ones for your most important sections.
  • The visible text of 3-5 money pages (home, pricing, top landing pages), which catches a template change that wipes copy.
  • Other plain-text control files you rely on, such as /ads.txt or /.well-known/security.txt.

What a text-change monitor will not catch

Most change monitors, ours included, compare a page's visible text. That's great for robots.txt and sitemaps, but several indexing controls live in markup or headers, not in text:

  • <meta name="robots" content="noindex"> (an attribute, not visible text)
  • the X-Robots-Tag: noindex HTTP header
  • rel="canonical" and hreflang changes
  • tags injected by JavaScript after load

For those, use Search Console's indexing reports, a scheduled crawl (Screaming Frog, Sitebulb and similar tools can run on a schedule), or better, a check that runs on every deploy.

A post-deploy SEO smoke test

Drop this into your CI after deploy (or run it from cron). It exits non-zero if production looks de-indexed:

#!/usr/bin/env bash
# seo-smoke.sh: fail loudly if production is telling crawlers to go away
set -uo pipefail
site="https://example.com"
pages=(/ /pricing /blog)
fail=0

robots=$(curl -fsS "$site/robots.txt") || { echo "robots.txt not reachable"; fail=1; }
if grep -Eiq '^[[:space:]]*disallow:[[:space:]]*/[[:space:]]*$' <<<"$robots"; then
  echo "robots.txt contains 'Disallow: /'. Check which user-agent group it's in"
  fail=1
fi

for path in "${pages[@]}"; do
  headers=$(curl -fsSI "$site$path") || { echo "$path: not reachable"; fail=1; continue; }
  body=$(curl -fsS "$site$path")
  if grep -qi '^x-robots-tag:.*noindex' <<<"$headers"; then
    echo "$path: X-Robots-Tag noindex"; fail=1
  fi
  if grep -Eiq '<meta[^>]*name=.?robots[^>]*noindex' <<<"$body"; then
    echo "$path: meta robots noindex"; fail=1
  fi
done

count=$(curl -fsS "$site/sitemap.xml" | grep -o '<loc>' | wc -l)
echo "sitemap.xml lists $count URLs"
if [ "$count" -lt 10 ]; then echo "sitemap looks too small"; fail=1; fi

exit $fail
Enter fullscreen mode Exit fullscreen mode

Adjust before trusting it:

  • The robots check flags Disallow: / anywhere in the file, including groups for specific bots you meant to block. Treat a hit as "look at this", not proof.
  • The meta check assumes name comes before content, which is the common order but not guaranteed.
  • Set the sitemap threshold to something meaningful for your site, e.g. 80% of last week's count. If you use a sitemap index, count the child sitemaps instead.

Between deploys: change alerts

The smoke test covers your deploys. It doesn't cover a CMS plugin update at 2am, a teammate editing robots.txt in an admin panel, or a hosting platform "helpfully" changing defaults. That's where a change monitor on the files above earns its keep.

Some practical notes for sitemaps specifically:

  • If your generator rewrites <lastmod> on every build, a text monitor will alert on every deploy. Full ISO timestamps like 2026-09-26T10:00:00+00:00 are ignored by monitors that mask timestamps (PingWhen does), but date-only values like 2026-09-26 are not. Watch a child sitemap that changes rarely, or accept a deploy-time alert.
  • Very large sitemaps (tens of thousands of URLs) are better handled by the count check above than by a diff.

With PingWhen

PingWhen watches public URLs and emails you, or POSTs a signed JSON webhook, when their text changes, when they go down, and when they come back. Because robots.txt is plain text, every edit produces an alert with a short excerpt of the added text. Point the webhook at your CI (anything that can take a JSON POST) and you can re-run the smoke test automatically whenever robots.txt or a sitemap changes.

It's $9/month for 10 URLs checked every 15 minutes, and there's no free plan. You can check whether your files are reachable and readable with our free page check.

Originally published on the PingWhen blog. Written with AI assistance for PingWhen, the page-change and uptime alert tool we make.

Top comments (1)

Collapse
 
devsupport profile image
Dev Support •

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‌​‍