DEV Community

Robin Beraud-Sudreau
Robin Beraud-Sudreau

Posted on

Find Broken Pages on Your Site From Its Sitemap, With One Python File

#hermesagent #ai #opensource #web

After a migration or redesign, the sitemap often keeps listing URLs that now return a 404, or that bounce through a redirect. Search engines notice before you do, and so do visitors.

Online link checkers work, but many cap the page count, want an account, or can't be scripted. I wanted something I could run from a terminal or a cron job and trust to fail loudly.

So I wrote a Hermes skill, sitemap-link-checker, around a single standard-library Python script. It reads a sitemap (plain, gzipped, or an index of sitemaps), requests every URL with a HEAD call, and falls back to GET when a server refuses HEAD. It then lists broken URLs and redirects. With --links, it also opens each page and checks the outgoing links, reporting which page each dead link was found on.

python3 scripts/sitemap_check.py https://example.com/sitemap.xml --links --max-urls 50
Enter fullscreen mode Exit fullscreen mode 
Checked 3 sitemap URLs: 1 broken, 1 redirected
  BROKEN 404 https://example.com/missing
  REDIRECT 301 https://example.com/old -> https://example.com/ok
  DEAD LINK 404 https://example.com/dead-link (on https://example.com/ok)
Enter fullscreen mode Exit fullscreen mode

The exit code is 1 when something is broken, so it plugs straight into a scheduled job. There's a --selftest that spins up a local server with a 404, a redirect and a dead link, and asserts all three are caught.

It has limits: it only sees what the sitemap lists, ignores JavaScript-rendered links, and some sites answer 403 to scripts. Only run it on sites you're allowed to test.

The skill is MIT licensed, in the hermes-skills library.

Skill (MIT): https://github.com/RobinBeraud/hermes-skills/tree/main/web/sitemap-link-checker
All skills: https://github.com/RobinBeraud/hermes-skills

This post was written with AI assistance and reviewed through automated checks.

Top comments (0)