DEV Community

lineagersiui-boop
lineagersiui-boop

Posted on Originally published at github.com

How to find broken URLs in your sitemap (free methods first)

#automation #seo #tutorial

A sitemap lists the URLs you want search engines to index. Entries that return 404, server errors, redirect chains or noindex waste crawl budget and send mixed signals. Check your sitemap at three moments: before a migration, after a CMS update, and on a regular schedule.

1. Free: check a few URLs by hand 

curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" https://example.com/some-page
Enter fullscreen mode Exit fullscreen mode

This works for 5 URLs. It gets tedious at 500.

2. Free: a short script (PowerShell) 

$xml = [xml](Invoke-WebRequest https://example.com/sitemap.xml -UseBasicParsing).Content
$xml.urlset.url.loc | ForEach-Object {
  $u = $_
  try { $r = Invoke-WebRequest $u -Method Head -MaximumRedirection 0 -UseBasicParsing -ErrorAction Stop; "$($r.StatusCode) $u" }
  catch { "$([int]$_.Exception.Response.StatusCode) $u" }
  Start-Sleep -Milliseconds 500
}
Enter fullscreen mode Exit fullscreen mode

The script prints one line per URL: 200 means OK, 3xx means a redirect, and 404 or 5xx means broken. It was tested in Windows PowerShell 5.1 against a public sitemap, a missing page (404) and an HTTP-to-HTTPS redirect (307).

Limits:

  • It doesn't handle sitemap indexes or .xml.gz files.
  • It doesn't follow redirect chains.
  • It doesn't check robots.txt.

Keep requests paced and respect the site's robots.txt.

3. Free desktop crawlers

Several desktop SEO crawlers have free tiers with URL limits, and other free link checkers exist. Check their current limits yourself.

4. Hosted option (I built this)

Broken Link & Sitemap Checker is an Apify Actor (lineagersiui/sitemap-url-health-checker) that does the same job on Apify's servers, and covers the gaps in the script above:

  • What it checks: the HTTP status of every URL in your sitemap. It flags 404s, server errors, redirects, redirect chains, https-to-http downgrades and noindex headers. It accepts sitemap indexes and .xml.gz files, and it honours robots.txt (URLs robots.txt disallows are skipped and not billed).
  • Scope: only URLs listed in the sitemap. It does not follow links inside pages.
  • Permission: real runs need you to tick confirmPermission (you own the site or may check it).
  • Pricing: pay-per-event, $0.0005 per URL checked (1,000 URLs = $0.50), plus Apify platform usage. Timeouts and network errors are not billed. Leave the input empty for a free demo of 20 URLs.
  • Track record: it is a new Actor with very few users, so there are no customer results to show.

What to do with the results

  1. Fix or remove sitemap entries that return 404 or 410.
  2. Point sitemap entries at the final URL, not at a redirect.
  3. Remove noindex URLs from the sitemap.
  4. Re-run the check on a schedule after releases.

Disclosure: I built the Actor mentioned here and earn its pay-per-event fees. I'm not affiliated with or endorsed by Apify beyond publishing on the Apify Store. This post was drafted with AI assistance under my direction. It first appeared as a guide on GitHub (canonical link above).

Top comments (0)