How to find broken URLs in your sitemap (free methods first)

lineagersiui-boop — Fri, 30 Oct 2026 05:21:49 +0000

A sitemap lists the URLs you want search engines to index. Entries that return 404, server errors, redirect chains or noindex waste crawl budget and send mixed signals. Check your sitemap at three moments: before a migration, after a CMS update, and on a regular schedule.

1. Free: check a few URLs by hand

curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" https://example.com/some-page

This works for 5 URLs. It gets tedious at 500.

2. Free: a short script (PowerShell)

$xml = [xml](Invoke-WebRequest https://example.com/sitemap.xml -UseBasicParsing).Content
$xml.urlset.url.loc | ForEach-Object {
  $u = $_
  try { $r = Invoke-WebRequest $u -Method Head -MaximumRedirection 0 -UseBasicParsing -ErrorAction Stop; "$($r.StatusCode) $u" }
  catch { "$([int]$_.Exception.Response.StatusCode) $u" }
  Start-Sleep -Milliseconds 500
}

The script prints one line per URL: 200 means OK, 3xx means a redirect, and 404 or 5xx means broken. It was tested in Windows PowerShell 5.1 against a public sitemap, a missing page (404) and an HTTP-to-HTTPS redirect (307).

Limits:

It doesn't handle sitemap indexes or .xml.gz files.
It doesn't follow redirect chains.
It doesn't check robots.txt.

Keep requests paced and respect the site's robots.txt.

3. Free desktop crawlers

Several desktop SEO crawlers have free tiers with URL limits, and other free link checkers exist. Check their current limits yourself.

4. Hosted option (I built this)

Broken Link & Sitemap Checker is an Apify Actor (lineagersiui/sitemap-url-health-checker) that does the same job on Apify's servers, and covers the gaps in the script above:

What it checks: the HTTP status of every URL in your sitemap. It flags 404s, server errors, redirects, redirect chains, https-to-http downgrades and noindex headers. It accepts sitemap indexes and .xml.gz files, and it honours robots.txt (URLs robots.txt disallows are skipped and not billed).
Scope: only URLs listed in the sitemap. It does not follow links inside pages.
Permission: real runs need you to tick confirmPermission (you own the site or may check it).
Pricing: pay-per-event, $0.0005 per URL checked (1,000 URLs = $0.50), plus Apify platform usage. Timeouts and network errors are not billed. Leave the input empty for a free demo of 20 URLs.
Track record: it is a new Actor with very few users, so there are no customer results to show.

What to do with the results

Fix or remove sitemap entries that return 404 or 410.
Point sitemap entries at the final URL, not at a redirect.
Remove noindex URLs from the sitemap.
Re-run the check on a schedule after releases.

Disclosure: I built the Actor mentioned here and earn its pay-per-event fees. I'm not affiliated with or endorsed by Apify beyond publishing on the Apify Store. This post was drafted with AI assistance under my direction. It first appeared as a guide on GitHub (canonical link above).

Your scraper says SUCCEEDED but the data is empty: three ways to catch it

lineagersiui-boop — Fri, 30 Oct 2026 01:43:01 +0000

Your scraper run says SUCCEEDED. The dataset has 3 items instead of 3,000, or every price field is empty, or the results are a cookie-consent page. Nothing alerted you, because the run status only says the process finished.

Here are three ways to catch this, cheapest first.

1. Use Apify's free monitoring first

Each Actor or task has a Monitoring tab. You can set alerts on result-count thresholds and run status. Alerts on dataset field statistics need a dataset schema. Apify also provides a free Monitoring Actor suite for validating content and checking duplicates, and its docs state the monitoring system is free for all users.

Where fixed thresholds struggle:

the item count varies a lot from run to run
you run Store Actors that have no dataset schema
you monitor many Actors and can't tune each threshold

2. Compare each run to its own history

If a fixed threshold doesn't fit, compare the latest run with the median of recent successful runs, and check that the fields you rely on aren't empty in too many sampled items.

The guide on GitHub has a small Node.js script that does this with three API calls (list runs, get dataset metadata, fetch the first 200 items). It exits with 1 for unhealthy data and 2 when the checker itself is broken, so a scheduler can tell the two apart. Use a read-only token stored as a secret.

Caveats, as stated in the guide:

the median needs at least three finished successful runs
check tasks separately
only the first 200 items are sampled
these are heuristics, not proof

I'm not pasting the script here. I haven't re-run it for this post, so read it and its tests in the repo before relying on it.

3. A ready-made Actor

There are third-party Actors with similar checks, such as Output Completeness Monitor and Dataset Drift Monitor. I built one too, Scraper Output Guard. It adds schema drift, duplicate detection and item-limit checks, at $0.005 per Actor checked.

Takeaway

Turn on the free monitoring first. If counts vary or you run many Actors, compare each run to its own history and check the fields you depend on. Alert on content, not just status.

Full guide with the script: https://github.com/lineagersiui-boop/apify-output-checks/tree/main/guides/silent-scraper-failures

Disclosure: I built the Apify Actor mentioned here and earn its pay-per-event fees. I'm not affiliated with Apify beyond publishing on the Apify Store. This post was drafted with AI assistance under my direction. It first appeared as a guide on GitHub (canonical link above).

DEV Community: lineagersiui-boop