DEV Community

Salih Küçükkayış
Salih Küçükkayış

Posted on

Find broken links and dead sitemap URLs without installing a crawler

#seo #webdev #automation #tutorial

Desktop crawlers are great until you need the check to run every week, on a server, for several sites, and land in a spreadsheet. Then you want something you can call from a schedule or an API.

I built two small Apify Actors for this. Disclosure: I am the developer, and both are paid per result (cents per run for a small site).

1. Broken links on the pages people actually see

Broken Link Checker crawls a site, collects every link and image on the pages it visits, and checks each URL once:

{
  "startUrls": [{ "url": "https://crawler-test.com/" }],
  "maxPagesToCrawl": 5,
  "maxLinksToCheck": 150
}
Enter fullscreen mode Exit fullscreen mode

On that test site, 5 pages gave 143 unique URLs and 8 broken ones in under a minute. Each row says which pages link to the broken URL and with what link text, so you know what to edit:

{
  "url": "https://crawler-test.com/robots_protocol/non_200_with_noindex",
  "status": 404,
  "severity": "error",
  "problem": "Not found (404)",
  "foundOn": [{ "page": "https://crawler-test.com/", "text": "Non-200 with Noindex", "tag": "a" }]
}
Enter fullscreen mode Exit fullscreen mode

Details that matter in practice:

  • HEAD first, and every error is confirmed with a GET (many servers answer HEAD wrongly), without downloading the file.
  • 401, 403 and 429 are warnings, not "broken": the page exists but refused a bot.
  • robots.txt is respected; a host that does not answer at all is reported as unreachable, not hidden.
  • With reportRedirects, internal links that go through a 301/302 are listed too. Updating them saves a hop.

2. Pages your sitemap promises but cannot deliver

A sitemap should list only live pages. Sitemap Extractor reads every sitemap (robots.txt, indexes, .gz) and, with checkStatus, checks each URL:

{
  "startUrls": ["https://crawler-test.com/test_sitemap.xml"],
  "checkStatus": true,
  "maxUrlsPerSite": 60
}
Enter fullscreen mode Exit fullscreen mode

18 URLs checked in 9 seconds. Redirects come back with their final URL, robots-blocked pages are marked, dead domains have a checkError.

Schedule it weekly with Changed since = 7 days to check only new or changed pages.

Run it from code 

curl -X POST "https://api.apify.com/v2/acts/saliga~broken-link-checker/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com/"}],"maxPagesToCrawl":50}'
Enter fullscreen mode Exit fullscreen mode

The response is the list of problem links as JSON. Pipe it into a Slack message, a GitHub issue or a sheet.

More copy-paste examples (Python, Node.js, n8n): github.com/msalihk/apify-actor-examples

Top comments (0)