Broken Link Checker in 50 Lines of Node.js — CheerioCrawler Approach

#webdev #seo #javascript #node

Broken links kill SEO and user experience. Here's a minimal broken link checker using CheerioCrawler.

The Approach

import { CheerioCrawler } from "crawlee";

const broken = [];
const crawler = new CheerioCrawler({
  maxRequestRetries: 1,
  async requestHandler({ $, request, response, enqueueLinks }) {
    if (response.statusCode >= 400) {
      broken.push({
        url: request.url,
        status: response.statusCode,
        foundOn: request.userData.parentUrl
      });
      return;
    }
    // Follow internal links
    await enqueueLinks({
      strategy: "same-domain",
      userData: { parentUrl: request.url }
    });
  },
  failedRequestHandler({ request }) {
    broken.push({ url: request.url, error: "Connection failed" });
  }
});

await crawler.addRequests([{ url: "https://your-site.com" }]);
await crawler.run();
console.log(`Found ${broken.length} broken links`);