DEV Community

Cover image for Build a broken-link review list without calling every failed request a dead page
Tim Zinin
Tim Zinin

Posted on

Build a broken-link review list without calling every failed request a dead page

A link checker that labels every unsuccessful request broken creates a noisy repair backlog. A forbidden page may still exist. A rate-limited site may recover later. An external URL that was deliberately excluded was never checked at all. Those states deserve different owners and different next actions.

Broken Link Checker build 0.1.2 collects a bounded set of links from public start pages and classifies supported checks. Its useful output includes the referring pages and the collection state, so an editor can distinguish a confirmed missing target from an operational or policy limitation.

Bound discovery before checking targets

An illustrative input uses a small approved site scope:

{
  "startUrls": ["https://example.com/"],
  "checkExternal": false,
  "maxPages": 5,
  "maxLinks": 20
}
Enter fullscreen mode Exit fullscreen mode

This is a schema example, not a new crawl report. The Actor accepts one to five exact absolute public start URLs. Credentials and fragments are excluded by the input contract. maxPages ranges from one to twenty, while maxLinks ranges from one to two hundred. These values bound discovery and checking rather than promising a complete link graph for an entire website.

checkExternal defaults to false. A discovered off-origin target in that mode is unchecked_external and is not requested. Keep that state visible when exporting a repair list; otherwise an untested target can appear to have passed simply because it never produced an HTTP error.

Separate a completed negative from an incomplete check

The checker uses HEAD and falls back to GET only for HTTP 405 or 501. That distinction avoids treating every failed HEAD as a reason for an unrestricted second fetch. A response in the 2xx range supports a completed successful check, while redirects have their own observed state.

HTTP 404 and 410 are completed negative findings: the checked target was observed as missing at that time. They belong in an editorial repair queue with source evidence. HTTP 401, 403, 429, and robots restrictions instead represent blocked or incomplete checks. They should not be rewritten into confirmed dead links.

The runtime checks robots policy for each origin and paces requests to at most one request per second per host. It does not render a browser. A link visible only after client-side interaction may therefore remain outside discovery. A crawl cap or blocked source is a coverage qualification, not an assertion that no other bad links exist.

Keep referrers with the target

A repair task needs both the target URL and the pages that link to it. The Actor deduplicates discovered links and retains bounded referrer evidence, with up to twenty referrers per link. Preserve that coverage limit rather than presenting the displayed referrer list as exhaustive.

An editorial handoff can group confirmed missing targets, attach the observed status and timestamp, and list the supported referrers for review. A separate operational queue can hold rate limits and access restrictions. unchecked_external belongs in a scope decision: someone must authorize and define an external check before interpreting its health.

For acceptance, use controlled fixtures for a successful target, a 404, a redirect, a 405 that requires GET fallback, and an off-origin link excluded by policy. Check that blocked outcomes and untested external targets do not become broken. These are proposed checks, not newly executed results.

Reconcile results before a repeat

A completed 404 or 410 check is billable just as a completed successful check is; the paid unit is the completed URL check, not a healthy link. Redirect hops and the supported method fallback do not create extra paid URL units within that check. Incomplete outcomes follow their separate free result semantics.

The Actor does not maintain a persistent replay ledger across separate runs. Running the same input again can charge again for the same completed targets. Save the run and delivery evidence, reconcile an uncertain attempt first, and start a later observation only when you intend to refresh the findings. That produces a repair backlog whose states and costs remain understandable.

Cost and source boundary

The live published event prices checked on 20 October 2026 are $0.001 per result-found event and $0.003 per automatic Actor-start event. The start event is memory-scaled: one event per GB, with a minimum of one event. A free result outcome therefore does not make an entire run free. Check the live pricing panel for the rates applicable to your account before execution. These prices describe the installed contract, not measured customer revenue or a new runtime test.

Inspect the Actor, input, and current pricing.

AI-assisted drafting; technical statements reviewed against the installed Actor source and current API metadata. Examples are illustrative unless explicitly identified as saved observations.

Top comments (0)