How I Built a Broken Link Scanner for Large Websites

A broken link checker sounds like a weekend project.

Fetch a page, collect every <a href>, request each URL, and report anything that returns 404.

That approach works surprisingly well—until you point it at a real website.

Large websites introduce duplicate URLs, redirects, external domains, missing assets, rate limits, malformed HTML, CSS imports, network failures, and thousands of pages competing for the same resources. At that point, a link checker stops being a script and starts becoming a distributed crawling system.

This is how I built one using Laravel as the control plane and Go as the scanning engine.

The architecture

I wanted Laravel to handle the application side of the product:

• Users and authentication
• Projects and scan settings
• Usage limits
• Scheduled scans
• Result storage
• Progress updates
• Reports and exports

Go was a better fit for the network-heavy part: fetching many URLs concurrently while keeping memory usage and concurrency under control.

The final flow looks roughly like this:

User starts a scan
        ↓
Laravel creates a queued scan
        ↓
A Go worker leases the scan
        ↓
The worker crawls and checks URLs concurrently
        ↓
Results and progress are sent back in batches
        ↓
Laravel stores the data and builds the report

The Go workers are independent processes. They register with the application, send heartbeats, and ask Laravel for available jobs.

Laravel does not need to start a new operating-system process for every scan. A worker leases a queued scan through an API, processes it, and periodically reports its progress.

This made it possible to add more scanner servers without changing the web application.

A queue is more than a list of URLs

At first, my crawler had a basic queue:

1. Add the starting URL.
2. Download it.
3. extract its links.
4. Add new internal URLs to the queue.
5. Repeat until the queue is empty.

The difficult part was defining what “new” meant.

These URLs might all point to the same resource:

https://example.com/about
https://example.com/about#team
https://EXAMPLE.com/about
https://example.com:443/about
/about

Without normalization, the crawler may request the same page several times or even enter a loop.

Every discovered URL therefore passes through a normalization step. Relative URLs are resolved against the current document, fragments are removed, hosts are normalized, and default ports are handled consistently.

I still keep the original value found in the document. The normalized URL is useful for deduplication, but the original value is much more useful when someone needs to fix the link.

The crawler also separates discovery from traversal.

Internal pages can be added to the crawl frontier. External links are checked, but their pages are not recursively crawled. Otherwise, scanning one website could slowly turn into scanning the entire web.

Why parallel scanning needs limits

Checking URLs sequentially is polite but painfully slow. On a large website, network latency dominates the scan time.

Go makes it easy to create a goroutine for every URL, but unlimited concurrency is not a real solution. It can overwhelm both the scanner and the website being scanned.

I ended up using bounded concurrency at several levels:

• A limit on simultaneous scans per worker
• A worker pool inside each scan
• A global limit on active HTTP requests
• Per-host concurrency limits
• Optional delays between requests to the same host

Per-host limits are especially important.

A page may contain resources from the main domain, a CDN, an analytics provider, and several external websites. The scanner can process different hosts in parallel without sending an unreasonable number of requests to any single server.

The crawler also reacts to temporary failures. Responses such as 429 Too Many Requests and some 5xx errors should not immediately become permanent broken-link reports. They may require a retry, a delay, or temporary backoff for that host.

Concurrency is not only a performance setting. It is part of the crawler’s behavior toward other websites.

HTTP status codes are not binary

One of the earliest mistakes was treating every response outside the 2xx range as broken.

In practice, the result is more nuanced:

• 2xx usually means the resource is available.
• 404 and 410 are strong broken-link signals.
• 3xx requires inspecting the complete redirect chain.
• 401 and 403 may mean the resource exists but rejects the scanner.
• 429 often means the scanner should slow down.
• 5xx may be a temporary server failure.
• DNS, TLS, connection, and timeout errors need separate classifications.

A useful report should distinguish a confirmed broken URL from an inconclusive check.

For example, an external website may return 403 to automated clients while loading normally in a browser. Marking that URL as definitely broken would create a false positive and reduce trust in the entire report.

Instead of storing only is_broken, I found it more useful to preserve context:

• HTTP status
• Error category
• Content type
• Request duration
• Whether the link is internal or external
• Verification state
• Final URL
• Redirect history

The report can then prioritize clear failures while still showing ambiguous cases separately.

Redirects deserve first-class treatment

Redirects are not just an implementation detail.

A URL may return 301, lead to another redirect, cross to a different hostname, and finally return 200. Technically it works, but it may still be worth updating the original link.

There are also redirect loops, excessively long chains, malformed Location headers, and redirects to unsupported URL schemes.

For every redirect, the scanner records the status and destination. That makes it possible to show a chain such as:

/old-page [301]
→ /new-page [302]
→ https://www.example.com/new-page [200]

This is much more actionable than reporting only the final status.

Redirect targets also have to go through the same safety and scope checks as directly discovered URLs. Validating only the first URL is not enough because a public URL can redirect somewhere the scanner should never access.

Links are not limited to <a href>

A website can have no broken navigation links and still load incorrectly because an image, stylesheet, font, or script is missing.

The scanner therefore looks at several kinds of references in HTML, including:

• Navigation links
• Images and responsive srcset values
• Stylesheets
• Script sources
• Iframes
• Audio and video resources
• Inline style URLs

CSS needs its own extraction step. A stylesheet may contain more resources through:

@import url("/theme.css");

.hero {
    background-image: url("/images/hero.webp");
}

@font-face {
    src: url("/fonts/site.woff2");
}

Those resources are easy to miss if the scanner only parses HTML.

JavaScript requires a careful boundary. The scanner checks referenced JavaScript files as resources, but it does not try to execute arbitrary application code or statically understand every URL constructed at runtime. Doing that reliably would require a browser-based rendering layer and would significantly change the cost and behavior of a scan.

Being explicit about that limitation is better than pretending a traditional crawler can see everything a browser can.

Keeping track of where a link came from

Knowing that a URL is broken is only half of the answer.

The next question is always:

Where do I need to fix it?

The same missing image or outdated URL may appear on hundreds of pages. Storing a separate complete result for every occurrence creates a lot of repetitive data, while keeping only the target URL loses the source information.

My solution was to treat the checked URL and its occurrences separately.

A result represents the target that was checked. It can also contain source information such as:

• The page where it was discovered
• The original URL value
• The HTML element and attribute
• The link text
• The CSS discovery method
• The number of occurrences

This allows the report to say that one missing asset affects 150 pages instead of displaying 150 nearly identical errors.

It also helps prioritize fixes. Repairing a broken global stylesheet is more important than fixing an unused link on an old article.

Sending results in batches

Large scans can produce a lot of data. Sending one API request per checked URL creates unnecessary overhead, while waiting until the entire crawl finishes risks losing everything if the worker stops.

The scanner sends progress and results in batches.

Each progress update can include:

• Newly completed results
• Current URL
• Number of checked and queued URLs
• Skipped URLs
• Crawl metrics
• A checkpoint of the remaining frontier

Laravel validates and stores the batch, updates the scan counters, and broadcasts the latest progress to the interface.

Batching reduced API overhead and made realtime progress inexpensive enough to be useful.

Making scans resumable

Long-running scans will eventually be interrupted.

A worker can restart, a deployment can happen, a network connection can fail, or a scan can exceed its allowed runtime. Starting again from the first page wastes time and generates duplicate traffic.

The crawler periodically saves enough state to continue:

• URLs waiting to be processed
• URLs already visited
• Pending tasks
• Crawl counters
• Per-host backoff state

Because the job is leased rather than permanently assigned, Laravel can recover an interrupted scan and make it available to a worker again.

Checkpointing turned out not to be an optional reliability feature. For large websites, it is part of the normal execution model.

Turning crawl data into a useful report

Raw crawler output is useful for debugging, but it is not yet a product.

The report needs to answer practical questions:

• Which links are definitely broken?
• Which checks were inconclusive?
• Which problems affect the most pages?
• Which URLs redirect?
• Where was each link found?
• Is the target a page, image, stylesheet, script, or another resource?
• Is the problem new, persistent, or resolved since the previous scan?

Laravel handles this part well. It stores the normalized results, calculates summaries, applies filters, generates CSV exports, and provides realtime progress while the Go scanner remains focused on crawling.

This separation also keeps presentation decisions out of the scanner. I can change how an issue is grouped or displayed without modifying the networking engine.

What I learned

The hardest part of building a broken link scanner was not making HTTP requests. It was deciding what every response meant and preserving enough context to make the result actionable.

A few lessons stood out:

1. Normalize URLs early, but keep the original values.
2. Bound concurrency at both the scan and host levels.
3. Treat redirects as data, not just transport behavior.
4. Separate confirmed failures from uncertain responses.
5. Record where every link was discovered.
6. Send incremental batches instead of waiting for completion.
7. Design for interrupted and resumed scans from the beginning.
8. Keep crawling and reporting as separate responsibilities.

A small link-checking script can be written in an afternoon. A scanner that works reliably across large, unpredictable websites needs queueing, backpressure, recovery, careful classification, and a reporting layer.

That engineering journey eventually became more than an internal experiment.

I ended up turning the scanner into BrokenLinks.pro — a tool for scanning websites, following redirects, checking assets, and turning crawl results into a practical report.