Find broken links on any website with Python, grouped by the page you need to fix (2026)

#seo #webdev #python #tutorial

A broken-link report is only useful if it tells you where to go to fix things. "This URL returns 404" is half the answer. "These three pages link to it" is the part you can act on. Most quick scripts check one page at a time and lose that mapping, or they check the same footer link 50 times.

This post builds a small crawler that does it properly: crawl the site, collect every outgoing URL once, check each one once, and write a CSV that maps every broken URL back to the pages that contain it. Then it looks at a real run, because the interesting part of link checking is the false positives.

The approach

Breadth-first crawl of same-site pages. Start at the homepage, follow <a href> links that stay on the same host, stop at a page limit.
Collect every outgoing URL with its source pages. Links, images, scripts and stylesheets all count. A missing image or script is as broken as a dead link. Strip #fragments so /docs#install and /docs#usage are one check.
Check each unique URL once, in parallel. Use GET with stream=True instead of HEAD. Plenty of servers answer HEAD with 405 or 404 even when the page is fine, and some just hang on it. A streamed GET reads only the headers and closes the connection.
Retry once before calling something broken. Transient timeouts are common.

The script

# Usage: python3 broken_links.py https://example.com [max_pages]
# Crawls same-site pages (BFS), checks every link and image once, writes broken.csv grouped by source page.
import csv, sys, time, collections
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urljoin, urldefrag, urlparse
import requests
from bs4 import BeautifulSoup

start = sys.argv[1]
max_pages = int(sys.argv[2]) if len(sys.argv) > 2 else 50
host = urlparse(start).netloc
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0 (compatible; link-check-tutorial/1.0)"

def check(url):
    # GET with stream=True reads headers only; many servers answer HEAD wrongly (405/404) or hang.
    for attempt in range(2):
        try:
            r = s.get(url, stream=True, timeout=15, allow_redirects=True)
            r.close()
            return r.status_code, r.url
        except requests.RequestException as e:
            err = type(e).__name__
            time.sleep(2)
    return err, url

sources = collections.defaultdict(set)   # link -> pages that contain it
queue, seen_pages, i = [start], {start}, 0
t = time.time()
while i < len(queue) and i < max_pages:
    page = queue[i]; i += 1
    try:
        r = s.get(page, timeout=20)
    except requests.RequestException:
        continue
    if "text/html" not in r.headers.get("content-type", ""):
        continue
    soup = BeautifulSoup(r.text, "html.parser")
    for tag, attr in (("a", "href"), ("img", "src"), ("link", "href"), ("script", "src")):
        for el in soup.find_all(tag):
            raw = el.get(attr)
            if not raw or raw.startswith(("mailto:", "tel:", "javascript:", "data:", "#")):
                continue
            link = urldefrag(urljoin(r.url, raw))[0]
            if not link.startswith("http"):
                continue
            sources[link].add(page)
            if tag == "a" and urlparse(link).netloc == host and link not in seen_pages:
                seen_pages.add(link); queue.append(link)

links = sorted(sources)
with ThreadPoolExecutor(10) as pool:
    results = dict(zip(links, pool.map(check, links)))

broken = [(u, st) for u, (st, _) in results.items() if not (isinstance(st, int) and st < 400)]
with open("broken.csv", "w", newline="") as f:
    w = csv.writer(f); w.writerow(["source_page", "broken_url", "status"])
    for u, st in sorted(broken):
        for p in sorted(sources[u]):
            w.writerow([p, u, st])
print(f"{i} pages crawled, {len(links)} unique URLs checked, {len(broken)} broken in {time.time() - t:.0f} s -> broken.csv")
for u, st in sorted(broken, key=lambda x: str(x[1]))[:15]:
    print(f"{str(st):<16} {u}  (on {len(sources[u])} page(s))")

Install the two dependencies with pip install requests beautifulsoup4 and run python3 broken_links.py https://your-site.com 200.

Results on crawlee.dev

I ran python3 broken_links.py https://crawlee.dev 50 on October 20, 2026:

50 pages crawled, 3,008 unique URLs checked in 245 seconds, 10 checks in parallel.
35 URLs flagged. That number is misleading, which is the main lesson of this post.

Here is what the 35 actually were:

Status	Count	Really broken?
403	14	No. Stack Overflow, npm, Crunchbase, Upwork, OpenAI and others block scripted requests. The Stack Overflow tag link alone appears on 37 pages.
999	2	No. LinkedIn's "go away, bot" status.
Connect timeout	7	No. All were plain `http://` links, and my test machine's network blocks outbound port 80. Worth knowing if you run this from a locked-down server or CI runner.
404	8	7 yes, 1 no (see below).
SSL error	4	Yes.

So 11 of the 35 were genuinely broken, and every one of them is a real typo or a bad link:

https://www.crawlee.dev/python/docs/examples/playwright-crawler-with-fingeprint-generator on a blog post, with "fingeprint" missing an "r".
Three links like https://github.com//github.com/apify/crawlee/pull/1890/..., where a full URL got pasted into a field that already prefixed https://github.com/.
https://www.en.wikipedia.org/wiki/Robots.txt and https://www.developer.mozilla.org/...: an extra www. in front of hosts that don't have one, so the TLS certificate doesn't match.
A github.com/renovate%5Bbot%5D profile link (bot accounts have no profile page) and two SwaggerHub API docs that no longer exist.

The one 404 that wasn't broken was https://5JC94MPMLY-dsn.algolia.net, a <link rel="preconnect"> to the search provider. Its root path 404s, but nobody ever navigates there.

Things that will bite you

Treat 401, 403, 429 and 999 as "blocked", not "broken". Report them in a separate column, or re-check them from a real browser. If you count them as broken, your report is mostly noise and people stop reading it.
Skip preconnect and dns-prefetch links, or at least don't treat them as pages. They point at hosts, not documents.
Check from a normal network. Corporate proxies and CI runners often block plain HTTP or whole domains, which shows up as timeouts.
Be polite on your own site. This script checks external URLs 10 at a time but has no per-host limit. On a big site most links point at your own host, so add a per-host concurrency cap and respect robots.txt before you crawl thousands of pages.
Watch the redirect chains. allow_redirects=True hides a 301 to a 404. The final status is what matters for users, but r.history tells you which links are worth updating to skip the redirect.

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this as a hosted tool on Apify: Broken Link Checker. It adds robots.txt handling, a per-host concurrency limit, a spending cap per run, and source-page grouping for every broken link, image and script. It costs $0.50 per 1,000 URLs checked, so Apify's free monthly credit covers small and medium sites, and you can run it on a schedule or call it by API. The script above is the whole idea, though, so use whichever fits.

DEV Community