DEV Community

Innovative Cookie
Innovative Cookie

Posted on

Find broken links on any website with plain HTTP (Python, free)

#python #seo #webdev #tutorial

Broken links waste crawl budget and frustrate readers. You do not need a desktop SEO suite to find them on a site you own or have permission to audit: a short Python script that fetches pages, collects <a href>, and checks each target is enough for a one-off pass.

This tutorial stays on plain HTTP. It does not bypass captchas, logins or bot protection.

1. Crawl a few pages and collect links

Install once:

pip install requests beautifulsoup4
Enter fullscreen mode Exit fullscreen mode

Save as find_broken_links.py. Start from one URL, stay on the same host, and keep a small page budget:

import time
from collections import deque
from urllib.parse import urljoin, urlparse, urldefrag

import requests
from bs4 import BeautifulSoup

START = "https://example.com/"
MAX_PAGES = 20
DELAY_S = 0.25
UA = "broken-link-audit/1.0 (+https://example.com; personal audit)"

session = requests.Session()
session.headers["User-Agent"] = UA


def same_host(a, b):
    return urlparse(a).netloc.replace("www.", "") == urlparse(b).netloc.replace("www.", "")


def normalize(base, href):
    if not href or href.startswith(("mailto:", "tel:", "javascript:")):
        return None
    absolute = urljoin(base, href)
    absolute, _ = urldefrag(absolute)
    return absolute


def links_on(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    out = []
    for a in soup.find_all("a", href=True):
        url = normalize(page_url, a["href"])
        if url:
            out.append((url, a.get_text(" ", strip=True)[:120] or ""))
    return out


def check(url):
    try:
        r = session.head(url, timeout=20, allow_redirects=True)
        if r.status_code in (405, 501):
            r = session.get(url, timeout=20, allow_redirects=True)
        return r.status_code, r.url
    except requests.RequestException as e:
        return None, str(e)


queue = deque([START])
seen_pages = set()
seen_targets = {}
broken = []

while queue and len(seen_pages) < MAX_PAGES:
    page = queue.popleft()
    if page in seen_pages or not same_host(START, page):
        continue
    seen_pages.add(page)
    time.sleep(DELAY_S)
    try:
        resp = session.get(page, timeout=30)
    except requests.RequestException as e:
        print("page fetch failed", page, e)
        continue
    if "text/html" not in resp.headers.get("content-type", ""):
        continue
    for target, anchor in links_on(resp.text, resp.url):
        if target in seen_targets:
            continue
        time.sleep(DELAY_S)
        code, final = check(target)
        seen_targets[target] = code
        internal = same_host(START, target)
        if code is None or code >= 400:
            broken.append(
                {
                    "url": target,
                    "statusCode": code,
                    "sourceUrl": page,
                    "anchorText": anchor,
                    "linkType": "internal" if internal else "external",
                    "finalUrl": final if isinstance(final, str) and final.startswith("http") else None,
                }
            )
        elif internal and target not in seen_pages:
            queue.append(target)

print(f"pages={len(seen_pages)} unique_links={len(seen_targets)} broken={len(broken)}")
for row in broken[:20]:
    print(row)
Enter fullscreen mode Exit fullscreen mode

Notes that match how a careful auditor behaves:

  • Prefer HEAD, then fall back to GET when the server rejects HEAD with 405 or 501.
  • Space requests to the same host (DELAY_S). Read the site’s robots.txt before a larger crawl and skip disallowed paths.
  • Only audit sites you own or have permission to check.

2. Read the result

Each broken row should answer three questions: which URL failed, which page linked to it, and what the anchor said. That is enough to open the template or CMS entry and fix the href.

Extend the script later with CSV export (csv.DictWriter) or a depth limit. For migrations, run once before cutover and once after, then diff the broken lists.

Hosted option (optional)

I built a small Apify Actor that does the same job as a crawl: start URLs, breadth-first crawl with page and depth caps, internal and optional external checks, HEAD then GET when needed, robots.txt and crawl-delay respected, no browser and no proxy. Each broken occurrence includes sourceUrl, anchorText, statusCode and redirect info. Disclosure: I am the creator.

There is no Actor fee for now — you only pay Apify platform usage (the Free plan includes monthly credit). Keep Max pages to crawl small for tests.

Written with AI assistance. I checked the Actor’s README, input schema and source before publishing; the script above is a standalone free method.

Top comments (0)