Check links in Markdown docs with a 60-line Python script

#python #markdown #documentation #testing

Documentation rots quietly. A README links to a blog post that moved, a docs page points to an API reference that was renamed, and nobody notices until a reader complains. If your docs live in Markdown, you can catch most of this with a short Python script that runs locally or in CI.

What the script does

Finds every .md file in a folder
Pulls out inline links [text](url) and bare <https://...> links
Checks relative links against the file system
Checks external links with an HTTP request
Prints a report and exits non-zero if anything is broken

The code

#!/usr/bin/env python3
import re, sys, pathlib, concurrent.futures as cf
import requests

LINK_RE = re.compile(r'\[[^\]]*\]\(([^)\s]+)(?:\s+"[^"]*")?\)|<(https?://[^>]+)>')
HEADERS = {"User-Agent": "Mozilla/5.0 (docs link check)"}
SKIP = ("mailto:", "tel:", "#")

def links_in(path):
    text = path.read_text(encoding="utf-8")
    # ignore fenced code blocks
    text = re.sub(r"```

.*?

```", "", text, flags=re.S)
    for m in LINK_RE.finditer(text):
        yield m.group(1) or m.group(2)

def check_url(url):
    try:
        r = requests.head(url, headers=HEADERS, allow_redirects=True, timeout=10)
        if r.status_code in (403, 405) or r.status_code >= 500:
            r = requests.get(url, headers=HEADERS, allow_redirects=True, timeout=15, stream=True)
        return url, r.status_code
    except requests.RequestException as e:
        return url, type(e).__name__

def main(root):
    root = pathlib.Path(root)
    external, problems = {}, []
    for md in root.rglob("*.md"):
        for link in links_in(md):
            if link.startswith(SKIP):
                continue
            if link.startswith(("http://", "https://")):
                external.setdefault(link, []).append(md)
            else:
                target = (md.parent / link.split("#")[0]).resolve()
                if not target.exists():
                    problems.append((md, link, "missing file"))
    with cf.ThreadPoolExecutor(max_workers=8) as pool:
        for url, status in pool.map(check_url, external):
            if status != 200 and not (isinstance(status, int) and status < 400):
                for md in external[url]:
                    problems.append((md, url, status))
    for md, link, why in problems:
        print(f"{md}: {link} -> {why}")
    return 1 if problems else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "."))

Run it:

pip install requests
python3 check_md_links.py docs/

Why HEAD first, then GET

A HEAD request asks for headers only, which is fast and light on the remote server. Some servers don't support it properly and answer 405 or 403, or even 500, to a HEAD while a normal GET works. Falling back to GET for those codes removes most false alarms. stream=True means the body isn't downloaded.

Relative links matter too

Most broken links in docs aren't external. They're relative links to a file that got renamed: [setup](../install.md) after install.md became installation.md. The script resolves each relative path from the file that contains it and checks it exists. It strips #section anchors first; checking that the heading still exists is a good next step if you need it.

Running it in CI

A GitHub Actions job:

name: docs-links
on:
  pull_request:
    paths: ["docs/**", "README.md"]
  schedule:
    - cron: "0 6 * * 1"
jobs:
  links:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install requests
      - run: python3 check_md_links.py docs/

The pull request trigger catches broken relative links before they merge. The weekly schedule catches external links that broke on their own, which no pull request would ever reveal.

Handling flaky results

External checks will occasionally fail for reasons that have nothing to do with your docs: rate limits (429), sites that block unknown clients, short outages. Some practical rules:

Treat 429 and timeouts as warnings, not failures, on pull requests
Keep an allowlist for domains that always block automated requests
Re-check failures once before reporting them

Where this stops

This covers the Markdown source. It doesn't see links generated by templates, navigation menus, or anything rendered by JavaScript after the site builds. For a published docs site, checking the built HTML pages on a schedule catches those too. That's what linkbruno.com does for whole websites: it crawls on a schedule, checks every link and image, and alerts by email or Slack when one breaks.

DEV Community