DEV Community

linkbruno.com
linkbruno.com

Posted on Edited on

How to find broken links inside PDFs with Python (a script for a whole folder of whitepapers, manuals and invoices)

Scheduled broken-link and redirect checks that tell you the moment a link breaks — linkbruno.com

PDFs are where links go to die quietly. A whitepaper from 2022, a product manual, a conference handout, a signed contract with a "see our terms at..." line. The file gets downloaded, emailed, attached to tickets and stored in shared drives for years. Every URL inside it is frozen at the moment it was exported.

And almost no link checker looks inside them. Crawlers follow HTML links, see a .pdf, check that the PDF itself returns 200, and move on. The 40 links inside it never get tested.

Here's a short Python script that opens every PDF in a folder, pulls out the links, checks them, and tells you which page each broken one is on.

Two kinds of links in a PDF

A PDF can contain a URL in two different ways, and you need both:

  1. Link annotations. The clickable areas. In the file, each page has an /Annots array; link annotations have /Subtype /Link and an action dictionary with a /URI. This is what you get when you export a document with hyperlinks from Word, Google Docs or LaTeX.
  2. Bare URLs in the text. A URL typed as plain text, which some PDF viewers make clickable on the fly and others don't. Readers still copy and paste them.

The script extracts both.

Setup

python3 -m venv .venv && . .venv/bin/activate
pip install pypdf requests
Enter fullscreen mode Exit fullscreen mode

The script

#!/usr/bin/env python3
"""Find broken links in every PDF in a folder."""
import re, sys, pathlib
import requests
from pypdf import PdfReader

URL_RE = re.compile(r'https?://[^\s<>"\)\]]+')
session = requests.Session()
session.headers["User-Agent"] = "pdf-link-check/1.0"
cache = {}

def links_in(path):
    reader = PdfReader(path)
    for n, page in enumerate(reader.pages, 1):
        for annot in page.get("/Annots") or []:
            a = annot.get_object()
            if a.get("/Subtype") != "/Link":
                continue
            action = a.get("/A")
            uri = action.get_object().get("/URI") if action else None
            if uri:
                yield n, str(uri), "link"
        text = page.extract_text() or ""
        for m in URL_RE.finditer(text):
            yield n, m.group(0).rstrip(".,;:"), "text"

def status(url):
    if url in cache:
        return cache[url]
    try:
        r = session.head(url, allow_redirects=True, timeout=15)
        if r.status_code in (403, 405) or r.status_code >= 500:
            r = session.get(url, allow_redirects=True, timeout=15, stream=True)
        result = r.status_code
    except requests.RequestException as e:
        result = type(e).__name__
    cache[url] = result
    return result

def main(folder):
    bad = 0
    for pdf in sorted(pathlib.Path(folder).rglob("*.pdf")):
        seen = set()
        for page, url, kind in links_in(pdf):
            if (page, url) in seen:
                continue
            seen.add((page, url))
            s = status(url)
            if not (isinstance(s, int) and s < 400):
                bad += 1
                print(f"{pdf}  p.{page}  [{kind}]  {s}  {url}")
    print(f"{bad} broken link(s)")
    return 1 if bad else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "."))
Enter fullscreen mode Exit fullscreen mode

Run it on a folder:

python3 pdf_links.py ./downloads
Enter fullscreen mode Exit fullscreen mode
downloads/whitepaper-2022.pdf  p.3  [link]  404  https://www.example.com/resources/benchmark-2022
downloads/whitepaper-2022.pdf  p.11  [text]  ConnectionError  https://beta.example-partner.io/signup
downloads/user-manual-v4.pdf  p.27  [link]  404  https://docs.example.com/v4/setup
3 broken link(s)
Enter fullscreen mode Exit fullscreen mode

Things the script handles (and why)

A cache of URLs. The same link often appears on every page footer. Checking it once instead of 40 times is faster and polite to the server.

HEAD, then GET as a fallback. HEAD is fast, but some servers return 403 or 405 for it, or misbehave. Retrying with GET (streamed, so it doesn't download whole files) avoids false positives.

Redirects are followed. A link that redirects to a working page isn't broken. If you want to find redirecting links too (they often point to the homepage after a site migration), record r.history and report anything with more than one hop.

Trailing punctuation. Text extraction often captures the full stop after a URL at the end of a sentence. The rstrip removes it.

Limitations to know about

  • Text extraction isn't perfect. URLs that wrap onto a second line in the PDF come out as two fragments. If a text URL looks truncated in the report, check the page by eye.
  • Some links aren't URI actions. Links to other pages in the same document (/GoTo) and links to other files (/GoToR) are skipped here, which is usually what you want.
  • Scanned PDFs have no text. If a document is just images, there are no links to find without OCR.
  • Rate limits and bot protection. Sites that block automated requests will show up as 403s. Treat those as "check by hand", not "broken".

A PDF can't be fixed once it's on someone's laptop, but the URLs inside it can be kept alive. linkbruno.com keeps checking the pages those documents point to, on a schedule, so a docs move that breaks the links in last year's manual gets noticed before readers do.

What to do with the results

You usually can't fix a PDF that's already out in the world. You can fix the destination:

  • Add a redirect from the dead URL to its current equivalent. This fixes every copy of the PDF at once, including the ones in people's inboxes. It's the most valuable thing you can do with this report.
  • Re-export and re-upload the PDF if it's still being distributed from your site, so new downloads get working links.
  • Use URLs you control in future documents. A short, stable path on your own domain (example.com/go/benchmark) that redirects to wherever the content lives today is far more durable than a deep link into a docs site that will be restructured.

That last point is the one that pays off long term. Then keep the destination URLs on your regular link checks, with linkbruno.com or your own scheduled script.

Related reading

Do you check the links inside your PDFs at all? I'm curious how many teams have a list of the URLs printed in documents they've sent out.

Top comments (0)