How to find broken outbound links and replace them (using the Wayback Machine)

Your own pages stay where you put them. The pages you link to don't. Other sites redesign, move their docs, shut down or let their domain expire, and every link from your content to them slowly goes bad.

Outbound link rot hurts in two ways. Readers hit dead ends in the middle of your article, and some "working" links quietly point somewhere you'd never recommend, like a parked domain full of ads. Here's how to find those links and what to replace them with.

The three ways outbound links fail

1. Hard failures. 404, 410, DNS errors, connection refused. Easy to detect.
2. Redirects to somewhere else. The old URL now redirects to the homepage or a generic landing page. It returns 200, but the content you cited is gone. This is a soft 404.
3. Expired domains. The site's owner let the domain lapse and someone else bought it. It now shows a "this domain may be for sale" page, ads, or worse. Also 200 OK.

Only the first kind shows up in a simple status check. The other two need you to look at where the link ends up.

Step 1: List every outbound link

Start with your sitemap so you check real pages, then collect every link that leaves your domain. This script prints each external URL with the page it's on:

import re
from html.parser import HTMLParser
from urllib.parse import urljoin, urlparse

import requests

SITE = "https://example.com"
HOST = urlparse(SITE).netloc

class Links(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hrefs = []
    def handle_starttag(self, tag, attrs):
        if tag == "a":
            href = dict(attrs).get("href")
            if href:
                self.hrefs.append(href)

sitemap = requests.get(f"{SITE}/sitemap.xml", timeout=15).text
pages = re.findall(r"<loc>(.*?)</loc>", sitemap)

outbound = {}
for page in pages:
    p = Links()
    p.feed(requests.get(page, timeout=15).text)
    for href in p.hrefs:
        url = urljoin(page, href).split("#")[0]
        host = urlparse(url).netloc
        if url.startswith("http") and host and host != HOST:
            outbound.setdefault(url, set()).add(page)

for url, found_on in sorted(outbound.items()):
    print(url, "|", ", ".join(sorted(found_on)))

If your sitemap is an index of other sitemaps, fetch those first. For a site with thousands of pages, cap pages while you test.

Step 2: Check each link and record where it ends up

Request each URL, follow redirects, and keep both the status and the final address:

import time

def check(url):
    try:
        r = requests.get(url, timeout=20, allow_redirects=True, stream=True,
                         headers={"User-Agent": "Mozilla/5.0 (outbound link audit)"})
        r.close()
        return r.status_code, r.url
    except requests.RequestException as e:
        return 0, type(e).__name__

for url in outbound:
    status, final = check(url)
    moved_host = urlparse(final).netloc not in ("", urlparse(url).netloc)
    went_home = urlparse(final).path in ("", "/") and urlparse(url).path not in ("", "/")
    if status == 0 or status >= 400 or moved_host or went_home:
        print(status, url, "->", final)
    time.sleep(1)

The time.sleep(1) keeps you from hammering any one site. The output flags:

• errors and unreachable hosts
• links that now land on a different domain, a common sign of an expired or sold domain
• links that now land on the homepage, a common sign the specific page was removed

Ignore 403, 429 and LinkedIn's 999 for now. Those usually mean the site blocks scripts, not that the page is gone. Open them in a browser.

Step 3: Look at the suspicious ones

For links that moved to a different domain or the homepage, open them and decide:

• Is this the same organisation under a new name? Then the redirect is fine, but update the link to the final URL.
• Is it a parked page, a domain marketplace, or unrelated content? Remove or replace the link now. Keeping it sends readers somewhere you don't control.

Step 4: Find a replacement

When the page you cited is gone, you have three options, in order of preference.

1. Find where it moved. Search the original site for the page title, or search the web for the exact title in quotes. Docs and blog posts often survive a redesign under a new URL.

2. Link to an archived copy. The Wayback Machine at the Internet Archive keeps snapshots of a huge number of pages. Its availability API returns the closest snapshot for a URL:

curl -s "https://archive.org/wayback/available?url=example.org/old-guide"

{"archived_snapshots": {"closest": {"available": true, "status": "200",
  "url": "http://web.archive.org/web/20230115000000/https://example.org/old-guide",
  "timestamp": "20230115000000"}}}

In Python:

def wayback(url):
    r = requests.get("https://archive.org/wayback/available", params={"url": url}, timeout=20)
    snap = r.json().get("archived_snapshots", {}).get("closest")
    return snap["url"] if snap and snap.get("available") else None

Open the snapshot before you use it. Check that it shows the actual content you cited, not an error page that was archived after the site broke. Adding a timestamp parameter (&timestamp=20220101) asks for the snapshot closest to that date, which helps you get a copy from when you first linked.

3. Replace the source. If neither works, cite a different page that makes the same point, or rewrite the sentence so it doesn't need a link.

Step 5: Update links in bulk

Most outbound link rot clusters around a few domains. If one docs site moved everything from /docs/ to /documentation/, a single search and replace in your CMS or repository fixes dozens of links at once. Sort your results by domain before you start editing.

In a static site repo:

grep -rl "olddocs.example.org" content/ | xargs sed -i 's#https://olddocs.example.org/#https://docs.example.org/#g'

Review the diff before you commit.

Keep it from piling up again

Outbound links break on other people's schedules, so a one-off audit goes stale quickly. Rerun the scripts above every few months, or use a monitor that does it for you. linkbruno.com, for example, checks external links on a schedule, flags parked and expired domains, and shows the Wayback Machine's last copy of dead pages so the replacement is one click away.

Either way, a few minutes of upkeep keeps your older articles useful, and your readers never land on a domain-for-sale page from a link you recommended.