DEV Community

Datawright Labs
Datawright Labs

Posted on

Bulk redirect checker in Python: 301 vs 302, redirect chains and site-migration QA (2026)

#seo #python #webdev #tutorial

Redirects are invisible until they go wrong. A migration ships, someone maps 2,000 old URLs to new ones, and three months later you find out that half of them go through three hops, a few land on the homepage, and one section quietly answers with a 302 that search engines treat as temporary.

curl -sIL https://example.com shows you one chain. This post builds a script that takes a list of URLs (or old -> expected new pairs) and returns, for each one, every hop, the final URL and status, and a plain list of what's wrong.

What counts as a problem

Check Why it matters
Chain longer than 1 hop Every hop costs latency, and crawlers give up on long chains. Point the old URL straight at the final one.
302, 303 or 307 Temporary redirects. For a permanent move you want 301 or 308.
HTTPS to HTTP A downgrade mid-chain undoes the point of HTTPS.
Loop or more than 10 hops Browsers stop and show an error.
Final status 4xx/5xx The redirect works, but it lands on a broken page.
Final page on plain HTTP Usually a missing HTTPS redirect at the end of the chain.
Final URL is not the expected one Migration QA: the old URL should land exactly where your redirect map says.

The script 

# Usage: python3 redirect_check.py urls.txt
# Bulk redirect checker: follows every redirect by hand, records each hop, and flags SEO problems
# (long chains, temporary redirects, HTTPS->HTTP, loops, broken final pages, wrong destination).
# Lines may be "url" or "old_url -> expected_new_url" for migration QA. Standard library only.
import csv, sys, time, http.client, ssl
from urllib.parse import urlsplit, urljoin
from concurrent.futures import ThreadPoolExecutor

UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36'
CTX = ssl.create_default_context()
MAX_HOPS = 10

def norm(u):
    return u.rstrip('/').lower()

def hop(url):
    u = urlsplit(url)
    Conn = http.client.HTTPSConnection if u.scheme == 'https' else http.client.HTTPConnection
    kw = {'context': CTX} if u.scheme == 'https' else {}
    c = Conn(u.hostname, u.port, timeout=15, **kw)
    c.request('GET', (u.path or '/') + ('?' + u.query if u.query else ''), headers={'User-Agent': UA})
    r = c.getresponse(); r.read(100_000); c.close()
    return r.status, r.getheader('Location')

def check(line):
    src, _, expected = (p.strip() for p in line.partition('->'))
    url, chain, seen, problems = src, [], set(), []
    try:
        for _ in range(MAX_HOPS + 1):
            if url in seen:
                problems.append('redirect loop'); break
            seen.add(url)
            status, loc = hop(url)
            chain.append((status, url))
            if status in (301, 302, 303, 307, 308) and loc:
                nxt = urljoin(url, loc)
                if url.startswith('https://') and nxt.startswith('http://'):
                    problems.append('HTTPS -> HTTP downgrade')
                if status in (302, 303, 307):
                    problems.append(f'temporary {status}')
                url = nxt
                continue
            break
        else:
            problems.append(f'more than {MAX_HOPS} redirects')
    except Exception as e:
        return {'input': src, 'error': f'{type(e).__name__}: {e}'[:120]}
    hops = len(chain) - 1
    final_status = chain[-1][0]
    if hops > 1: problems.append(f'{hops}-hop chain')
    if final_status >= 400: problems.append(f'final {final_status}')
    if url.startswith('http://'): problems.append('final page on plain HTTP')
    if expected and norm(url) != norm(urljoin(src, expected)):
        problems.append(f'expected {expected}')
    return {'input': src, 'final_url': url, 'final_status': final_status, 'hops': hops,
            'chain': ' > '.join(f'{s} {u}' for s, u in chain),
            'problems': '; '.join(dict.fromkeys(problems))}

if __name__ == '__main__':
    lines = [l.strip() for l in open(sys.argv[1]) if l.strip() and not l.startswith('#')]
    t = time.time()
    with ThreadPoolExecutor(16) as ex:
        rows = list(ex.map(check, lines))
    cols = ['input', 'final_url', 'final_status', 'hops', 'problems', 'chain', 'error']
    with open('redirect_report.csv', 'w', newline='') as f:
        w = csv.DictWriter(f, cols); w.writeheader(); w.writerows(rows)
    for r in rows:
        flag = 'ERR' if 'error' in r else ('!!' if r['problems'] else 'ok')
        print(f"{flag:3} {r.get('hops', ''):>2}  {r['input'][:38]:38} {r.get('problems') or r.get('error', '') or r['final_url']}"[:160])
    print(f'{len(lines)} URLs in {time.time() - t:.1f} s -> redirect_report.csv')
Enter fullscreen mode Exit fullscreen mode

Two choices matter. Redirects are followed by hand with http.client instead of letting urllib or requests do it silently, because the hops are the whole point: you want to see 301 > 301 > 200, not only where you ended up. And each Location header is resolved with urljoin, because relative redirects (Location: /get) are common and legal.

Results

The input file mixes plain URLs and migration pairs:

https://github.com
https://bbc.com -> https://www.bbc.com/
https://google.com
https://python.org
https://twitter.com
https://httpbin.org/redirect/3
https://httpbin.org/redirect/12
https://apple.com/ipad -> https://www.apple.com/ipad/
https://mozilla.org/firefox -> https://www.mozilla.org/en-US/firefox/new/
https://wikipedia.org/wiki/Python -> https://en.wikipedia.org/wiki/Python
https://expired.badssl.com
Enter fullscreen mode Exit fullscreen mode 
$ python3 redirect_check.py urls.txt
ok   0  https://github.com                     https://github.com
ok   1  https://bbc.com                        https://www.bbc.com/
ok   1  https://google.com                     https://www.google.com/
ok   1  https://python.org                     https://www.python.org/
ok   1  https://twitter.com                    https://x.com/
!!   3  https://httpbin.org/redirect/3         temporary 302; 3-hop chain
!!  10  https://httpbin.org/redirect/12        temporary 302; more than 10 redirects; 10-hop chain
!!   2  https://apple.com/ipad                 2-hop chain
!!   2  https://mozilla.org/firefox            2-hop chain; expected https://www.mozilla.org/en-US/firefox/new/
!!   2  https://wikipedia.org/wiki/Python      2-hop chain
ERR     https://expired.badssl.com             SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_s
11 URLs in 1.4 s -> redirect_report.csv
Enter fullscreen mode Exit fullscreen mode

What this sample shows:

  • Two-hop chains are everywhere. apple.com/ipad goes 301 to www.apple.com/ipad, then 301 again to add the trailing slash. Wikipedia goes apex, then www, then en. Each is harmless for one URL, but on a site you control, pointing every old URL straight at its final address is a free win.
  • The expected-destination check catches real drift. I expected mozilla.org/firefox to land on www.mozilla.org/en-US/firefox/new/, but Firefox now lives on its own domain (www.firefox.com). In a migration, that's exactly the kind of row you want flagged instead of assuming the map is right.
  • Temporary redirects show up as such. httpbin's test endpoints use 302, so they're flagged, along with the 12-redirect chain that hits the 10-hop limit.
  • Clean ones are clean. bbc.com and google.com take one 301 to www, and the BBC row matches its expected destination.

Gotchas I hit

  • Port 80 can be blocked. My test box times out on plain http:// URLs, so the sample uses https:// inputs. In real use, test the http:// versions too: the HTTP-to-HTTPS redirect is the one most often misconfigured.
  • Trailing slashes and case. https://www.bbc.com and https://www.bbc.com/ are the same page. The script normalizes both sides before comparing; decide whether your site should treat case as significant.
  • Bots get different answers. Some sites redirect unknown user agents to a challenge page or answer with 403. Send a browser User-Agent, and also try a Googlebot one if SEO is the reason you're checking.
  • Not every redirect is a 3xx. <meta http-equiv="refresh"> and JavaScript redirects don't show up in headers. Parse the final HTML if you need them.
  • Expired certificates throw before the first hop. The script reports them as errors, which is usually what you want.
  • Don't point it at internal addresses. If you expose this as a service that takes user input, block private and reserved IPs, or it becomes an SSRF tool.

Running it at scale

For a few hundred URLs the script is enough. If you'd rather not handle meta refresh, canonical and noindex checks on the destination, "deep URL dumped on the homepage" detection, Googlebot and mobile user agents and SSRF guards yourself, I also published it as a cloud tool: Redirect Checker on Apify (disclosure: I built it). It takes plain URLs or old -> new pairs (or a JSON redirect map), can return only the problem rows, and costs $0.50 per 1,000 URLs checked; unreachable URLs and invalid inputs are free.

Either way, run it before and after every migration. Redirect mistakes are cheap to fix the day you ship and expensive to find three months later.

Top comments (0)