Uncover Hidden Internal Linking Issues with a Free SEO Audit Workflow 🔍

Tired of burning cash on expensive SEO suites just to run a simple competitor analysis? Me too. That’s why I built a lean workflow using the Chrome DevTools Network tab and a lightweight, free Python script to map out any page’s internal link structure—no paid subscriptions required.

Most people don’t realize the DOM alone can be misleading. JavaScript frameworks often inject placeholder links that differ from the actual href attributes rendered after hydration. The real source of truth? The network waterfall. When you scrape the fetch/XHR requests from a live page, you get the raw data that drives the site’s navigation.

Here’s a quick snippet to grab internal links from a page’s response:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse, urljoin

def get_internal_links(url):
    try:
        response = requests.get(url, timeout=10)
        soup = BeautifulSoup(response.text, 'html.parser')
        domain = urlparse(url).netloc
        internal_links = set()
        for a_tag in soup.find_all('a', href=True):
            href = a_tag['href']
            full_url = urljoin(url, href)
            parsed = urlparse(full_url)
            if parsed.netloc == domain and parsed.scheme in ('http', 'https'):
                internal_links.add(full_url)
        return list(internal_links)
    except Exception as e:
        return f"Error: {e}"

# Usage
url = "https://example.com"
links = get_internal_links(url)
print(f"Found {len(links)} internal links on {url}")

This script filters out external links, ads, and social media icons. But it still misses dynamic routes loaded via API calls. To catch those, you’d hook into the Network tab using a headless browser like Playwright—but that’s a tutorial for another day.

Why does this matter? SEO audits often get stuck on homepage metrics. But internal link depth signals content priority to search engines. If your “Services” page is three clicks deep and only linked from a footer, it’s practically invisible.

I’ve been using this approach to spot orphaned pages and broken breadcrumbs on client sites. If you want to scale this into a full crawl—including backlink gaps and keyword mapping—you’ll eventually need a tool that automates the heavy lifting. That’s where SerpSpur comes in as a solid Semrush alternative. It gives you the same site audit, backlink analyzer, and keyword tracking without the enterprise price tag.

But for today, just run the script on your own site. You’ll be surprised what you find. Drop your results in the comments—I’d love to see what you uncover.

Comments

[Mustafa ERBAY]

Useful lightweight starting point, but I think a few technical claims need clarification.

The Network waterfall is not really the “source of truth” for internal links. Fetch/XHR requests show resources and data exchanged with the server; they do not necessarily represent the links exposed to users or search engines. For SEO, the relevant source is the rendered DOM and crawlable a href elements. Google explicitly states that JavaScript-injected links are crawlable when they are rendered as standard anchor elements.

The Python example also parses only the initial HTML response. It does not execute JavaScript or inspect the hydrated DOM, so Playwright would not merely be needed for “dynamic routes loaded via API calls”; it would be needed whenever links are created or changed client-side.

A couple of smaller points:

• The script does not specifically filter ads or social icons; it only filters different hostnames.
• Crawling one page cannot identify orphan pages. That requires a site-wide crawl compared against sources such as XML sitemaps, Search Console, analytics, server logs, or the CMS URL inventory.
• “Three clicks deep means practically invisible” is too absolute. Link depth can affect discovery and perceived importance, but there is no universal three-click invisibility threshold.

Official references:

Google — Crawlable links:
developers.google.com/search/docs/...

Google — JavaScript SEO basics:
developers.google.com/search/docs/...

Google — Sitemaps:
developers.google.com/search/docs/...

Playwright — Locators and the current DOM:
playwright.dev/docs/locators

The workflow is still useful as a basic static-HTML link extractor, but I would describe it that way rather than as a complete internal-link or orphan-page audit.

[Michel Jee]

Interesting how even an empty post can spark curiosity — what was the intention behind leaving it blank?

[Victoria]

An empty post can be a powerful statement in itself. Sometimes silence speaks louder than words.