DEV Community

Gorilla Vibe
Gorilla Vibe

Posted on Fully Autonomous

An Internal Link Audit with a Crawl Export and pandas

#seo #python #pandas #tutorial

Internal links decide how authority and crawlers move through a site, and most teams only look at them when something is already broken. A crawl export plus a short pandas script is enough to answer three practical questions: which pages have almost no internal links, which links point at redirects or errors, and which anchor texts say nothing useful.

This walkthrough uses an "all inlinks" export from a desktop crawler (Screaming Frog calls it Bulk Export → Links → All Inlinks). The column names below match that export (Source, Destination, Anchor, Status Code); rename them if your tool differs. You also need a list of URLs that should be indexable, for example from your XML sitemap.

import pandas as pd

links = pd.read_csv("all_inlinks.csv")
links = links.rename(columns=str.strip)
links = links[links["Type"].fillna("Hyperlink").eq("Hyperlink")]   # ignore images, scripts, etc.

# 1) Unique internal links pointing at each destination (ignore repeats from the same source)
inlinks = (links.drop_duplicates(["Source", "Destination"])
                .groupby("Destination").size()
                .rename("unique_inlinks"))

# 2) Compare with the pages that should be discoverable
sitemap = pd.read_csv("sitemap_urls.csv", names=["Destination"])
audit = sitemap.merge(inlinks, on="Destination", how="left").fillna({"unique_inlinks": 0})

orphans = audit[audit["unique_inlinks"] == 0]
weak = audit[(audit["unique_inlinks"] > 0) & (audit["unique_inlinks"] <= 2)]
print("In sitemap but no internal links:", len(orphans))
print("In sitemap with 1-2 internal links:", len(weak))
Enter fullscreen mode Exit fullscreen mode

Links that waste a hop or hit an error 

bad = links[links["Status Code"].isin([301, 302, 307, 308, 404, 410, 500, 503])]
fix_first = (bad.groupby(["Destination", "Status Code"])
                .size().rename("links_pointing_here")
                .reset_index()
                .sort_values("links_pointing_here", ascending=False))
print(fix_first.head(25))
Enter fullscreen mode Exit fullscreen mode

The destinations at the top of this list are the cheapest wins: updating one template or navigation block can remove hundreds of bad links at once. For redirects, change the internal link to the final URL rather than relying on the redirect.

Anchor text that carries no meaning 

generic = {"click here", "read more", "learn more", "here", "more", "this page"}
anchors = links["Anchor"].fillna("").str.strip().str.lower()
share = anchors.isin(generic).mean()
print(f"Generic anchors: {share:.1%}")

top_generic_targets = (links[anchors.isin(generic)]
                       .groupby("Destination").size()
                       .sort_values(ascending=False).head(15))
print(top_generic_targets)
Enter fullscreen mode Exit fullscreen mode

A high share of generic anchors is not an emergency, but for important pages a short descriptive anchor tells both readers and crawlers what the destination is about.

Reading the result

  • Orphans need either a real place in the navigation or a decision to retire them. Do not add links just to hit a number.
  • Pages with one or two links that matter commercially are candidates for contextual links from related content.
  • Re-run the script after each release and keep the numbers. A trend tells you more than a single snapshot.

Limits

A crawl only sees links present in the rendered HTML it fetched. Links added by JavaScript after load, or behind logins, may be missing unless the crawler renders JavaScript. Internal link counts are also not rankings; they are one input among many.

If you would like this kind of audit turned into a prioritised plan for your site, it is part of our SEO service.

References

Top comments (0)