Checking Whether a Backlink Actually Fits: A Small Python Script

#seo #python #webdev #marketing

If you build sites for clients, sooner or later someone hands you a list of backlinks they've paid for and asks, "are these any good?" Metrics tools will give you a number. What they won't tell you quickly is whether each link fits the page it points to.

At Unfair Links we spend a lot of time on that question, because our whole approach is matching links to the country, language and niche of a specific page. This post shares a stripped-down version of the kind of first-pass check you can run yourself. It's not magic, and it won't replace a human looking at the page, but it catches the obvious mismatches in seconds.

What we're checking

For each linking page, we want to know four things:

Is the link actually there? Pages get edited, links get removed.
Is it crawlable? A noindex on the page or a nofollow on the link changes things.
Does the language match? A French target page linked from an English page is a mismatch worth flagging.
Is the surrounding text related? A rough keyword overlap tells you whether the link sits in relevant context.

The script

import re
import requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse

HEADERS = {"User-Agent": "link-fit-check/0.1"}

def fetch(url):
    r = requests.get(url, headers=HEADERS, timeout=15)
    r.raise_for_status()
    return BeautifulSoup(r.text, "html.parser")

def page_lang(soup):
    html = soup.find("html")
    return (html.get("lang") or "").split("-")[0].lower() if html else ""

def is_noindex(soup):
    tag = soup.find("meta", attrs={"name": re.compile("robots", re.I)})
    return bool(tag and "noindex" in tag.get("content", "").lower())

def words(text):
    return {w for w in re.findall(r"[a-zà-ÿ]{4,}", text.lower())}

def check(source_url, target_url):
    src = fetch(source_url)
    tgt = fetch(target_url)
    target_host = urlparse(target_url).netloc.replace("www.", "")

    link = None
    for a in src.find_all("a", href=True):
        if target_host in a["href"]:
            link = a
            break

    result = {
        "source": source_url,
        "link_found": link is not None,
        "nofollow": bool(link and "nofollow" in (link.get("rel") or [])),
        "source_noindex": is_noindex(src),
        "lang_source": page_lang(src),
        "lang_target": page_lang(tgt),
    }
    result["lang_match"] = result["lang_source"] == result["lang_target"]

    if link:
        context = link.find_parent(["p", "li", "section", "article"]) or src
        overlap = words(context.get_text()) & words(tgt.get_text())
        result["context_overlap"] = len(overlap)
    return result

if __name__ == "__main__":
    import sys
    target = sys.argv[1]
    for line in open(sys.argv[2]):
        url = line.strip()
        if url:
            try:
                print(check(url, target))
            except Exception as e:
                print({"source": url, "error": str(e)})

Run it with your client's page and a text file of linking URLs:

pip install requests beautifulsoup4
python fit_check.py https://example.fr/page-cible links.txt

Reading the output

link_found: False is the first thing to chase. Either the link was removed or it points somewhere slightly different from what was reported.
source_noindex: True means the linking page has asked not to be indexed. That link is doing very little.
lang_match: False is a mismatch worth a human look. It isn't automatically bad, but a run of English pages linking to a German local business is a pattern you'd want to explain.
context_overlap is crude. Treat a very low number as "go and read this page", not as a verdict.

What the script can't tell you

Plenty of the things that make a link valuable aren't in the HTML.

Is there a real business behind the site? Search the business name, look for a Google Business Profile, read some reviews. A site attached to a genuine local business with years of history behaves very differently from one built only to host links.

Does it share patterns with other link sites? Same theme, same hosting, same outbound link profile across a seller's inventory is a warning sign. You can only spot this by comparing several sites side by side.

Is the page indexed? The script checks for noindex, but the only real test is whether the page appears in search. A site: query is a quick manual check.

Why this matters

The links that pass all of these checks are hard to come by, which is exactly why they're worth having. It's also the reasoning behind our name: "unfair" means a competitive advantage your rivals can't just buy off the same list, not rule-breaking. If you want the longer version of that argument, there's a post on GBP-matched local backlinks that goes into it.

Feel free to adapt the script. If you extend it, say with hreflang checks or a proper relevance model, I'd love to see what you build.

DEV Community