DEV Community

Cover image for Find link-building prospects with Python: a competitor backlink gap for $1 per 1,000 links
Albin Johansson
Albin Johansson

Posted on Fully Autonomous

Find link-building prospects with Python: a competitor backlink gap for $1 per 1,000 links

#python #api #seo #webdev

The best link-building prospects are sites that already link to your competitors but not to you. They cover your niche, they link out, and you're missing from the list. Ahrefs calls this Link Intersect and Semrush calls it Backlink Gap, and both sit behind subscriptions starting at $129/month (Ahrefs Lite) and $139/month (Semrush SEO plan).

Here's how to get a competitor backlink gap, plus the full backlink list of any site, from Python, paid per result.

1. Setup 

pip install "apify-client>=3"
export APIFY_TOKEN=...   # free account at console.apify.com
Enter fullscreen mode Exit fullscreen mode

I'm using the Backlink Checker on Apify (disclosure: I built it). It reads a commercial backlink index and costs $1 per 1,000 backlinks. Filters run at the source, so you only pay for rows you keep.

2. The link gap: who links to them, but not to you?

Say you're Allbirds, and your competitors are Rothy's and Vivobarefoot:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("jesting_grass/backlink-checker").call(run_input={
    "mode": "linkGap",
    "yourDomain": "allbirds.com",
    "competitors": ["rothys.com", "vivobarefoot.com"],
    "minCompetitorsLinked": 2,     # links to BOTH competitors
    "minDomainRank": 50,
    "maxResultsPerTarget": 10,
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(f'{row["referringDomain"]:<30} rank {row["domainRank"]:<4} links to {", ".join(row["linksToCompetitors"])}')
Enter fullscreen mode Exit fullscreen mode

Real output (September 2026):

grandviewresearch.com          rank 94   links to rothys.com, vivobarefoot.com
essence.com                    rank 92   links to rothys.com, vivobarefoot.com
williamb409.sg-host.com        rank 92   links to rothys.com, vivobarefoot.com
app.welcometothejungle.com     rank 90   links to rothys.com, vivobarefoot.com
player.captivate.fm            rank 90   links to rothys.com, vivobarefoot.com
podchaser.com                  rank 90   links to rothys.com, vivobarefoot.com
community.whattoexpect.com     rank 89   links to rothys.com, vivobarefoot.com
yourtango.com                  rank 89   links to rothys.com, vivobarefoot.com
www-ft-com.ezproxy.brunel.ac.uk rank 89   links to rothys.com, vivobarefoot.com
askmen.com                     rank 89   links to rothys.com, vivobarefoot.com
Enter fullscreen mode Exit fullscreen mode

It compares the top 1,000 referring domains of each site. Not every row is a pitch: a university proxy (ezproxy.brunel.ac.uk) or a staging host (sg-host.com) is noise. But essence.com, askmen.com and yourtango.com are lifestyle publishers that wrote about both competitors, and that's an outreach list.

The run above scans three sites ($0.25 each) and returns 10 prospects ($0.002 each), about $0.77 in total.

3. Every backlink of a site → CSV

For audits or anchor-text analysis, switch to backlinks mode:

import csv

run = client.actor("jesting_grass/backlink-checker").call(run_input={
    "mode": "backlinks",
    "targets": ["allbirds.com"],
    "maxResultsPerTarget": 200,
    "onePerDomain": True,      # best link from each linking website
    "dofollowOnly": True,
})
cols = ["domainRank", "pageRank", "sourceUrl", "anchor", "dofollow", "targetUrl", "firstSeen", "lastSeen"]
with open("backlinks.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=cols, extrasaction="ignore")
    w.writeheader()
    for row in client.dataset(run.default_dataset_id).iterate_items():
        if "error" not in row:
            w.writerow(row)
Enter fullscreen mode Exit fullscreen mode

First rows of the real CSV:

domainRank,pageRank,sourceUrl,anchor,dofollow
100,25,https://www.shopify.com/za/blog/12206313-the-ultimate-diy-guide-to-beautiful-product-photography,Allbirds,True
100,0,https://www.godaddy.com/garage/best-practices-for-using-images-on-ecommerce-product-pages,allbirds,True
100,0,https://blog.adobe.com/jp/publish/2021/05/31/cc-web-effective-use-images-graphics-ux-design,AllBirds,True
Enter fullscreen mode Exit fullscreen mode

Other filters: minDomainRank, anchorContains (e.g. only links mentioning your brand), and referringDomains mode for one row per linking site.

4. No Python? One HTTP call 

curl -X POST "https://api.apify.com/v2/acts/jesting_grass~backlink-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"backlinks","targets":["allbirds.com"],"maxResultsPerTarget":20,"onePerDomain":true}'
Enter fullscreen mode Exit fullscreen mode

It also runs from n8n, Make and Zapier, and AI agents can call it through the Apify MCP server.

A note on the numbers

Every backlink tool runs its own crawler, so Ahrefs, Semrush, Moz and this index each find a different set of links, and the authority scores differ too. Compare sites within one tool rather than across tools.

All examples (backlinks CSV, link gap, Node.js, cURL): github.com/emiohr/backlink-checker-api. Questions or feature requests? Drop a comment.

This article was written by an AI agent under my direction, and all code was tested against the live API. Not affiliated with Ahrefs, Semrush or Moz.

Top comments (0)