DEV Community: Albin Johansson

Find link-building prospects with Python: a competitor backlink gap for $1 per 1,000 links

Albin Johansson — Sat, 26 Sep 2026 12:17:31 +0000

The best link-building prospects are sites that already link to your competitors but not to you. They cover your niche, they link out, and you're missing from the list. Ahrefs calls this Link Intersect and Semrush calls it Backlink Gap, and both sit behind subscriptions starting at $129/month (Ahrefs Lite) and $139/month (Semrush SEO plan).

Here's how to get a competitor backlink gap, plus the full backlink list of any site, from Python, paid per result.

1. Setup

pip install "apify-client>=3"
export APIFY_TOKEN=...   # free account at console.apify.com

I'm using the Backlink Checker on Apify (disclosure: I built it). It reads a commercial backlink index and costs $1 per 1,000 backlinks. Filters run at the source, so you only pay for rows you keep.

2. The link gap: who links to them, but not to you?

Say you're Allbirds, and your competitors are Rothy's and Vivobarefoot:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("jesting_grass/backlink-checker").call(run_input={
    "mode": "linkGap",
    "yourDomain": "allbirds.com",
    "competitors": ["rothys.com", "vivobarefoot.com"],
    "minCompetitorsLinked": 2,     # links to BOTH competitors
    "minDomainRank": 50,
    "maxResultsPerTarget": 10,
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(f'{row["referringDomain"]:<30} rank {row["domainRank"]:<4} links to {", ".join(row["linksToCompetitors"])}')

Real output (September 2026):

grandviewresearch.com          rank 94   links to rothys.com, vivobarefoot.com
essence.com                    rank 92   links to rothys.com, vivobarefoot.com
williamb409.sg-host.com        rank 92   links to rothys.com, vivobarefoot.com
app.welcometothejungle.com     rank 90   links to rothys.com, vivobarefoot.com
player.captivate.fm            rank 90   links to rothys.com, vivobarefoot.com
podchaser.com                  rank 90   links to rothys.com, vivobarefoot.com
community.whattoexpect.com     rank 89   links to rothys.com, vivobarefoot.com
yourtango.com                  rank 89   links to rothys.com, vivobarefoot.com
www-ft-com.ezproxy.brunel.ac.uk rank 89   links to rothys.com, vivobarefoot.com
askmen.com                     rank 89   links to rothys.com, vivobarefoot.com

It compares the top 1,000 referring domains of each site. Not every row is a pitch: a university proxy (ezproxy.brunel.ac.uk) or a staging host (sg-host.com) is noise. But essence.com, askmen.com and yourtango.com are lifestyle publishers that wrote about both competitors, and that's an outreach list.

The run above scans three sites ($0.25 each) and returns 10 prospects ($0.002 each), about $0.77 in total.

3. Every backlink of a site → CSV

For audits or anchor-text analysis, switch to backlinks mode:

import csv

run = client.actor("jesting_grass/backlink-checker").call(run_input={
    "mode": "backlinks",
    "targets": ["allbirds.com"],
    "maxResultsPerTarget": 200,
    "onePerDomain": True,      # best link from each linking website
    "dofollowOnly": True,
})
cols = ["domainRank", "pageRank", "sourceUrl", "anchor", "dofollow", "targetUrl", "firstSeen", "lastSeen"]
with open("backlinks.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=cols, extrasaction="ignore")
    w.writeheader()
    for row in client.dataset(run.default_dataset_id).iterate_items():
        if "error" not in row:
            w.writerow(row)

First rows of the real CSV:

domainRank,pageRank,sourceUrl,anchor,dofollow
100,25,https://www.shopify.com/za/blog/12206313-the-ultimate-diy-guide-to-beautiful-product-photography,Allbirds,True
100,0,https://www.godaddy.com/garage/best-practices-for-using-images-on-ecommerce-product-pages,allbirds,True
100,0,https://blog.adobe.com/jp/publish/2021/05/31/cc-web-effective-use-images-graphics-ux-design,AllBirds,True

Other filters: minDomainRank, anchorContains (e.g. only links mentioning your brand), and referringDomains mode for one row per linking site.

4. No Python? One HTTP call

curl -X POST "https://api.apify.com/v2/acts/jesting_grass~backlink-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"backlinks","targets":["allbirds.com"],"maxResultsPerTarget":20,"onePerDomain":true}'

It also runs from n8n, Make and Zapier, and AI agents can call it through the Apify MCP server.

A note on the numbers

Every backlink tool runs its own crawler, so Ahrefs, Semrush, Moz and this index each find a different set of links, and the authority scores differ too. Compare sites within one tool rather than across tools.

All examples (backlinks CSV, link gap, Node.js, cURL): github.com/emiohr/backlink-checker-api. Questions or feature requests? Drop a comment.

This article was written by an AI agent under my direction, and all code was tested against the live API. Not affiliated with Ahrefs, Semrush or Moz.

Keyword research in Python: search volume, difficulty and AI Overviews in bulk (2026)

Albin Johansson — Sat, 26 Sep 2026 11:39:11 +0000

In 2026 the first question about a keyword isn't only "how many searches?" but also "does Google answer it with an AI Overview?". Those keywords often send fewer clicks, and they're exactly the ones to target if you want to be cited in AI answers (AEO).

Google Keyword Planner is free, but it shows volume ranges unless you run active ads, and it has no keyword difficulty or AI Overview data. Ahrefs and Semrush have all of it, starting at $129/month and $139/month.

Here's how to get search volume, keyword difficulty, intent, CPC and the AI Overview flag for a whole list of keywords, paid per keyword, in a few lines of Python.

1. Setup

pip install "apify-client>=3"
export APIFY_TOKEN=...   # free account at console.apify.com

I'm using the Keyword Research Tool on Apify (disclosure: I built it). Search volume, CPC and the 12-month trend come from Google Ads data; difficulty, intent and SERP features come from a commercial SEO database. Keywords without data aren't charged.

2. A keyword list → CSV

import csv, os
from apify_client import ApifyClient

KEYWORDS = ["protein powder", "cold brew coffee", "keyword research tool", "standing desk", "how to make cold brew"]
COLUMNS = ["keyword", "searchVolume", "keywordDifficulty", "intents", "hasAiOverview"]

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("jesting_grass/keyword-research-tool").call(run_input={"keywords": KEYWORDS, "country": "us"})

with open("keywords.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=COLUMNS, extrasaction="ignore")
    w.writeheader()
    for row in client.dataset(run.default_dataset_id).iterate_items():
        if row.get("searchVolume"):
            w.writerow({**row, "intents": ", ".join(row.get("intents") or [])})
            print(f'{row["keyword"]:<24} vol {row["searchVolume"]:>8,}  KD {row["keywordDifficulty"]:<4} AI Overview: {row["hasAiOverview"]}')

Real output (US, September 2026):

protein powder           vol  368,000  KD 72   AI Overview: True
cold brew coffee         vol   40,500  KD 50   AI Overview: True
keyword research tool    vol    4,400  KD 66   AI Overview: False
standing desk            vol  135,000  KD 74   AI Overview: False
how to make cold brew    vol   12,100  KD 48   AI Overview: True

Each row also has cpc (with a low/high bid range), adsCompetition, serpFeatures (featured snippet, People Also Ask, local pack…) and monthlySearches for the last 12 months, so you can spot seasonality.

3. Find easy wins from one seed keyword

Give it a seed and it returns related keyword ideas with the same metrics. Then filter for decent volume and low difficulty:

run = client.actor("jesting_grass/keyword-research-tool").call(run_input={
    "seedKeywords": ["cold brew coffee"],
    "ideasPerSeed": 100,
    "country": "us",
})
ideas = [r for r in client.dataset(run.default_dataset_id).iterate_items() if r.get("source") == "idea"]

easy = sorted((r for r in ideas if (r.get("searchVolume") or 0) >= 500 and (r.get("keywordDifficulty") or 100) <= 40),
              key=lambda r: -r["searchVolume"])
for r in easy[:10]:
    print(f'{r["keyword"]:<40} vol {r["searchVolume"]:>7,}  KD {r["keywordDifficulty"]}')

aio = [r for r in ideas if r.get("hasAiOverview")]
print(f"{len(aio)} of {len(ideas)} ideas trigger a Google AI Overview")

Real output:

coffee and cold brew                     vol  40,500  KD 32
brew cold brew coffee                    vol  40,500  KD 1
cold brew using french press             vol  22,200  KD 35
coarse ground coffee cold brew           vol  22,200  KD 15
grind for cold brewed coffee             vol  12,100  KD 33
cold brew coffee grind                   vol  12,100  KD 18
beans for cold brew                      vol   8,100  KD 26
coffee gourmet                           vol   6,600  KD 24
iced coffee from coffee maker            vol   6,600  KD 18
what is a cold brew coffee               vol   6,600  KD 37
43 of 64 ideas trigger a Google AI Overview

Two things stand out:

Two thirds of this niche gets an AI Overview. For a coffee blog, that changes the plan: those pages need to be written to be cited, not just to rank.
Google groups close variants. "coffee and cold brew" and "brew cold brew coffee" share the 40,500 of "cold brew coffee", so that's one page, not three. The grind keywords ("coarse ground coffee cold brew", "cold brew coffee grind") are a separate, lower-difficulty topic worth its own article.

4. No Python? One HTTP call

curl -X POST "https://api.apify.com/v2/acts/jesting_grass~keyword-research-tool/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keywords":["protein powder","cold brew coffee"],"country":"us"}'

It works for 17 countries (US, UK, Canada, Australia, the big EU markets, the Nordics, Brazil, Mexico, India), runs from n8n, Make and Zapier, and AI agents can call it through the Apify MCP server.

A note on the numbers

Every SEO tool estimates volume and difficulty its own way, so Ahrefs, Semrush and this will not match exactly. Volumes here are Google Ads 12-month averages. Compare keywords within one tool rather than across tools.

All examples (bulk CSV, easy-win finder, Node.js, cURL): github.com/emiohr/keyword-research-api. Questions or feature requests? Drop a comment.

This article was written by an AI agent under my direction, and all code was tested against the live API. Not affiliated with Ahrefs, Semrush or Google.

Check domain authority for 1,000 sites at once with Python (a pay-per-use Ahrefs alternative)

Albin Johansson — Sat, 26 Sep 2026 11:37:34 +0000

You have a list of 500 guest-post prospects, expired domains or client competitors, and you want to know which ones are actually strong. Browser extensions check one site at a time. Ahrefs and Semrush can do it in bulk, but they start at $129/month (Ahrefs Lite) and $139/month (Semrush SEO plan).

If you only need it now and then, or inside a script, a pipeline or an AI agent, here's how to get domain authority and backlink numbers for a whole list, paid per domain, in a few lines of Python.

1. Setup

pip install "apify-client>=3"
export APIFY_TOKEN=...   # free account at console.apify.com

I'm using the Bulk Domain Authority & Backlink Checker on Apify (disclosure: I built it). For every domain it returns a domain rank (0–100), referring domains, backlinks, the dofollow/nofollow split and 30-day changes. Invalid domains and domains without backlink data aren't charged.

2. A list of domains → CSV

import csv, os
from apify_client import ApifyClient

DOMAINS = ["dev.to", "allbirds.com", "gymshark.com", "hashnode.com", "indiehackers.com"]
COLUMNS = ["domain", "domainRank", "referringDomains", "backlinks", "dofollowRatio", "referringDomainsChange"]

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("jesting_grass/bulk-domain-authority-checker").call(run_input={"domains": DOMAINS})

with open("domain_authority.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=COLUMNS, extrasaction="ignore")
    w.writeheader()
    for row in sorted(client.dataset(run.default_dataset_id).iterate_items(), key=lambda r: -(r.get("domainRank") or 0)):
        if "error" not in row:
            w.writerow(row)
            print(f'{row["domain"]:<18} rank {row["domainRank"]:<4} ref. domains {row["referringDomains"]:>8,}  backlinks {row["backlinks"]:>10,}')

Real output (September 2026):

dev.to             rank 70   ref. domains   46,486  backlinks  1,167,777
hashnode.com       rank 60   ref. domains   10,787  backlinks    683,515
indiehackers.com   rank 58   ref. domains    7,520  backlinks     66,592
allbirds.com       rank 57   ref. domains    7,391  backlinks     30,318
gymshark.com       rank 56   ref. domains    5,540  backlinks    178,435

You can paste full URLs: https://, www. and paths are stripped and duplicates removed, so a raw export from a spreadsheet works as-is.

3. Link building: keep only strong prospects

Add minDomainRank and weaker sites are dropped from the results:

run = client.actor("jesting_grass/bulk-domain-authority-checker").call(run_input={
    "domains": ["dev.to", "hashnode.com", "indiehackers.com", "allbirds.com", "gymshark.com"],
    "minDomainRank": 60,
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    if "error" not in row:
        print(row["domain"], "rank", row["domainRank"], "| dofollow ratio", row["dofollowRatio"])

dev.to rank 70 | dofollow ratio 0.752
hashnode.com rank 60 | dofollow ratio 0.819

Two more columns help when screening expired domains or link sellers:

dofollowRatio: a site whose backlinks are almost all nofollow passes little link equity.
maliciousReferringDomains: referring domains flagged as malicious, a quick toxic-link signal.

4. No Python? One HTTP call

curl -X POST "https://api.apify.com/v2/acts/jesting_grass~bulk-domain-authority-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains":["dev.to","allbirds.com"]}'

It also runs from n8n, Make and Zapier, and AI agents can call it as a tool through the Apify MCP server. Schedule it weekly and the referringDomainsChange column shows who is gaining or losing links.

A note on "domain authority"

There's no single universal DA. Moz has Domain Authority, Ahrefs has Domain Rating, and this uses Serpstat's Domain Rank. Each is computed from the vendor's own backlink index, so the same site gets different numbers in different tools. What matters is comparing domains with one score consistently, not matching another tool's number.

All examples (Python CSV export, link-building filter, Node.js, cURL): github.com/emiohr/bulk-domain-authority-checker. Questions or feature requests? Drop a comment.

This article was written by an AI agent under my direction, and all code was tested against the live API. Not affiliated with Ahrefs, Semrush or Moz.

Find any website's tech stack in bulk with Python (a pay-per-use BuiltWith / Wappalyzer alternative)

Albin Johansson — Sat, 26 Sep 2026 10:01:09 +0000

Want to know which of 500 prospects run Shopify, which use HubSpot, or what analytics and payment tools your competitors use? That's technographics — and the usual tools are subscriptions: BuiltWith starts at $295/month (API access only on higher plans) and Wappalyzer Pro is $250/month.

If you only need it now and then, or want it inside a script, a pipeline or an AI agent, here's how to do it pay-per-website in a few lines of Python.

1. Setup

pip install "apify-client>=3"
export APIFY_TOKEN=...   # free account at console.apify.com

I'm using the Tech Stack Detector on Apify (disclosure: I built it). It returns up to 20 technologies per site with categories, and invalid, offline or undetectable sites aren't charged.

2. Tech stack of a list of domains → CSV

import csv, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("jesting_grass/tech-stack-detector").call(run_input={
    "domains": ["allbirds.com", "gymshark.com", "klarna.com", "hubspot.com", "ikea.com"],
})

cols = ["domain", "cms", "ecommerce", "payments", "analytics", "crm", "hosting", "technologyNames"]
with open("tech_stacks.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=cols, extrasaction="ignore")
    w.writeheader()
    for row in client.dataset(run.default_dataset_id).iterate_items():
        if "error" not in row:
            w.writerow(row)
            print(f'{row["domain"]:<14} CMS: {row["cms"] or "-":<22} payments: {row["payments"] or "-"}')

Real output (September 2026):

klarna.com     CMS: Contentful             payments: Klarna Checkout
gymshark.com   CMS: Contentful, Shopify    payments: -
allbirds.com   CMS: Shopify                payments: Stripe
ikea.com       CMS: WordPress              payments: -
hubspot.com    CMS: HubSpot CMS Hub        payments: -

Besides the full technologies list, every row has flat columns — cms, ecommerce, payments, analytics, marketingAutomation, crm, hosting, cdn, jsFrameworks — so the CSV drops straight into Sheets or a CRM.

3. Lead generation: keep only sites using a technology

Selling a Shopify app? Only want prospects on HubSpot? Add one field:

run = client.actor("jesting_grass/tech-stack-detector").call(run_input={
    "domains": ["allbirds.com", "gymshark.com", "klarna.com", "hubspot.com", "ikea.com"],
    "onlyDomainsUsing": ["Shopify"],
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    if "error" not in row:
        print(row["domain"], "uses", row["matchedTechnologies"])

gymshark.com uses ['Shopify']
allbirds.com uses ['Shopify']

Feed it domains from anywhere — a lead list, a CRM export, Google Maps or search results.

4. No Python? One HTTP call

curl -X POST "https://api.apify.com/v2/acts/jesting_grass~tech-stack-detector/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains":["stripe.com","allbirds.com"]}'

It also runs from n8n, Make and Zapier, and AI agents can call it as a tool through the Apify MCP server.

A note on accuracy

Any tech-stack detector — BuiltWith and Wappalyzer included — can only see what a website exposes publicly: scripts, headers, cookies, markup. Back-office tools that leave no trace on the site can't be detected by anyone. Treat results as strong signals, not a complete inventory.

All examples (Python CSV export, Shopify lead filter, Node.js, cURL): github.com/emiohr/builtwith-wappalyzer-alternative. Questions or feature requests — drop a comment.

This article was written with AI assistance and all code was tested against the live API. Not affiliated with BuiltWith or Wappalyzer.

Google Trends API in Python without pytrends (2026 guide)

Albin Johansson — Sat, 26 Sep 2026 09:07:45 +0000

pytrends was archived in April 2025. If you still depend on it, you've probably met this:

pytrends.exceptions.TooManyRequestsError: The request failed: Google returned a response with code 429

pytrends loads the Google Trends website like a browser, and Google rate-limits that traffic. Sleeping, rotating proxies and retrying helps a bit — until it doesn't. And Google's official Trends API is still an application-only alpha (announced July 2025, no release date yet).

Here's how I get the same data (interest over time, related queries, interest by region) reliably in a few lines of Python.

1. Setup

pip install "apify-client>=3" pandas
export APIFY_TOKEN=...   # free account at console.apify.com

I'm using the Google Trends Scraper & API on Apify (disclosure: I built it). It gets the data from commercial Google Trends data providers, so you never hit Google from your own IP or fight 429s yourself, and failed lookups aren't charged.

2. Interest over time → pandas (the `interest_over_time()` replacement)

import os
import pandas as pd
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("jesting_grass/google-trends-api").call(run_input={
    "keywords": ["bitcoin", "ethereum"],   # up to 5 compared in one query
    "timeRange": "past_5_years",
})

series = [
    pd.Series({p["date"]: p["value"] for p in row["timeline"]}, name=row["keyword"])
    for row in client.dataset(run.default_dataset_id).iterate_items()
]
df = pd.concat(series, axis=1)
df.index = pd.to_datetime(df.index)
print(df.tail())

            bitcoin  ethereum
2026-08-23       26         3
2026-08-30       23         2
2026-09-06       19         2
2026-09-13       19         2
2026-09-20       22         3

Each row also comes with averageInterest, peakInterest, peakDate and latestInterest, so you often don't need pandas at all.

3. Related queries — the keyword-research goldmine

"Rising" and Breakout related queries are where new keyword ideas come from. Flip one switch:

run = client.actor("jesting_grass/google-trends-api").call(run_input={
    "keywords": ["protein powder"],
    "location": "United States",
    "searchType": "youtube",          # web | youtube | news | images | froogle
    "includeRelatedQueries": True,
    "includeInterestByRegion": True,
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    for q in row["relatedQueries"]["rising"][:5]:
        print(f'{q["query"]:<35} {q["value"]}')

lead in protein powder              +550%
protein powder scam                 +350%
flavorless protein powder           +200%
costco protein powder               +170%
orgain protein powder               +160%

That's YouTube search trends — handy if you make videos. Same for News, Images and Google Shopping.

4. pytrends → API cheat sheet

pytrends	API input / output
`build_payload(kw, timeframe='today 12-m', geo='US')`	`keywords`, `timeRange: "past_12_months"`, `location: "United States"`
`interest_over_time()`	`row["timeline"]`
`related_queries()`	`includeRelatedQueries: true`
`related_topics()`	`includeRelatedTopics: true`
`interest_by_region()`	`includeInterestByRegion: true`
`gprop='youtube'`	`searchType: "youtube"`

5. Not using Python?

One HTTP call, JSON back:

curl -X POST "https://api.apify.com/v2/acts/jesting_grass~google-trends-api/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keywords":["chatgpt","claude"],"location":"United States","timeRange":"past_90_days"}'

It also works from n8n / Make / Zapier, and AI agents can call it as a tool through the Apify MCP server.

A note on the 0–100 values

Google Trends numbers are relative: 100 is the peak within the keywords, place and time you compared. Two separate queries aren't on the same scale — if you need that, include a shared "anchor" keyword in each group.

All examples (Python, pandas, Node.js, cURL) are in this repo: github.com/emiohr/google-trends-api. Questions or feature requests — drop a comment.

This article was written with AI assistance and all code was tested against the live API.

DEV Community: Albin Johansson

Find link-building prospects with Python: a competitor backlink gap for $1 per 1,000 links

1. Setup

2. The link gap: who links to them, but not to you?

3. Every backlink of a site → CSV

4. No Python? One HTTP call

A note on the numbers

Keyword research in Python: search volume, difficulty and AI Overviews in bulk (2026)

1. Setup

2. A keyword list → CSV

3. Find easy wins from one seed keyword

4. No Python? One HTTP call

A note on the numbers

Check domain authority for 1,000 sites at once with Python (a pay-per-use Ahrefs alternative)

1. Setup

2. A list of domains → CSV

3. Link building: keep only strong prospects

4. No Python? One HTTP call

A note on "domain authority"

Find any website's tech stack in bulk with Python (a pay-per-use BuiltWith / Wappalyzer alternative)

1. Setup

2. Tech stack of a list of domains → CSV

3. Lead generation: keep only sites using a technology

4. No Python? One HTTP call

A note on accuracy

Google Trends API in Python without pytrends (2026 guide)

1. Setup

2. Interest over time → pandas (the interest_over_time() replacement)

3. Related queries — the keyword-research goldmine

4. pytrends → API cheat sheet

5. Not using Python?

A note on the 0–100 values

2. Interest over time → pandas (the `interest_over_time()` replacement)