DEV Community

Wrenfield Data
Wrenfield Data

Posted on

Competitor keyword research without Ahrefs or Semrush: exact search volume first (Python + n8n)

#seo #python #automation #webdev

Disclosure: I built the five Apify Actors used here: Keyword Search Volume, SEO Toolkit, Competitor Keywords, Keyword Difficulty Checker and People Also Ask Scraper. All data below is from real runs on October 18, 2026 (US). I'm not affiliated with Ahrefs, Semrush or the example competitor.

Ahrefs and Semrush are excellent. But if you do competitor research a few times a month, or you want the data inside a script or an n8n workflow, a $100–$500/month seat is a lot. This post runs a complete competitor analysis with five pay-per-use calls and ends with a concrete content plan. The step that does the most work is the boring one: exact Google search volume and CPC for every candidate keyword, so you rank opportunities on real demand instead of estimates.

The example: a new site that wants to compete with upliftdesk.com, a standing desk brand.

The workflow

  1. Domain overview: how big is the competitor?
  2. Competitor keywords: what does it rank for, and which URL earns the traffic?
  3. Exact search volume + CPC: how much demand, and how much is it worth to advertisers?
  4. Keyword difficulty + intent: which of those keywords can a smaller site win?
  5. People Also Ask: what should the article answer?

Setup 

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])

def run(actor, inp, cap=1.0):
    r = client.actor(actor).call(run_input=inp, max_total_charge_usd=cap)
    assert r.status == "SUCCEEDED", r.status
    return list(client.dataset(r.default_dataset_id).iterate_items())
Enter fullscreen mode Exit fullscreen mode

Step 1: domain overview 

ov = run("wrenfield-data/seo-toolkit", {"tool": "domain_overview", "targets": ["upliftdesk.com"],
                              "location": "United States"})[0]
Enter fullscreen mode Exit fullscreen mode

Result (5 seconds):

Metric upliftdesk.com
Organic keywords (US) 15,414
Estimated monthly organic traffic 74,939
Estimated traffic value / month $228,543
Keywords in top 3 / top 10 2,628 / 6,536
Referring domains / backlinks 4,221 / 44,647
Spam score 4

A strong, well-linked competitor. Going after its head terms head-on is a losing plan, so we need to be selective.

Step 2: what it ranks for 

kws = run("wrenfield-data/competitor-keywords", {"domains": ["upliftdesk.com"], "location": "United States",
                                       "maxKeywordsPerDomain": 25, "sortBy": "traffic"})
Enter fullscreen mode Exit fullscreen mode

Top non-branded rows (4 seconds):

Keyword Position Volume Est. traffic Ranking URL
standing desk 5 135,000 3,532 /2-leg-standing-desk/
adjustable desk 1 22,200 2,035 /standing-desks/
stand up desk 1 18,100 1,401 /standing-desks/
electric standing desk 2 27,100 1,379 /standing-desks/
ergonomic office chair 8 165,000 912 /ergonomic-chairs/
monitor arm 2 22,200 864 /desk-accessories/monitor-arms/
height adjustable desk 2 12,100 676 /standing-desks/
kneeling chair 2 14,800 578 /ergonomic-kneeling-chair-by-uplift-desk/

Each row also has keyword_difficulty, search_intent, serp_features, previous_position and position_change, so you can sort by "keywords they're losing" too. Notice how much traffic one category page (/standing-desks/) earns.

Step 3: exact volume and CPC

Competitor tools estimate volume. For the shortlist you are going to plan content around, pull the numbers Google Ads itself reports:

vol = run("wrenfield-data/keyword-search-volume", {"keywords": [
    "standing desk", "best standing desk", "standing desk converter", "standing desk benefits",
    "l shaped standing desk", "standing desk mat", "walking pad"], "location": "United States"})
Enter fullscreen mode Exit fullscreen mode
Keyword Avg. monthly searches CPC Competition
walking pad 246,000 $1.24 HIGH
standing desk 135,000 $5.16 HIGH
standing desk converter 18,100 $2.24 HIGH
l shaped standing desk 12,100 $5.47 HIGH
best standing desk 8,100 $7.28 HIGH
standing desk benefits 2,900 $0.73 LOW
standing desk mat 1,900 $3.94 HIGH

Each row also carries top-of-page bids and a 12-month monthly_searches history. Two things already change the plan: "walking pad" has almost twice the demand of "standing desk" at a quarter of the CPC, and "best standing desk" has the highest CPC on the list ($7.28), which says comparison content on it monetizes. This costs $4 per 1,000 keywords, and keywords Google has no data for are free. (Full walkthrough: How to get exact Google search volume without a Google Ads account.)

Step 4: difficulty and intent 

kd = run("wrenfield-data/keyword-difficulty-checker", {"keywords": [
    "standing desk", "best standing desk", "standing desk converter", "standing desk benefits",
    "l shaped standing desk", "standing desk mat", "walking pad", "standing desk for small spaces"],
    "location": "United States", "includeSearchIntent": True})
Enter fullscreen mode Exit fullscreen mode
Keyword KD (0–100) Intent Volume*
standing desk 38 transactional 135,000
best standing desk 36 commercial 8,100
standing desk converter 7 transactional 18,100
standing desk benefits 31 commercial 2,900
l shaped standing desk 10 transactional 12,100
standing desk mat 7 transactional 1,900
walking pad 53 informational 246,000
standing desk for small spaces 10 commercial —

*Volume from Step 3 (Keyword Search Volume, Google Ads data).

"standing desk converter" (KD 7, 18,100/mo, $2.24 CPC, transactional) and "l shaped standing desk" (KD 10, 12,100/mo, $5.47 CPC) are the standouts: real buyer demand that advertisers pay for, and low difficulty.

Step 5: People Also Ask 

paa = run("wrenfield-data/people-also-ask-scraper", {"keywords": ["standing desk benefits"],
                                           "location": "United States", "clickDepth": 1})
Enter fullscreen mode Exit fullscreen mode

Questions Google showed (6 seconds):

  • Is standing for 7 hours a day unhealthy?
  • What is the 20/8-2 rule for standing desks?
  • What's the ideal height for a standing desk?
  • What are the downsides of using a standing desk?
    • ↳ Is a standing desk actually healthier?
    • ↳ How long should you stand at a standing desk per day?

These become the H2s of the explainer article.

The plan the data produces

  1. Buyer guide: standing desk converters (KD 7, 18.1K/mo).
  2. Buyer guide: L-shaped standing desks (KD 10, 12.1K/mo).
  3. Explainer: standing desk benefits, structured around the PAA questions, linking to both guides.
  4. Skip "standing desk" (KD 38) for now. Revisit once the site has links.

Five API calls, under a minute of total run time.

Same thing in n8n

Use the official Apify node, operation Run an Actor and get dataset, once per step: wrenfield-data/competitor-keywords → wrenfield-data/keyword-search-volume (exact volume + CPC for the whole list; keywords without data are free) → wrenfield-data/keyword-difficulty-checker → Filter (keyword_difficulty < 20) → wrenfield-data/people-also-ask-scraper on the survivors → Google Sheets. Paste each JSON input above into the node's input field.

One API instead of four

The SEO Toolkit exposes eight reports behind one tool parameter: domain_overview, ranked_keywords, competitors, top_pages, backlinks, referring_domains, keyword_overview, serp. That makes it a single tool for AI agents via the Apify MCP server.

{"tool": "ranked_keywords", "targets": ["upliftdesk.com"], "location": "United States", "maxResults": 1000}
Enter fullscreen mode Exit fullscreen mode

Cost and caveats

Everything is pay per result or per request, with no subscription; each Actor's Pricing tab lists the current prices and paid Apify plans get volume discounts. Traffic and traffic value are estimates (position × volume × CTR model), and every SEO data vendor's numbers differ. Use them to rank opportunities, not to forecast revenue.

Top comments (0)