DEV Community

Eioki
Eioki

Posted on Edited on

How I Automated SEO Competitor Outlines by Combining Google SERP Data, Async HTML Crawling, and Gemini Structured Outputs

Writing SEO content that actually ranks requires analyzing what already sits on Page 1—their heading structures, word counts, FAQ coverage, and schema types. Doing that manually across 5–10 tabs for every keyword is slow, and asking an LLM to "write an outline" without live SERP data produces generic fluff.

To automate this research step, I built an Apify Actor (SERP & Competitor Content-Gap Analyzer) that fetches top Google results, crawls their live HTML concurrently, measures structural metrics, and uses Gemini Flash to generate an outrank blueprint.

How the Pipeline Works

  1. Live SERP & People Also Ask Extraction: Queries Google search results for your target keyword, country (gl), and language (hl), capturing organic rankings and People Also Ask questions.
  2. Concurrent Lightweight Page Analysis (httpx + BeautifulSoup): Crawls top competitor URLs asynchronously without headless browsers, extracting exact word counts, H1/H2 heading hierarchies, JSON-LD Schema.org types, and FAQ presence.
  3. Structured Content-Gap Blueprint (Pydantic + Gemini Flash): Outputs searchIntent, keywordDifficultyEstimate, recommendedWordCount, competitorBlindSpots, and a 6–10 heading outrankH2Outline.

Try It Out

👉👉 SERP & Competitor Content-Gap Analyzer on Apify Store

Top comments (0)