Here's a number that hurt: 50,000 Google impressions over three months, seven clicks total.
That was the scoreboard for a batch of AI-written comparison articles I shipped earlier this year. The content read fine. Grammar was clean. Structure was solid. And Google wanted absolutely nothing to do with it — average position 76, which is politely described as "page 8."
The problem wasn't the writing. It was that the model writing those articles had never seen what actually ranks for those keywords. It was writing in a vacuum and hoping.
The core insight
Every "AI content doesn't rank" complaint I've read comes down to the same root cause: the model generates from its training data's statistical average of the topic, not from what the search results reward today.
The top 10 results for a keyword are the most direct signal you can get about what Google's algorithm and users jointly consider good content for that query. Word count. Heading structure. Which subtopics every result covers. Which ones only two results bother with. Whether the intent is "compare tools" or "learn a concept."
If your AI writer doesn't receive that signal, it's guessing. If it does, it's filling in a template that already provably works.
So I built the pipeline I wished I had: SERP data in, structured brief out, article generated from that brief.
The pipeline, step by step
Step 1: Pull the live SERP
I use SerpAPI for this — their free tier (250 searches/month) is enough to bootstrap. One call gets you the top 10 organic results with their URLs, titles, and snippets for a target keyword.
def fetch_serp(keyword: str, api_key: str) -> list[dict]:
resp = requests.get(
"https://serpapi.com/search.json",
params={"q": keyword, "num": 10, "api_key": api_key},
timeout=30,
)
return resp.json().get("organic_results", [])
Nothing fancy. The magic is what you extract next.
Step 2: Extract the content pattern
For each ranking result, I fetch the page and pull out the bones:
- Word count — if all 10 results are 2,400+ words and you generate 800, you lose before you start
- H2/H3 outline — which section headings repeat across results? Those are the subtopics the intent demands
- Entity coverage — which product names, features, and concepts appear in most results? The model needs to know these are table stakes
- Intent classification — comparison posts, listicles, how-to guides, and concept explainers have different skeletons. A keyword like "asana vs monday" wants tables and pricing sections; "what is kanban" wants definitions and diagrams
The output is a structured brief that looks something like:
{
"keyword": "asana vs monday",
"intent": "comparison",
"target_word_count": 2400,
"required_sections": [
"quick comparison table",
"pricing breakdown",
"feature differences",
"use-case fit"
],
"must_mention_entities": [
"timelines", "workload management", "free tier",
"automation rules", "integrations"
]
}
This brief is the whole ballgame. It's the difference between "write an article about Asana vs Monday" and "here is exactly what the top 10 results collectively look like — match the pattern, then add something they missed."
Step 3: Generate with the brief as context
The prompt construction matters more than the model choice. My system prompt doesn't ask for an article — it asks the model to fill a structural slot:
You are writing one section of a B2B comparison article.
Keyword: {keyword}
Search intent: {intent}
This section: {section_title}
Cover these entities where natural: {entities}
The other sections in this article cover: {other_sections}
Do not repeat their content.
Write 150-250 words. No filler. No "in conclusion".
Generating section by section instead of all at once does two things: each call stays well under token limits, and you can regenerate one weak section without touching the rest.
I run DeepSeek as the default engine — at ~120 tokens/sec it's fast enough that a full article lands in about a minute and a half. But here's a decision I'd recommend to anyone building similar tooling: make the provider swappable from day one. My pipeline takes three env vars (base URL, API key, model name) and works with any OpenAI-compatible API. When a better or cheaper model ships next month, it's a config change, not a rewrite.
Step 4: The gotchas that cost me a weekend
Three things the tutorials don't mention:
Timeouts will lie to you. A 90-second generation timeout sounds generous until you're generating five sections sequentially and one hangs at 89 seconds. Set the timeout per-section, retry once, and if a section fails twice, generate it standalone later — don't block the whole article.
Max tokens is not a suggestion. DeepSeek's 8192 max_tokens cap will silently truncate a long generation mid-sentence if you're not explicit. Section-by-section generation mostly sidesteps this, but set the cap anyway.
Structured output drifts. If you ask for JSON and the model wraps it in markdown fences "to be helpful," your parser dies. Strip fences before parsing, always. (resp.strip().removeprefix("json").removesuffix("") has earned its place in my codebase.)
Does it work?
The honest answer: it's early, and one pipeline can't fix a brand-new domain with zero backlinks. But the articles generated from SERP briefs are visibly different from vacuum-generated ones — they cover the subtopics the intent demands, they're the right length, and they include the entities that would look conspicuous by their absence.
The comparison articles from my batch that were closest to the SERP pattern were also the only ones that cracked the top 50. Small sample, but the direction matches what every content strategist says offline: meet the intent, then differentiate.
What I did with it
I packaged the whole pipeline into SerpCraft — you paste a keyword, it pulls the live SERP, extracts the pattern, and drafts the article section by section. There's a free tier (3 analyses/month, no card) if you want to see the brief it generates for one of your keywords — even if you write the article yourself, the extracted pattern is a decent editorial checklist.
The interesting questions now are all on the distribution side, which is a different kind of hard. If you've taken an AI content tool from zero to actual users, I'd genuinely like to hear what worked.
Building in public, slowly. Stack: Next.js 14, Supabase, DeepSeek (swappable), Cloudflare Pages.
Top comments (2)
tbh seeing 7 clicks out of 50k is wild, but how do you actually parse the SERP data so it stays fresh if you're writing in batches a week later
Fair question — the pattern degrades slower than the rankings. What I extract is mostly structural: word count bands, which H2s repeat across results, entity coverage, intent class. Those shift on a monthly cadence, not weekly. The volatile parts are positions and pricing snippets.
My rule: if the gap between fetch and generation is under 48 hours, write straight from the brief. If a batch sits longer than a week, I re-pull the SERP and diff the brief — one API call per keyword, and anything that changed gets regenerated.
For pricing-sensitive articles I treat the SERP as a structure guide and verify current numbers at publish time anyway. The SERP tells you what to cover; it shouldn't be your only source for facts that change.