Originally published on AIdeazz — cross-posted here with canonical link.
My first attempt at an AI content pipeline failed to move the needle. After two months, publishing 30 articles, my Google Search Console (GSC) impressions were up 120%, but clicks only increased by 15%. The pipeline was efficient, generating drafts from Claude 3.5 Sonnet, refining with a custom agent, and publishing to Dev.to and my own site. The problem wasn't the how of publishing, but the what. I was generating content based on broad keyword research, not actual user intent gaps.
The "content gap" everyone talks about isn't a missing article on a topic. It's a missing answer to a specific, high-intent query where your existing content underperforms. For me, with a nascent site and minimal authority, this meant focusing on queries where I already had some impressions but zero clicks, or where my average position was 20+ for a relevant term. This isn't about finding new keywords; it's about optimizing for the 15 GSC queries that are already sending you traffic, however small.
Identifying the True Content Gap with GSC
I started by exporting GSC data for the last 90 days, focusing on "Queries" and "Pages." The critical filters were:
- Impressions > 10: To ensure some visibility.
- Clicks = 0: The ultimate gap – people see you, but don't click.
- Average Position > 15: Indicating I'm on page 2 or 3, a prime candidate for improvement.
This typically yielded a list of 10-20 queries. For my AIdeazz.xyz site, a query like "oracle cloud free tier gpu" showed 50 impressions, 0 clicks, and an average position of 22. My existing article on Oracle Cloud Free Tier mentioned GPUs but didn't dedicate a section to their availability or common pitfalls. This was a clear gap.
The next step was to map these queries to existing pages. GSC provides this mapping directly. If a query mapped to a page with an average position of 5 and 50% CTR, that wasn't a gap. If it mapped to a page with an average position of 25 and 0% CTR, that was a target. If it didn't map to any page, that was a new topic.
Agent-Driven Gap Analysis and Topic Generation
My content agent, running on Oracle Cloud Infrastructure (OCI) with a custom Python Flask backend, takes this GSC data. It's not a single LLM call; it's a multi-stage process:
- Query Grouping: For each identified "gap query," the agent first checks for semantic similarity against my existing article titles and summaries using
sentence-transformers. If a query is highly similar to an existing article, it's flagged for optimization (adding a section, rephrasing). If not, it's flagged for new content. - Intent Extraction: For new content, the agent uses Claude 3.5 Sonnet (routed via Groq for speed if the context window is small, otherwise directly to Anthropic) to extract the core user intent behind the query. For "oracle cloud free tier gpu," the intent is "can I get a free GPU on OCI, and if so, how, and what are the limitations?"
- Outline Generation: Based on the extracted intent, the agent generates a detailed article outline. This outline isn't just headings; it includes specific points to cover, potential sub-questions, and a target word count (usually 1200-1800 words for technical articles). This is crucial for controlling LLM output and ensuring depth. I found that providing 5-7 specific sub-points per section drastically improved article quality compared to just giving a heading.
This entire process, from GSC export to outline generation, takes about 3 minutes per query cluster. I process these in batches of 5-10 queries, manually reviewing the generated outlines for accuracy and relevance before proceeding. This human-in-the-loop step is non-negotiable for quality control.
Drafting with Claude and Iterative Refinement
The drafting phase is where Claude 3.5 Sonnet shines. I've experimented with GPT-4o and Llama 3, but for detailed technical explanations and maintaining a consistent, slightly formal tone, Claude has been superior for my use case.
The agent feeds the generated outline, along with 3-5 reference articles (scraped from top-ranking results for similar queries, filtered for quality), to Claude. The prompt is highly structured:
You are a senior technical writer. Your task is to write a comprehensive, accurate, and engaging article based on the provided outline and reference materials.
Tone: Professional, informative, slightly opinionated where appropriate (e.g., "I recommend X because...").
Audience: Developers, technical founders. Assume they are skeptical of hype.
Constraints:
- Adhere strictly to the outline provided.
- Integrate insights from the reference articles but do not plagiarize. Synthesize and rephrase.
- Use code examples where relevant.
- Avoid jargon where simpler terms suffice.
- Word count target: [X-Y words].
- Focus on practical advice and real-world implications.
- Do not include an introduction or conclusion section unless explicitly in the outline.
The first draft is rarely perfect. My agent then performs a series of automated checks:
- Fact-checking (limited): For specific numbers or product names, it queries a custom knowledge base (a small vector database of my own articles and verified documentation).
- Readability Score: Flesch-Kincaid score is calculated. If too high, a revision prompt is sent to Claude.
- Keyword Density: Checks for natural inclusion of the target query and related terms.
- Structure Validation: Ensures all outline points are covered.
If any of these checks fail, the agent sends a targeted revision prompt to Claude. For example, "The section on 'OCI GPU pricing' is too vague. Please add specific pricing examples for A10 and V100 instances in different regions." This iterative refinement loop typically involves 1-2 revisions and reduces manual editing time by 60-70%.
Automated Publishing and Caching
Once the article draft passes automated and manual review, it's ready for publishing. My pipeline targets two platforms:
- Dev.to: This is my primary external distribution channel. The agent uses the Dev.to API to create a new post, including Markdown content, tags, and a canonical URL pointing back to my site. I found that publishing to Dev.to first often gives a small initial boost in visibility and backlinks.
- AIdeazz.xyz: My own site, built with a custom Flask app. The agent pushes the Markdown content to a specific endpoint, which then renders it into an HTML page and updates the sitemap. This also triggers a cache invalidation for that specific URL.
The entire publishing process, from final draft approval to live articles on both platforms, takes less than 30 seconds. The canonical URL setup is critical to avoid duplicate content penalties from Google.
The Impact: 15 Queries, 30% CTR Increase
After implementing this GSC-driven gap analysis for 15 specific queries, I saw a measurable improvement. For the targeted queries, the average CTR increased from 0% to 3.2% within 6 weeks. Impressions for these queries also increased by 40%, indicating that Google was re-evaluating the relevance of my content. My average position for these terms improved from 22 to 11.
This isn't a "hockey stick" growth story. It's a story of incremental, targeted improvements based on real user data. The key was shifting from broad keyword targeting to hyper-specific query optimization. The AI agents didn't replace my strategic thinking; they amplified my ability to execute on it, turning GSC data into published content at scale.
Frequently Asked Questions
Q: How do you handle GSC data privacy and API limits when automating?
A: I use the GSC API with a service account, storing credentials securely in OCI Vault. API limits are generous for typical usage (2000 queries/day), but I batch requests and implement exponential backoff for robustness.
Q: What's the cost breakdown for running this pipeline on Oracle Cloud?
A: My OCI VM.Standard.E4.Flex instance (4 OCPUs, 64GB RAM) costs about $120/month. LLM API calls (Claude 3.5 Sonnet, Groq) vary, but average $50-80/month for 20-30 articles. Total infrastructure and LLM costs are under $200/month.
Q: How do you ensure the AI-generated content doesn't sound generic or repetitive?
A: The detailed outlines and iterative refinement prompts are key. I also inject specific "voice" instructions into the initial prompt (e.g., "use a slightly opinionated tone," "focus on practical advice"). Manual review of the first draft is still essential to catch generic phrasing.
Q: What if the GSC data is too sparse for a new site?
A: For truly new sites with minimal GSC data, I start with competitor analysis. I identify 3-5 competitors, scrape their top-performing articles (using tools like Ahrefs or Semrush, or manual inspection), and use those topics as initial targets. Once GSC data accumulates, I pivot to the gap analysis.
Q: How do you prevent the AI from hallucinating or providing outdated information?
A: For critical facts, the agent queries a custom knowledge base (vector DB of verified documentation). For general information, the reference articles help. I also explicitly instruct Claude to state when information might be subject to change (e.g., "pricing is subject to change, check official documentation").
Top comments (0)