Originally published on AIdeazz — cross-posted here with canonical link.
My first attempt at an AI content pipeline failed to generate any meaningful traffic. The system was elegant: a custom agent analyzed Google Search Console (GSC) for content gaps, fed the top 15 underperforming queries to Claude 3 Opus, which drafted articles, then another agent formatted and published to Dev.to and my personal site, aideazz.xyz. I built it to automate my content strategy, aiming for a consistent stream of SEO-optimized posts without manual intervention. After three months, the GSC data showed a flat line for new content. The problem wasn't the AI's writing quality or the publishing mechanics; it was my definition of a "content gap" and the underlying assumption that 15 GSC queries could drive a pipeline.
The "Content Gap" Fallacy with Limited Data
My initial GSC gap analysis agent was simple:
- Fetch all queries with impressions > 100 but average position > 20.
- Filter for queries not already covered by existing articles (based on URL mapping).
- Sort by impressions, pick the top 15.
This approach, while technically sound for identifying some gaps, completely missed the mark for a nascent site like aideazz.xyz. With only a few dozen articles published, my GSC data was sparse. "Impressions > 100" often meant 101 impressions for a query with zero clicks, indicating extreme irrelevance or a highly competitive keyword I had no business targeting. My agent was effectively picking low-hanging fruit that was actually rotten. The articles generated by Claude were technically correct, but they addressed topics where my site had no authority, no existing backlinks, and no chance of ranking. The "gap" was real, but my ability to fill it was non-existent.
For example, one of the top 15 queries was "Oracle Cloud Infrastructure free tier limits." While relevant to my tech stack, my site had one article mentioning OCI, not a deep dive. Claude produced a 1,500-word piece. It got 3 impressions and 0 clicks in a month. The competition was Oracle's own documentation and major cloud blogs. My AI content pipeline was generating high-quality, irrelevant content.
Re-engineering the GSC Gap Analysis Agent
The fix required a shift from purely quantitative metrics to a qualitative understanding of my site's existing authority and potential. I introduced several new filters and scoring mechanisms:
- Topical Cluster Affinity: Instead of just "not covered," the agent now calculates a cosine similarity score between the candidate query and the embeddings of my top 10 performing articles (by clicks and average position < 10). This uses
sentence-transformers/all-MiniLM-L6-v2locally. Queries with a similarity score below 0.6 are discarded. This ensures new content reinforces existing topical authority. - Keyword Difficulty Proxy: I integrated a simple proxy for keyword difficulty. For each candidate query, the agent performs a Google search and counts the number of results from high-authority domains (e.g., Wikipedia, major tech blogs, official documentation). If more than 5 of the top 10 results are from domains with a Moz Domain Authority > 70, the query is penalized. This is a crude but effective filter against highly competitive terms.
- Click-Through Rate (CTR) Potential: For queries with existing impressions, I now prioritize those with a higher-than-average CTR for their position. For instance, a query at position 30 with a 0.5% CTR is more interesting than one at position 25 with 0.1% CTR, assuming similar impression counts. This indicates existing user interest that can be amplified.
- Long-Tail Focus: I added a minimum word count filter for queries (at least 4 words). Short, broad keywords are almost always too competitive for a small site.
This revised GSC agent now outputs a prioritized list of 5-7 queries, not 15. The reduction in quantity is a deliberate trade-off for increased relevance and potential impact.
The Multi-Agent Orchestration: Claude, Groq, and Oracle Cloud
My AI content pipeline runs entirely on Oracle Cloud Infrastructure (OCI) using a mix of serverless functions (OCI Functions) and always-on compute instances for specific agents.
The GSC analysis agent, written in Python, runs as an OCI Function triggered weekly. It pulls data via the GSC API, processes it, and stores the prioritized queries in an OCI Object Storage bucket.
The article generation agent is more complex. It's a multi-stage process:
- Topic Selection: A dedicated agent monitors the Object Storage bucket. When new queries appear, it picks one.
- Outline Generation (Groq): For speed and cost-efficiency, the first step is to generate a detailed outline. I use Groq's Llama 3 8B model for this. The prompt is simple: "Generate a detailed, SEO-optimized outline for an article titled '[Query]' targeting developers. Include H2 and H3 headings, and suggest 3-5 internal links to aideazz.xyz/portfolio." Groq returns this in under 200ms for about $0.00005.
- Drafting (Claude 3 Opus): The outline is then passed to Claude 3 Opus via the Anthropic API. Claude is instructed to write a 1,200-1,800 word article, adhering strictly to the outline, maintaining a technical tone, and integrating the suggested internal links. I've found Claude 3 Opus to be superior for long-form, coherent technical content compared to other models, despite its higher cost ($15/M tokens). Each article costs roughly $2-$4 to generate.
- Review & Refinement (Custom Agent): Before publishing, a custom Python agent performs several checks:
- Plagiarism Check: Uses a combination of fuzzy matching against existing web content and a small internal database of my own articles.
- SEO Keyword Density: Ensures the target query and related keywords appear naturally.
- Readability Score: Checks Flesch-Kincaid grade level (aiming for 10-12 for technical content).
- Code Snippet Validation: If code is present, it attempts a basic syntax check using
astmodule for Python oresprimafor JavaScript. This is a weak check but catches obvious errors. - Internal Link Validation: Confirms suggested internal links are valid URLs on aideazz.xyz.
This refinement agent flags issues and sends a Telegram notification to me. If no issues, it proceeds.
Automated Publishing to Dev.to and aideazz.xyz
The final stage is publishing. I maintain two publishing agents:
- Dev.to Agent: This Python script uses the Dev.to API. It takes the refined article, converts it to Markdown, adds appropriate tags (derived from the article content using a small local LLM,
phi-3-mini), and sets the canonical URL to the aideazz.xyz version. It then publishes the article as a draft, sending me a Telegram notification with a link for final review. I manually hit "publish" on Dev.to. This semi-manual step is crucial for maintaining quality and preventing accidental spam. - aideazz.xyz Caching Agent: My personal site is a static site generated with Hugo. The agent takes the Markdown, converts it to a Hugo-compatible
.mdfile, and places it in the correct directory. It then triggers agit pushto my OCI Code Repository, which in turn triggers an OCI DevOps pipeline to rebuild and redeploy the site. This ensures the article is live on my primary domain and acts as the canonical source.
The entire process, from GSC analysis to site deployment, is designed to be hands-off, with human intervention only at the final Dev.to review stage. The cost per article, including all API calls and OCI compute, averages around $5-$7.
Results and Future Iterations
Since implementing the revised GSC gap analysis and multi-agent pipeline, I've seen a noticeable improvement. While I'm not generating viral content, the new articles are consistently ranking higher (average position 15-20 within a month) and attracting a small but steady stream of clicks. My GSC data for these new articles shows a 2-3% CTR, a significant improvement over the near-zero CTR of the previous iteration.
The biggest lesson was that "content gap" isn't just about what's missing, but what's achievable for your specific context. A small site needs to target micro-gaps within its existing authority, not broad, competitive terms.
My next iteration will focus on:
- Dynamic LLM Routing: Instead of fixed Claude 3 Opus for drafting, I'll implement a routing agent that evaluates the complexity of the outline and dynamically selects between Claude 3 Haiku, Sonnet, or Opus, or even a fine-tuned Llama 3 model on OCI, to optimize for cost and quality.
- Feedback Loop Integration: I plan to integrate GSC performance data back into the agent's query scoring algorithm. Articles that perform well will boost the topical affinity score for related queries, creating a self-reinforcing loop.
- Social Media Promotion Agent: An agent that drafts short promotional posts for Twitter/LinkedIn/Telegram, linking to the new article, using a model like Llama 3 8B.
This AI content pipeline isn't a magic bullet for instant SEO dominance, but it's a robust system for consistent, targeted content generation that respects the realities of a small, independent publisher.
Frequently Asked Questions
Q: Why not just use a commercial SEO tool for gap analysis?
A: Commercial tools are expensive ($100+/month) and often provide generic advice. My custom agent is tailored to my specific GSC data and site authority, costing me only OCI compute time (pennies per run) and giving me full control over the logic.
Q: Why use Groq for outlines and Claude for drafting? Why not one model?
A: Groq is incredibly fast and cheap for short, structured outputs like outlines. Claude 3 Opus excels at long-form, nuanced technical writing. This split leverages each model's strengths, optimizing for both speed/cost (Groq) and quality (Claude) in their respective tasks.
Q: How do you handle image generation for articles?
A: Currently, I don't automate image generation. I manually add relevant screenshots or diagrams. Automated image generation is a future enhancement, likely using a model like DALL-E 3 or Stable Diffusion, but it adds complexity and cost I haven't prioritized yet.
Q: What happens if the AI generates factually incorrect information?
A: The refinement agent catches some basic errors (e.g., broken links, syntax). For factual accuracy, the manual Dev.to review is my primary safeguard. I also prompt Claude to cite sources where appropriate, though this is not foolproof.
Q: Why publish to Dev.to and your own site? Isn't that duplicate content?
A: Publishing to Dev.to expands reach to a developer-focused audience. By setting the canonical URL on Dev.to to my aideazz.xyz article, I signal to search engines that my site is the original source, avoiding duplicate content penalties and consolidating SEO authority.
Top comments (0)