GSC Gap Analysis: From 15 Queries to Automated Publishing

#ai #programming #machinelearning

Originally published on AIdeazz — cross-posted here with canonical link.

My first AI content pipeline failed to move the needle. I spent $120 on Claude 3 Opus API calls, generating 30 articles. My Google Search Console (GSC) impressions for my target keywords barely budged. The "content gap" I thought I was filling was an illusion. I had 15 GSC queries, each with 2-3 impressions, and zero clicks. My initial assumption: if I write about these, I'll rank. This was a fundamental misunderstanding of what a content gap truly means for a bootstrapped operation.

The Illusion of the "Content Gap"

A "content gap" isn't just a keyword you don't rank for. For a new site, it's a keyword where you have some existing, albeit minimal, GSC impressions, but no clicks. More critically, it's a keyword where the intent is clear and the competition is manageable. My 15 queries were long-tail, low-volume, and often irrelevant to my core offerings. For example, "oracle cloud free tier limitations" had 2 impressions. While technically a "gap," writing a 2000-word article on it wouldn't bring me clients for AI agent development. It would bring people looking for free stuff.

My revised definition of a GSC content gap for AIdeazz:

Existing Impressions (10+): At least 10 impressions in the last 90 days, indicating Google already associates my site with the query.
Zero Clicks: No clicks, meaning my current content isn't satisfying the search intent.
High Relevance: The query directly relates to my AI agent development, multi-agent systems, or Oracle Cloud infrastructure expertise.
Low Competition (SERP analysis): A quick manual check of the top 5 results shows forums, outdated articles, or generalist blogs, not direct competitors with highly optimized content.

This refined definition immediately cut my "gap" list from 15 to 2. One was "oracle cloud always free instance stopped," and the other was "multi-agent system architecture." These were actionable.

Building the Automated GSC Gap Analysis Agent

I needed an agent to automate this filtering. My existing Oracle Cloud Infrastructure (OCI) tenancy, running on an Ampere A1 instance, was already hosting my core AI agents. I spun up a new Python environment.

The agent's flow:

GSC Data Fetch: Using the google-api-python-client library, I connect to the GSC API. I pull query data for the last 90 days, filtered by my aideazz.xyz property.

# Simplified GSC API call
service = build('searchconsole', 'v1', credentials=creds)
request = {
    'startDate': (datetime.now() - timedelta(days=90)).strftime('%Y-%m-%d'),
    'endDate': datetime.now().strftime('%Y-%m-%d'),
    'dimensions': ['query'],
    'rowLimit': 5000 # Fetch enough to analyze
}
response = service.searchanalytics().query(siteUrl='https://aideazz.xyz/', body=request).execute()
queries = response.get('rows', [])

Filtering Logic: I iterate through queries, applying my refined criteria.
```
actionable_gaps = []
for row in queries:
    query = row['keys'][0]
    impressions = row['impressions']
    clicks = row['clicks']

    if impressions >= 10 and clicks == 0 and is_relevant(query): # is_relevant is a simple keyword match for now
        actionable_gaps.append(query)
```
The is_relevant function is a simple regex match against a predefined list of keywords like "AI agent," "Oracle Cloud," "multi-agent," etc. For more sophistication, I'd embed a small, local LLM (like Llama 3 8B via Ollama) to classify relevance, but for now, keyword matching is sufficient and cheaper.
SERP Analysis (Manual for now, but planned for automation): For each actionable_gap, I manually perform a quick Google search. I look for weak competition. This step is crucial and currently the bottleneck. Automating this involves scraping SERP results, extracting titles/descriptions, and using an LLM to assess content quality and competitor strength. I'm prototyping this with BeautifulSoup and a local Llama 3 instance, but it's not production-ready due to rate limits and CAPTCHAs.

The output of this agent is a list of 1-3 highly targeted keywords.

Drafting with Claude 3 Haiku (and Groq for speed)

Once I have a target keyword, the next agent in the pipeline takes over: the drafting agent. My initial attempts used Claude 3 Opus, which was excellent but expensive ($15/M tokens). For blog posts, I found Claude 3 Haiku ($0.25/M tokens) to be perfectly adequate, especially when coupled with a detailed prompt.

My prompt template for Haiku:

You are an expert technical writer specializing in AI agent development and Oracle Cloud Infrastructure.
Write a detailed, practical blog post for technical founders and developers.
The topic is: "[ACTIONABLE_KEYWORD]".
Focus on:
- Real-world challenges and solutions.
- Specific technical details (e.g., API calls, infrastructure choices).
- Avoid hype and generic statements.
- Use a direct, no-nonsense tone.
- Include at least 3 code examples (Python or Bash).
- Structure with clear headings (H2, H3).
- Target word count: 1200-1800 words.
- Conclude with 3-5 specific, non-obvious FAQs.

I route this prompt through my custom agent router. This router dynamically selects the LLM based on cost, speed, and complexity requirements. For initial drafts, I often hit Groq's Llama 3 8B for a quick outline, then feed that outline to Claude 3 Haiku for the full draft. This hybrid approach saves significant cost and time. Groq's Llama 3 8B is incredibly fast (500+ tokens/sec) and cheap, making it ideal for rapid prototyping and outlining.

The drafting agent outputs the full Markdown content.

Automated Publishing to Dev.to and aideazz.xyz

The final stage is publishing. I maintain two primary content destinations:

Dev.to: For broader developer reach and community engagement.
aideazz.xyz: My canonical source, where I control the content and SEO.

Dev.to Publishing Agent

Dev.to has a well-documented API. My agent uses the requests library to post the Markdown content.

import requests
import json

DEVTO_API_KEY = os.getenv('DEVTO_API_KEY')
DEVTO_API_URL = "https://dev.to/api/articles"

def publish_to_devto(title, body_markdown, tags, series=None, canonical_url=None):
    headers = {
        "api-key": DEVTO_API_KEY,
        "Content-Type": "application/json"
    }
    article_data = {
        "article": {
            "title": title,
            "body_markdown": body_markdown,
            "published": True,
            "tags": tags,
            "canonical_url": canonical_url
        }
    }
    if series:
        article_data["article"]["series"] = series

    response = requests.post(DEVTO_API_URL, headers=headers, data=json.dumps(article_data))
    response.raise_for_status() # Raise an exception for HTTP errors
    return response.json()

# Example usage:
# publish_to_devto("My New Article", "## Heading...", ["ai", "oracle"], canonical_url="https://aideazz.xyz/my-new-article")

The canonical_url is critical here. It tells search engines that aideazz.xyz is the original source, preventing duplicate content penalties. I also automatically add relevant tags based on the article's content, again using a small LLM (Llama 3 8B) for classification.

aideazz.xyz Caching and Display

My main site, aideazz.xyz, is a static site generated with a custom Python script. When a new article is drafted and approved (currently, I manually review before final publish), the publishing agent:

Generates a unique slug: From the article title.
Creates a Markdown file: In my _posts directory.
Updates the site generator: Triggers a rebuild of the static site.
Invalidates Cloudflare cache: Ensures the new content is immediately visible.

This process ensures that aideazz.xyz always has the freshest content, and it's served directly from Cloudflare's CDN, making it fast and resilient. The entire process, from GSC analysis to publishing, takes about 15 minutes of compute time on my OCI Ampere instance, costing pennies. The most significant cost is the LLM API calls, which I've optimized down to $0.50-$1.00 per article using Haiku and Groq.

The real "gap" isn't just missing content; it's missing content that matters to your business, for which you already have a tiny footprint, and where you can realistically outcompete. My automated pipeline now focuses on this narrow, high-leverage target.

Frequently Asked Questions

Q: How do you handle the "is_relevant" function for GSC queries without a large LLM?
A: Currently, it's a simple keyword matching function. I maintain a YAML file of core business keywords (e.g., "multi-agent," "AI agent," "Oracle Cloud," "OCI," "autonomous database"). The function checks if at least two of these keywords are present in the GSC query. This is a heuristic, not perfect, but effective for filtering out noise.

Q: What's your process for manual review before publishing?
A: The drafting agent sends the Markdown output to a private Telegram channel via a custom bot. I review it on my phone, checking for factual accuracy, tone, and overall coherence. If it passes, I send a /publish command to the bot, which triggers the final publishing agents. This takes about 5-10 minutes per article.

Q: How do you manage LLM routing for cost and speed?
A: My custom router agent uses a simple decision tree. For outlines or quick summaries, it defaults to Groq's Llama 3 8B. For detailed, high-quality content generation (like blog posts), it prioritizes Claude 3 Haiku due to its balance of quality and cost. If Haiku is unavailable or too slow, it falls back to GPT-3.5 Turbo. This logic is configurable via environment variables.

Q: What if the GSC API returns too many queries to manually analyze SERPs?
A: If the filtered list of actionable gaps exceeds 5-7 queries, I prioritize based on the highest impression count. I also have a prototype SERP analysis agent that uses requests and BeautifulSoup to fetch the top 5 results for each query and then uses a local Llama 3 8B to summarize the content and assess competitor strength. This is still experimental due to rate limits and CAPTCHA challenges from Google.

— Elena Revicheva · AIdeazz · Portfolio