Darian Vance

Posted on Jan 3 • Originally published at wp.me

Solved: Do Ahrefs & SemRush Still Work for Keyword Research After Google's Updates?

#devops #programming #tutorial #cloud

🚀 Executive Summary

TL;DR: Google’s algorithm shifts and the rise of AI Overviews (SGE) challenge the traditional utility of Ahrefs and SemRush for keyword research, making exact-match keyword efficacy and traffic predictions less reliable. The solution involves a blended strategy, integrating Google-first data (GSC, GKP, manual SERP analysis) with intent-driven topic clustering and leveraging these tools for competitive intelligence and technical SEO.

🎯 Key Takeaways

Google Search Console (GSC) is the most valuable, unfiltered source of truth for actual query performance, CTR, and emerging keywords, directly from Google, providing factual data over third-party tool estimates.
Modern SEO requires shifting focus from exact-match keywords to understanding diverse search intent (informational, navigational, transactional, commercial investigation) and building comprehensive content hubs with topic clusters to demonstrate deep topical authority.
The prevalence of AI Overviews and rich snippets leads to ‘zero-click’ searches, necessitating content that is exceptionally comprehensive or offers a unique perspective to entice clicks, as many queries are now answered directly on the SERP.

The recent shifts in Google’s search algorithms and the rise of AI Overviews are challenging traditional keyword research methodologies. This post explores how Ahrefs and SemRush remain valuable, despite these changes, by focusing on a blended strategy leveraging Google-first data and intent-driven analysis.

The Keyword Research Conundrum: Are Traditional Tools Still Relevant?

The digital landscape is in constant flux, with Google’s search algorithms evolving at an unprecedented pace. Recent updates, particularly those emphasizing user experience, E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness), and the integration of AI Overviews, have led many IT professionals and SEO strategists to question the enduring utility of established keyword research platforms like Ahrefs and SemRush.

The Reddit thread “Are Ahrefs and SemRush currently useful for keyword research? Or did the recent Google search result changes limit their usefulness?” perfectly encapsulates this industry-wide apprehension. While these tools have long been the backbone of SEO, their efficacy for pinpointing exact-match keywords and predicting traffic volumes seems to be diminishing. This article delves into the symptoms of this shift and provides actionable, problem-solving solutions for navigating the new SEO reality, ensuring your digital strategy remains robust and data-driven.

Symptoms of Eroding Keyword Tool Efficacy

Before we dive into solutions, it’s crucial to identify the manifestations of this problem within your own operations. Recognizing these symptoms helps in accurately diagnosing the challenge and implementing targeted strategies.

Volatility in SERP Rankings & Traffic Predictions

One of the most noticeable symptoms is the increased unpredictability of Search Engine Results Pages (SERPs). What ranked well last month might have significantly shifted, not due to overt penalization, but a re-evaluation by Google’s algorithms. Keyword difficulty scores and estimated traffic volumes from Ahrefs or SemRush may no longer align with actual performance, leading to misallocated resources and inaccurate forecasting for content teams.

Discrepancies Between Tool Data and Google Analytics

Teams frequently observe a growing disparity between the traffic and ranking data reported by third-party SEO tools and the direct, first-party data from Google Analytics and Google Search Console (GSC). While some variance is expected, significant and consistent gaps indicate that the third-party tools might be struggling to keep pace with Google’s real-time index and nuanced ranking factors. For instance, a keyword showing high search volume in a tool might drive minimal actual traffic, or vice-versa, due to nuances in intent or the prevalence of AI Overviews satisfying queries directly.

The Rise of AI Overviews and Zero-Click Searches

The introduction of AI Overviews (formerly SGE) and enhanced rich snippets means an increasing number of user queries are answered directly on the SERP, leading to “zero-click” searches. This fundamentally changes the value proposition of targeting specific keywords. Users get immediate answers, reducing the need to click through to a website. Tools designed for traditional click-through models struggle to account for this shift, making their volume estimates less indicative of potential website traffic.

Solution 1: Augmenting Traditional Tools with Google-First Data

Instead of abandoning Ahrefs or SemRush, the first solution is to integrate them more closely with the authoritative data sources directly from Google. This approach provides a more accurate, real-time understanding of user behavior and search performance.

Leveraging Google Search Console (GSC)

Google Search Console is your most valuable, unfiltered source of truth directly from Google. It shows you exactly which queries your site is ranking for, your average position, click-through rate (CTR), and impressions. This data is not an estimate; it’s factual performance data.

Identify Emerging Queries: Use GSC’s “Performance” report to filter queries by position (e.g., positions 8-20) and impressions. These are keywords you’re almost ranking for, and with a bit of optimization, could jump significantly.

  # Conceptual Python script to analyze GSC data exports
  import pandas as pd

  def analyze_gsc_queries(csv_path):
      df = pd.read_csv(csv_path)

      # Filter for queries with impressions, but not yet top positions
      emerging_queries = df[(df['Impressions'] > 100) & 
                            (df['Average position'] > 7) & 
                            (df['Average position'] <= 20)].sort_values(by='Impressions', ascending=False)

      print("Top Emerging Queries (Positions 8-20, high impressions):")
      print(emerging_queries[['Queries', 'Impressions', 'Average position', 'Clicks']].head(10))

      # Identify potential content gaps (queries with high impressions but low clicks)
      low_ctr_queries = df[(df['Impressions'] > 500) & 
                           (df['Clicks'] < 20)].sort_values(by='Impressions', ascending=False)

      print("\nQueries with High Impressions but Low Clicks (Potential CTR issues or AI Overviews):")
      print(low_ctr_queries[['Queries', 'Impressions', 'Average position', 'Clicks', 'CTR']].head(10))

  # To use:
  # 1. Go to GSC -> Performance -> Export the 'Queries' data (Google Sheets, CSV, Excel).
  # 2. Save as 'gsc_queries.csv'
  # 3. Call the function:
  # analyze_gsc_queries('gsc_queries.csv')

Prioritize Based on Actual CTR: Instead of relying on estimated click-through rates from third-party tools, use GSC’s actual CTR to understand what’s truly resonating with users on the SERP. Low CTR for high-impression queries might indicate an AI Overview is answering the question, or your snippet isn’t compelling enough.
Spot Disappearing Keywords: Monitor keywords that previously drove traffic but are now seeing declining impressions or average positions. This could signal a major algorithm shift affecting your content.

Utilizing Google Keyword Planner (GKP)

While primarily an advertising tool, Google Keyword Planner offers direct volume data from Google. Its data is often aggregated and less granular than Ahrefs/SemRush, but it represents Google’s own understanding of search demand.

Broad Volume Validation: Use GKP to validate the general search volume trends for broad topics identified through Ahrefs/SemRush. If a tool shows extremely high volume for a niche term, cross-reference with GKP for a reality check.
New Keyword Discovery: GKP can still uncover new keyword ideas, especially long-tail variations, by providing “Related keywords” based on Google’s own understanding of topical relevance.

Manual SERP Analysis & “People Also Ask”

Directly observing the SERP for your target keywords is critical. Look beyond the top organic results:

Identify AI Overviews: Does Google immediately answer the query with an AI Overview or a rich snippet? If so, your content needs to be exceptionally comprehensive or offer a unique perspective to entice a click.
“People Also Ask” (PAA) Integration: The PAA section is a goldmine for understanding related user intent and common follow-up questions. This allows you to build out more holistic content that satisfies multiple aspects of a user’s query. Incorporate these into your content strategy to improve topical authority.
Analyze Top-Ranking Pages: What content format, depth, and unique angles are the top-ranking pages using? This helps you understand what Google currently considers the “best” answer for a given query, informing your content strategy.

Solution 2: Shifting Focus from Keywords to Topics and Intent

The era of hyper-focusing on exact-match keywords is waning. Google is increasingly sophisticated at understanding natural language and user intent. Our strategy must evolve to match this by focusing on comprehensive topic coverage and satisfying diverse user intents.

Understanding Search Intent

Before optimizing for a keyword, understand *why* someone is searching for it. There are generally four types of search intent:

Informational: Users seeking answers to specific questions (e.g., “how to fix a leaky faucet”).
Navigational: Users looking for a specific website or page (e.g., “Amazon login”).
Transactional: Users looking to buy something (e.g., “buy ergonomic chair”).
Commercial Investigation: Users researching products or services before purchasing (e.g., “best cloud hosting providers”).

Your content must align with the primary intent of the target query. Ahrefs and SemRush can still help by showing the SERP features (e.g., shopping results, local packs, knowledge panels) for a keyword, which hints at its dominant intent.

Topic Clustering and Content Hubs

Instead of creating isolated articles for individual keywords, adopt a content hub and spoke model. This involves creating a comprehensive “pillar page” on a broad topic, supported by multiple “cluster pages” that delve into specific sub-topics related to the pillar. This approach demonstrates deep topical authority to Google.

For example, a pillar page on “Cloud Computing Security” could be supported by cluster pages on “AWS Security Best Practices,” “Azure Firewall Configuration,” “Container Security,” and “DevSecOps Workflows.”

# Conceptual approach to identifying topics for clustering using natural language processing (NLP)
# This isn't a runnable script but illustrates the concept.

import spacy
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

# Load a pre-trained NLP model (e.g., for English)
# nlp = spacy.load("en_core_web_sm") 

def analyze_content_for_topics(documents):
    """
    Analyzes a collection of text documents to identify underlying topics.

    Args:
        documents (list): A list of strings, where each string is a document (e.g., blog post content).

    Returns:
        dict: A dictionary mapping topic clusters to keywords/documents.
    """

    # 1. Preprocess text: Tokenization, lemmatization, stop word removal (using spaCy)
    processed_docs = []
    # for doc in documents:
    #     tokens = [token.lemma_ for token in nlp(doc.lower()) if not token.is_stop and token.is_alpha]
    #     processed_docs.append(" ".join(tokens))

    # Placeholder for actual processed docs
    processed_docs = [
        "cloud security best practices aws azure container",
        "devsecops CI CD pipelines automation tools",
        "aws IAM roles policies security groups",
        "azure network security group firewall configuration",
        "kubernetes docker container orchestration vulnerability"
    ]

    # 2. Vectorize documents using TF-IDF
    vectorizer = TfidfVectorizer(max_features=1000, ngram_range=(1, 2))
    X = vectorizer.fit_transform(processed_docs)

    # 3. Cluster documents using KMeans
    num_clusters = 3 # Example: We want to find 3 main topics
    kmeans = KMeans(n_clusters=num_clusters, random_state=0, n_init=10)
    kmeans.fit(X)

    # 4. Extract top keywords for each cluster
    feature_names = vectorizer.get_feature_names_out()
    cluster_keywords = {}
    for i in range(num_clusters):
        top_indices = X[kmeans.labels_ == i].sum(axis=0).argsort().tolist()[0][::-1][:10]
        cluster_keywords[f"Topic {i+1}"] = [feature_names[j] for j in top_indices]

    print("Identified Topic Clusters and Top Keywords:")
    for topic, keywords in cluster_keywords.items():
        print(f"- {topic}: {', '.join(keywords)}")

    # You can then map documents to their clusters
    # for i, doc in enumerate(documents):
    #     print(f"Document {i+1} -> Cluster {kmeans.labels_[i] + 1}")

# Example usage with placeholder documents (replace with real content from your site)
# your_website_content = [
#     "A comprehensive guide to securing your cloud infrastructure on AWS and Azure...",
#     "Implementing DevSecOps principles for continuous integration and deployment...",
#     "Understanding AWS Identity and Access Management (IAM) for robust security...",
#     "Configuring Azure Network Security Groups and firewalls...",
#     "Best practices for securing Docker and Kubernetes containers..."
# ]
# analyze_content_for_topics(your_website_content)

# This conceptual script helps demonstrate how one might programmatically find
# relationships between content pieces to form topic clusters.
# Real-world implementation would involve more sophisticated NLP and larger datasets.

Solution 3: Integrating AI-Powered and Predictive Analytics

The future of keyword research lies in leveraging advanced analytics and AI to predict trends, understand nuance, and identify opportunities that traditional methods might miss. While Ahrefs and SemRush are adapting, it’s essential to understand their evolving role alongside other tools and methodologies.

Exploring Next-Generation Tools and Techniques

Beyond the established giants, consider tools and approaches that focus on emerging trends and question-based research:

AnswerThePublic / AlsoAsked: These tools visualize questions people are asking around a keyword. They are excellent for identifying informational intent and crafting content that directly answers user queries, which is increasingly important for AI Overviews and rich snippets.
Google Trends: For identifying seasonal interest, comparing multiple keywords, and spotting emerging trends before they hit peak search volume. This is crucial for proactive content planning.
Large Language Models (LLMs) for Content Ideation: Use tools like ChatGPT or Gemini to brainstorm related topics, generate outlines, or even draft initial content based on a core keyword/topic. While not for direct content creation, they can accelerate the research and ideation phase.

The Role of Predictive Analytics

Instead of just reacting to current search volumes, aim to anticipate future demand. This involves:

Industry Trend Analysis: Stay abreast of broader industry shifts, technological advancements, and public discourse. These often precede changes in search behavior.
Social Listening: Monitor social media platforms and forums for discussions, questions, and pain points related to your niche. These can be early indicators of unmet information needs.
Competitor Analysis (with a twist): Use Ahrefs/SemRush not just to see what competitors rank for, but to identify *new* content areas they are exploring. Are they publishing on topics not yet heavily saturated? This could indicate a forward-looking strategy.

Ahrefs/SemRush in the New Paradigm (Comparison)

So, are Ahrefs and SemRush still useful? Absolutely, but their utility has shifted from being the sole source of keyword truth to powerful analytical platforms for competitive intelligence, technical SEO, and content gap analysis within a broader, more nuanced strategy.

Feature / Aspect	Ahrefs (New Paradigm Focus)	SemRush (New Paradigm Focus)
Keyword Research (Traditional)	Still provides broad volume estimates, difficulty scores. Best used for identifying keyword clusters and long-tail variations, then cross-referenced with GSC.	Similar to Ahrefs, offers volume & difficulty. Strong for finding related keywords and questions. Emphasize intent filters for better targeting.
Competitive Analysis	Exceptional for competitor backlink profiles, identifying top-performing content, and finding content gaps where competitors rank but you don’t.	Robust for competitor keyword rankings, traffic analysis, and “gap analysis” to find opportunities where you can outrank. Strong for PPC competitor intelligence.
Backlink Analysis	Industry-leading database for backlink profiles. Crucial for understanding competitor authority, identifying link building opportunities, and monitoring your own profile.	Strong backlink audit and analysis tools. Useful for disavowing toxic links and finding outreach prospects, though Ahrefs often has a larger index.
Content Gap & Topic Discovery	Identifies keywords competitors rank for that you don’t. Use “Content Explorer” to find high-performing content on topics relevant to your niche, irrespective of keyword volume.	“Topic Research” tool helps find subtopics and questions related to a broad seed. “Content Gap” feature effectively shows competitive content opportunities.
Technical SEO Audits	“Site Audit” is powerful for identifying technical issues (crawlability, indexability, site speed, broken links). Crucial for ensuring your site is Google-friendly.	Comprehensive “Site Audit” with detailed reports on crawl errors, performance issues, and on-page SEO factors. Excellent for ongoing site health monitoring.
AI Integration / Future Readiness	Investing in AI-driven features, but primary strength remains in its extensive data index for competitive analysis and technical SEO.	More proactive in integrating AI features, like content creation suggestions and writing assistants that help optimize for intent and comprehensiveness.
Best Use Case (Modern)	Strategic competitive intelligence, in-depth backlink analysis, and identifying content/topic gaps from a ‘what’s working for others’ perspective.	Holistic SEO management, deep keyword intent analysis, local SEO, PPC research, and integrating content writing workflows.

Conclusion: A Hybrid Approach for Sustainable SEO

The question isn’t whether Ahrefs and SemRush are still useful, but how their utility has evolved. In the post-AI Overview era, relying solely on their estimated keyword volumes for content strategy is a suboptimal approach. Instead, IT professionals and SEO strategists must adopt a hybrid methodology:

Data Augmentation: Always cross-reference third-party tool data with Google Search Console and Google Keyword Planner for a realistic understanding of performance.
Intent-First Strategy: Shift focus from isolated keywords to comprehensive topic coverage that addresses diverse user intents, building content hubs and pillar pages.
Competitive Intelligence: Leverage Ahrefs and SemRush for their unparalleled competitive analysis, backlink insights, and technical SEO auditing capabilities. Use them to understand *how* competitors are adapting, not just *what* they rank for.
AI and Predictive Analytics: Integrate next-generation tools and methodologies to anticipate trends, understand question-based search, and generate more relevant content ideas.

By embracing this multi-faceted, adaptive strategy, your team can navigate Google’s evolving landscape effectively, ensuring sustainable organic growth and maximizing the value of your SEO investments.