Recently, I conducted a research project on the relationship between Google AI Overviews and organic search rankings, and published the findings on my website:
https://heysho.com/en/geo/research-aio-organic-ranking.html
Rather than going into detail about the findings themselves, in this article I’d like to share some of the behind-the-scenes aspects of the research that I couldn’t fully cover in the original report, including:
- Why I decided to conduct this research
- How I analyzed around 1,000 search queries
- What I found challenging during the implementation
- What I learned from actually looking at the data
I’ll also share some of the challenges, discoveries, and interesting lessons I encountered throughout the research process.
1. Do Google Rankings Still Matter in the AI Overview Era?
Over the past six months or so, I’ve been spending a lot of time researching GEO and AI search. I’ve created my own reports and looked closely at how brands appear in actual AI-generated answers.
One question kept coming up:
“How much does a brand’s Google ranking affect its chances of appearing in AI-generated answers?”
As I looked at different brands, I started noticing some interesting patterns.
Some brands don’t seem particularly focused on SEO, yet they are frequently recommended by ChatGPT and other AI search platforms.
On the other hand, some companies perform very well in Google Search, but their brand names don’t appear nearly as often in AI-generated answers.
That made me wonder:
How much do traditional SEO rankings really influence visibility in AI search?
There was also a more practical question I wanted to answer: What ranking position should we actually aim for?
When working on SEO for a large website, you might track KPIs such as:
- The number of non-branded keywords ranking in the Top 10
- The number of keywords ranking in the Top 5
But once AI search enters the picture, it becomes less clear which ranking thresholds actually matter.
- Should we aim for the Top 3?
- Is the Top 5 enough?
- Does ranking in the Top 10 still make a meaningful difference?
- Or can a position around 15th still have some impact?
The answer could affect how we prioritize SEO initiatives.
That’s what motivated me to collect the data myself and compare Google organic rankings with visibility in AI Overviews.
2. How I Analyzed 1,000 Keywords
In the end, I analyzed around 1,000 keywords, but I didn’t start by simply putting together a list of 1,000 queries.
First, I used Semrush to collect a large set of product-category keywords. The initial dataset contained nearly 100,000 keywords.
After removing obviously irrelevant terms, I narrowed the list down to around 50,000.
Classifying that many keywords manually would have taken a huge amount of time, so I used Python, LangChain, and LLM APIs to help automate the process.
For this study, I then narrowed the scope further to queries in the format “product category + recommended.”
For example:
- “T-shirt recommended”
- “hair dryer recommended”
- “tent recommended”
The reason I chose this query format was simple: it makes it easier to analyze whether specific brands appear in AI Overview responses.
If someone searches only for “T-shirt,” the intent is relatively broad and ambiguous.
With a query like “recommended T-shirts,” however, the response is much more likely to include specific product or brand recommendations.
I created keyword pools across five categories—beauty, consumer electronics, apparel, furniture, and outdoor—and eventually narrowed the dataset down to around 15,000 candidate keywords.
From there, I selected the top 200 keywords in each category by search volume, giving me roughly 1,000 queries in total.
I didn’t jump straight to 1,000 queries, though.
I started small, testing with 20, then 50, and finally 100 queries.
By the time I had analyzed around 100 queries, the overall pattern was already becoming fairly clear. That suggested I probably didn’t need to analyze tens of thousands of keywords to see the broader trend.
So for the final study, I scaled the analysis up to around 1,000 queries.
3. What Did I Collect with Python + SearchApi?
I used SearchApi to retrieve Google search results.
Each request returns a JSON response containing a wide range of SERP data, including:
- Organic search results
- AI Overviews
- References and citations within AI Overviews
- Ads
- Shopping results
Technically, it’s possible to build your own Google Search scraping pipeline, and I’ve written similar scripts myself in the past.
But once you need to collect hundreds or thousands of SERPs reliably, managing the entire process yourself can become quite a lot of work.
Being able to retrieve all of this data quickly through an API made the research process much easier.
I think this kind of service is particularly useful for SEO agencies that need to collect SERP data at scale across multiple clients, as well as for teams working on large websites or global SEO projects.
For individual use, however, there is still a trade-off between convenience and monthly cost. That was my impression after using it for this project.
For this research, I mainly extracted Google organic rankings along with the AI Overview response text and citations, then linked the two datasets together for analysis.
4. The Biggest Challenge: Brand Name Variations
One thing that turned out to be surprisingly difficult during the implementation was identifying brand names accurately.
Brand names can appear in many different forms in AI-generated answers.
For example, the same brand might appear as:
- Its English name
- Its name written in katakana
- The full legal company name
- Just the brand name
- An abbreviation
- A version with or without spaces
If you rely only on exact string matching, the same brand can easily be treated as multiple different brands.
To address this, I created a brand-name dictionary and added a normalization process so that different variations could be mapped back to the same brand.
Even then, it was difficult to automate the process completely, so I also manually reviewed some of the results.
This made me realize that brand-name variation is an important issue when using GEO / AI search tracking tools for Japanese queries.
If you’re using a GEO tool developed outside Japan, it may be worth checking how well it recognizes an English brand name, its katakana version, and the Japanese legal entity name as the same brand.
A dashboard may show clean-looking numbers, but if the underlying brand matching is inaccurate, the reported mention rate can change significantly.
It’s not the most glamorous part of the process, but it ended up being one of the most important lessons from this research.
5. How Often Do AI Overviews Actually Appear?
Of the roughly 1,000 queries I analyzed, AI Overviews appeared for around 70% of them.
Personally, I found that surprisingly high.
Of course, this was probably influenced by the fact that the study focused heavily on “product category + recommended” queries.
For example, Google may show very different AI Overview results for a search like “bicycle” or “T-shirt” compared with “recommended bicycles” or “recommended T-shirts.”
So if your goal is to measure how often your brand is recommended in AI search, it may be more useful to track queries that are likely to trigger brand recommendations rather than relying only on generic category terms.
For example:
- “recommended [product category]”
- “popular [product category]”
- “recommended [product category] brands”
- “[product category] rankings”
Designing queries around recommendation intent makes it much easier to measure how often your brand appears in AI-generated answers.
6. If You Rank #1 on Google, Are You More Likely to Appear in AI Overviews?
This was the question I was most interested in answering.
If a brand ranks #1 on Google, is it also more likely to appear in AI Overviews?
The results were quite interesting.
Among companies and brands ranking #1 in Google Search:
- The AI Overview citation rate was around 52%.
- The brand mention rate was around 20%.
The citation rate was higher than I expected. In other words, pages ranking #1 in organic search were also cited as a source in AI Overviews in roughly half of the cases.
The brand mention rate, however, was much lower at around 20%.
Before running the study, I had assumed that if a brand ranked #1 on Google, its name would also appear in AI Overviews fairly often.
In reality, that happened only about one in five times.
That gap between citation rate and brand mention rate was one of the findings that stood out to me the most.
7. The Gap Between the Top 5 and Lower Rankings
When I grouped the rankings into buckets, another interesting pattern emerged.
The brand mention rates were:
- Positions 1–5: around 13%
- Positions 6–10: around 6%
- Positions 11–15: around 3%
The citation rates were:
- Positions 1–5: around 38%
- Positions 6–10: around 11%
- Positions 11–15: around 3%
What stood out to me most was the gap between the Top 5 and lower-ranking positions.
The difference was especially noticeable for citations. The citation rate was around 38% for positions 1–5, but dropped to roughly 11% for positions 6–10.
Of course, based on this dataset alone, I can’t say that ranking in the Top 5 will cause a brand to appear in AI Overviews.
Still, the results suggest that, when AI Overview visibility is taken into account, reaching the Top 5 may be significantly more valuable than simply thinking, “As long as we’re in the Top 10, that’s good enough for SEO.”
One of the questions that originally motivated this research was: Should our SEO KPI be Top 3, Top 5, or Top 10?
So for me, this was one of the most useful findings from the study.
8. AI Overviews Seem to Rely Heavily on Pages That Rank Highly in Google Search
Looking at the citation rates, I found a fairly strong relationship between Google organic rankings and AI Overview citations, at least within the dataset I analyzed.
AI search and SEO are sometimes discussed as if they are completely separate disciplines. But when it comes to Google AI Overviews, I’m starting to think we may not need to treat them as entirely separate.
AI Overviews draw on information from across the web when generating answers.
In that sense, it seems reasonable that pages already performing well in Google Search may also be more likely to be used as sources in AI-generated answers.
There is an important caveat, though: what I observed here is correlation, not causation.
This study does not prove that improving a page’s organic ranking will directly increase its likelihood of being cited in an AI Overview.
It is also possible that both organic rankings and AI Overview citations are influenced by some of the same underlying factors, such as:
- Providing comprehensive and useful information
- Being highly relevant to the topic
- Being well established and recognized across the web
So the takeaway is not:
“Improve your rankings, and your citation rate will automatically increase.”
Rather, the results suggest that organic search visibility and AI Overview citations are closely related and may, at least in part, be influenced by the same underlying signals.
9. So If You Rank #1 in SEO, Is Your GEO Work Done?
This is where it becomes important to separate citations from brand mentions.
When companies invest in GEO, I think what they ultimately want to increase is not just citations, but brand mentions.
For example, imagine a user asks an AI:
“What are some recommended T-shirt brands?”
From a company’s perspective, what matters more is probably not whether the AI says, “I used this page as a source,” but whether the brand itself is recommended in the answer.
And in this study, even brands ranking #1 in Google Search had a brand mention rate of only around 20%.
So ranking highly in organic search alone does not appear to be enough.
What else might matter?
From this point on, I’m going beyond what this dataset can prove, so the following is my own hypothesis.
When AI systems answer questions about brands, they are likely drawing on information from across the web, not just from a brand’s own website.
If that is the case, the important question may not simply be:
“Does our own website rank highly?”
but also:
“Is our brand being recommended across many different places on the web?”
For an e-commerce brand, that could include:
- Comparison and review sites
- Media articles
- YouTube
- Social media
- Influencers
- Affiliate websites
- Marketplaces such as Amazon and Rakuten
- User reviews
For example, if a brand is consistently recommended on comparison or review sites that also rank well in Google Search, such as Kakaku.com or mybest, that could potentially help AI systems recognize and recommend the brand as well.
My current hypothesis is that the brands that perform well in AI search will not simply be the ones with strong SEO.
Instead, they may be the brands with strong visibility across the web as a whole: ranking well in search, being discussed on social media, appearing in YouTube videos, being featured on comparison sites, and accumulating a large number of reviews.
In other words, broader digital visibility may ultimately put brands in a stronger position in AI search.
This is one of the areas I’d like to investigate next.
10. Final Thoughts on This Research
For this research, I combined a range of tools, including Semrush, Python, LangChain, LLM APIs, and SearchApi.
I also used Google Antigravity and Codex to support the coding process.
For tasks such as classifying large numbers of keywords, where I didn’t necessarily need the most powerful models, I used lightweight LLM APIs to keep processing costs down.
One thing this project really made me realize is that it has become possible for an individual to conduct research at a surprisingly large scale.
A few years ago, doing all of the following would probably have required a huge amount of time and effort:
- Collecting 100,000 keywords
- Classifying 50,000 of them
- Retrieving 1,000 Google search results
- Analyzing AI Overview responses
- Normalizing brand names
- Aggregating and analyzing the results
Today, by combining Python, APIs, and AI coding tools, one person can handle a surprisingly large portion of that workflow.
There is still a lot we don’t know about GEO and AI search.
So rather than only reading other people’s research, I’d like to keep taking the following approach:
“If something makes me curious, collect the data and test it myself.”
That’s the mindset I want to keep as this field continues to evolve.
11. Summary: Google Rankings Still Seem to Matter a Lot in the AI Overview Era
If I had to summarize the three most important takeaways from this research, they would be:
1. Google rankings and AI Overview visibility appear to be closely related
The relationship was particularly clear for citations: the higher a page ranked in Google Search, the more likely it was to be cited in an AI Overview.
2. The Top 5 seems especially important
There was a clear gap in both brand mention rates and citation rates between positions 1–5 and positions 6–10.
So when setting SEO KPIs with AI search in mind, aiming for the Top 5 may be more meaningful than simply reaching the first page of Google.
3. But even at #1, the brand mention rate was only around 20%
This suggests that GEO probably cannot be solved through SEO alone.
If you want AI systems to actually recommend your brand, it may be necessary to build a presence beyond your own website and create an environment where your brand is recognized, discussed, and recommended across different parts of the web.
Going forward, I’d like to conduct similar research not only on AI Overviews, but also on Google AI Mode, ChatGPT, Gemini, and other AI search platforms.
I’m also interested in exploring how visibility on YouTube, social media, review sites, and other platforms relates to brand mentions in AI-generated answers.
If you’re interested in the full methodology, charts, and data behind this study, I’ve published a more detailed version of the research on my website, heysho.com:
https://heysho.com/en/geo/research-aio-organic-ranking.html
About the Author(Heysho)
I currently work in SEO and GEO / AI search.
I’ve been using Python for around three to four years, and I also spent about five years working as a frontend engineer. I enjoy using programming for marketing analysis, research, and automation.
There is still a lot to learn about GEO and AI search, so I plan to keep exploring the field by collecting real data and testing things for myself.
If I come across any interesting findings, I’ll share them again.
Top comments (2)
The correlation between AI-generated content and keyword rankings highlights the need for adaptive SEO strategies. Understanding how AI impacts search visibility can inform optimization efforts and content creation processes, ensuring relevance in evolving search landscapes.
Thanks for your comment! I agree, as AI search evolves, SEO strategies will need to evolve as well. So I guess SEO jobs are not getting replaced by AI