DEV Community: Noraina Nordin

Build an AI Overview Rank Tracker using Python

Noraina Nordin — Wed, 04 Feb 2026 01:46:22 +0000

Search engines are evolving rapidly. Instead of presenting only a list of websites, Google now frequently displays AI Overviews. It is an AI-generated summary that combines information from multiple sources and presents answers directly at the top of the search results page.

As a result, visibility is no longer defined solely by rankings. This shift has introduced a new concept known as Generative Engine Optimization (GEO).

In this tutorial, we'll walk through how to build a simple AI Overview Rank Tracker using Python.

What Is an AI Overview Rank Tracker?

An AI Overview Rank Tracker is a tool that answers a new kind of visibility question:

"Is my website cited inside Google's AI Overview, and how prominent is that citation?"

Traditional rank trackers focus on organic positions (1–10).

An AI Overview Rank Tracker focuses on:

Whether an AI Overview appears for a query
Which sources does the AI references
The order in which those sources appear
Whether your brand or competitors are included

This is a core component of Generative Engine Optimization (GEO).

Why AI Overviews Change How Ranking Works

AI Overviews often appear above organic results and summarize information directly on the search results page. In many cases, users get their answers without clicking any links.

This introduces several challenges:

Traditional rankings become less visible
Click-through rates are harder to attribute
Competition shifts toward citations, not just rankings
Existing SEO tools offer little insight into AI-generated answers

As AI Overviews become more common, tracking AI citation presence becomes essential.

Read more details explanation about Rank Tracking in the Age of AI Overviews from our previous post below:

Rank Tracking in the Age of AI Overviews: What's Changed

Why Traditional Scraping Is Not Enough

Tracking AI Overviews via browser automation or raw HTML scraping is unreliable due to:

JavaScript-rendered content
Frequent layout changes
Regional and language variations
Bot detection and captchas
Fragile DOM-based selectors

These issues make traditional scraping brittle and difficult to maintain at scale.

Using SerpApi to Build an AI Overview Rank Tracker

SerpApi provides real-time access to structured search engine data, including AI-generated elements where available. It handles:

IP rotation and request routing
JavaScript rendering
Layout changes across regions
High-volume request reliability

By using a Web Search API, developers can focus on analysis and insights rather than infrastructure maintenance.

Check out the tutorial below on how to get results from AI Overviews using our API:

How to Scrape Google AI Overviews (AIO)

Prefer watching instead of reading? You can also follow along with the video walkthrough below:

Building an AI Overview Rank Tracker using Python

Below is a step-by-step implementation that:

Detects AI Overviews
Fetches AI Overview details
Extracts cited sources
Prints title and link only
Determines whether a specific website is cited
Reports the site’s rank inside the AI Overview

Step 1: Install dependencies

pip install google-search-results python-dotenv

Step 2: Run a Google search and detect AI Overview presence

AI Overview does not appear for every query.

The first step is to detect whether one exists and retrieve its page_token.

from serpapi import GoogleSearch
import os

SERPAPI_API_KEY = os.getenv("SERPAPI_API_KEY")

def detect_ai_overview(query):
    params = {
        "engine": "google",
        "q": query,
        "hl": "en",
        "gl": "us",
        "api_key": SERPAPI_API_KEY
    }

    search = GoogleSearch(params)
    results = search.get_dict()

    ai_overview = results.get("ai_overview")
    if not ai_overview:
        return None

    return ai_overview.get("page_token")

The query here refers to target keyword or search phrase you want to monitor for AI Overview visibility.

If no page_token is returned, the query does not trigger an AI Overview.

Step 3: Fetch the AI Overview result page

Once a page token is available, use the google_ai_overview engine to retrieve full AI Overview data.

def fetch_ai_overview(page_token):
    params = {
        "engine": "google_ai_overview",
        "page_token": page_token,
        "api_key": SERPAPI_API_KEY
    }

    search = GoogleSearch(params)
    return search.get_dict()

This response includes structured AI content and cited references.

Note: Since this is another API endpoint, it uses another one search credit.

Step 4: Extract citations and calculate rank

In AI Overviews, ranking is determined by the order in which references are returned by the API.

from urllib.parse import urlparse

def normalize_domain(url):
    return urlparse(url).netloc.replace("www.", "")

def analyze_ai_overview(ai_results, target_url):
    references = ai_results.get("ai_overview", {}).get("references", [])
    target_domain = normalize_domain(target_url)

    found_rank = None

    print("\nAI Overview References:\n")

    for rank, ref in enumerate(references, start=1):
        title = ref.get("title")
        link = ref.get("link")

        print(f"{rank}. {title}")
        print(f"   {link}\n")

        if target_domain in normalize_domain(link):
            found_rank = rank

    if found_rank:
        print(f"Your site appears in the AI Overview at position #{found_rank}")
    else:
        print("Your site is not referenced in the AI Overview")

Step 5: Run the full AI Overview Rank Tracker

if __name__ == "__main__":
    query = input("Enter target keyword to track: ").strip()
    website = input("Enter your website URL: ").strip()

    page_token = detect_ai_overview(query)

    if not page_token:
        print("No AI Overview detected for this query.")
    else:
        ai_results = fetch_ai_overview(page_token)
        analyze_ai_overview(ai_results, website)

Result

Full code is available in our GitHub serpapi/tutorials repository.

Conclusion

AI Overviews are redefining search visibility. As generative search experiences continue to expand, being cited by AI systems becomes as important as ranking organically.

An AI Overview Rank Tracker allows developers and SEO teams to measure this new form of visibility using structured, reliable data. By leveraging Google Search API and Google AI Overview API, we can move beyond fragile scraping and build scalable GEO monitoring systems for the future of search.

How AI Can Predict the Success of Your Business Using Data from Google Maps

Noraina Nordin — Wed, 17 Sep 2025 10:59:22 +0000

Introduction

Coffee shops are booming in my area. Every month, I hear about a new one opening, which makes me wonder:

Will these business last long? Or are they just part of a seasonal hype?

This curiosity inspired me to create an AI agent that can explore the coffee shop market and predict its potential success. While I'm focusing on coffee shops in this project, the same framework can be applied to other businesses, such as restaurants, salons, gyms, and retail stores. However, because LLMs are limited by their knowledge cutoff dates, having access to real-time data is crucial for obtaining the most up-to-date insights.

Traditionally, entrepreneurs relied on guesswork, small surveys, or costly market research. Now, thanks to tools like Google Maps and Google Reviews, we can access real-time information to guide better decisions.

In this project, we will use these two API solutions from SerpApi to help us gather data easily:

-Google Maps API: To scrape business listings and locations.
-Google Maps Reviews API: To extract detailed customer reviews and ratings.

Check out the Scrape Google Maps data and reviews using Python, a comprehensive Getting Started Tutorial.

Please keep in mind that this AI agent is intended for educational purposes only. While it provides useful insights, predictions may not always be entirely accurate. To get the best results, it's a good idea to combine AI insights with your own careful market research.

How the AI Agent Works

Search for coffee shops using SerpApi's Google Maps API
1. Input either a location (e.g., "Austin, TX") or GPS coordinates.
2. Scrape all the local coffee shops with metadata.
Pull reviews with SerpApi's Google Maps Reviews API
1. Extract customer opinions, ratings, and timestamps for each competitor.
AI-Powered Analysis
1. Summarize competitor density and pricing tiers.
2. Perform sentiment analysis on customer reviews.
3. Predict the success probability of a new coffee shop in the area.
4. Output structured tables and an action plan in a clean Markdown file.

Using SerpApi to scrape data from Google Maps

SerpApi's Google Maps API allows us to gather organized data from Google Maps quickly and easily, without the hassle of CAPTCHA, proxies, or manual scraping stress.

With just one API request, you can access a wealth of information. Feel free to explore our Google Maps API playground to see what data you can retrieve. Below, there's a sample screenshot from our playground to give you an idea.

However, this is the data we need:

title
data_id (We need this information to be able to retrieve the reviews later)
gps_coordinates
address
rating
reviews
price
operating_hours

All data will be passed to the AI model for analysis, except for the data_id, which is required to scrape Google Maps Reviews.

Beyond ratings, reviews tell the real story. Using SerpApi's Google Maps Reviews API, we can pull recent customer feedback for each competitor. You can visit our Google Maps Reviews API playground to explore the additional information we can collect.

The data that we're interested in collecting:

snippet which is the customer's review
rating
date

Set up the scraper

Prerequisites

Python 3.9+
Install dependencies:
- pip install google-search-results openai python-dotenv requests
SerpApi API keys (sign up here to get 250 free searches/month).
OpenAI API key or download the gpt-oss model locally via ollama.
Create a .env file with your API keys.

SERPAPI_API_KEY=your_serpapi_key_here
OPENAI_API_KEY=your_openai_key_here # If you're using OpenAI

Import the necessary libraries

Before we begin, let's import the necessary libraries for making API calls, logging, managing environment variables, and handling dates and times.

import os
import json
import time
import logging
from datetime import datetime
from dotenv import load_dotenv
from serpapi import GoogleSearch
from openai import OpenAI
import requests

For our convenience, we will create a log to record debug information. It will be saved in the debug.log file.

logger = logging.getLogger("coffee_agent")
logger.setLevel(logging.DEBUG)
file_handler = logging.FileHandler('debug.log')
file_handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S'))
logger.addHandler(file_handler)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S'))
logger.addHandler(stream_handler)

Scrape data from Google Maps

The first step is to query Google Maps for coffee shops around a given location or coordinates.

We define a helper function to build the search parameters:

BUSINESS_TYPE = "Coffee Shop"

def get_search_params(user_input, api_key):
    if "," in user_input and all(part.replace('.', '', 1).replace('-', '', 1).isdigit() for part in user_input.split(',')):
        logger.info(f"Fetching {BUSINESS_TYPE.lower()} near coordinates: {user_input}")
        return {
            "engine": "google_maps",
            "q": BUSINESS_TYPE,
            "ll": f"@{user_input},14z",
            "api_key": api_key
        }
    else:
        logger.info(f"Fetching {BUSINESS_TYPE.lower()} in location: {user_input}")
        return {
            "engine": "google_maps",
            "q": f"{BUSINESS_TYPE} in {user_input}",
            "api_key": api_key
        }

This function builds the right parameters depending on whether the user types a city name or latitude/longitude coordinates.

You can change the value for BUSINESS_TYPE based on your business of interest.

Now let's use it to scrape nearby coffee shops:

def fetch_shops_details(search_params):
    logger.info(f"Sending request to SerpApi for {BUSINESS_TYPE.lower()} search...")
    response = requests.get("https://serpapi.com/search", params=search_params)
    data = response.json()
    logger.debug(f"Received response: {json.dumps(data)
[:500]}")
    logger.info(f"Received response from SerpAPI.")
    local_results = data.get("local_results", [])
    logger.info(f"Found {len(local_results)} {BUSINESS_TYPE.lower()}s around the area.")
    return local_results

This part of the script sends a request to SerpApi's Google Maps API to retrieve local business results based on the area specified by the user's input.

Visit our Google Maps API documentation for more information.

Scrape data from Google Maps Reviews

Each coffee shop result has a unique data_id, which we get from the previous API call. We can use this data_id to retrieve customer reviews.

def fetch_reviews(data_id, shop_title):
    logger.info(f"Fetching reviews for: {data_id} ({shop_title})")
    review_params = {
        "api_key": os.getenv("SERPAPI_API_KEY"),
        "engine": "google_maps_reviews",
        "data_id": data_id,
        "hl": "en"
    }
    review_search = GoogleSearch(review_params)
    review_results = review_search.get_dict()
    reviews = review_results.get("reviews", [])
    logger.info(f"Found {len(reviews)} reviews for shop {shop_title}")
    time.sleep(1)
    return [
        {
            "review_text": review.get("snippet"),
            "review_star_rating": review.get("rating"),
            "timestamp": review.get("date")
        }
        for review in reviews
    ]

This script sends a request to SerpApi's Google Maps Reviews API to scrape each customer's reviews and returns the collected reviews to each coffee shop's data.

In our example, 20 coffee shops were found in the area from the previous search, resulting in 20 requests for Google Maps Reviews. Check out your search history here.

Visit our Google Maps Reviews documentation for detailed information.

Build Competitor Dataset

Finally, let's put everything together into a string format for the AI to analyze.

def format_competitor_data(competitors):
    competitor_data_str = ""
    for shop in competitors:
        competitor_data_str += f"\n- {shop['business_name']}\n  - Address: {shop['address']}\n  - GPS: {shop['GPS_coordinates']}\n  - Star Rating: {shop['star_rating']}\n  - Review Count: {shop['review_count']}\n  - Opening Hours: {shop['opening_hours']}\n  - Price Level: {shop['price_level']}\n  - Customer Reviews:\n"
        for review in shop["customer_reviews"]:
            competitor_data_str += f"    - \"{review['review_text']}\" | {review['review_star_rating']} stars | {review['timestamp']}\n"
    return competitor_data_str

Set up an AI agent

Now that we have the competitor data, let's build an AI agent to analyze it.

Design the AI prompt

LLMs perform best when provided with clear, structured instructions. Here's an example prompt.

def build_prompt(competitor_data_str):
    return f"""
                You are **{BUSINESS_TYPE} Success Forecaster**, an AI agent that predicts the potential success of a new {BUSINESS_TYPE.lower()} in a given location.

                You will be fed structured data about competitor {BUSINESS_TYPE.lower()}s, including:
                {competitor_data_str}

                Your tasks are:

                1. **Market Landscape Analysis**
                - Summarize competitor density within 1–3 km.
                - Identify star-rating patterns, price tiers, and operating hours.
                - Highlight underserved niches (pricing gaps, late-night, early-morning).
                - Output:
                    - A **2-column table** with | Metric | Observations |.
                    - A short **bullet list of underserved slots/pricing gaps**.

                2. **Sentiment & Customer Insight**
                - Perform sentiment analysis on reviews.
                - Extract recurring positives, negatives, and unmet needs.
                - Output:

                    | Sentiment Category | Positive Highlights | Negative Highlights | Unmet Needs / Opportunities |
                    |--------------------|---------------------|---------------------|-----------------------------|

                - Conclude with exactly **3 key takeaways** in bullet points.

                3. **Success Prediction**
                - Estimate a **success probability score (0–100%)** in bold.
                - Output:
                    - A **2-column table** with | Metric | Value |.
                    - A bullet list of the **Top 3 drivers of success/failure**.

                4. **Recommendations**
                - Output:
                    - A **2-column table** with | Differentiator | Rationale |.
                    - A **2-column Risks & Mitigations table**.
                    - End with an **Action Plan** that:
                        - Is written as **one concise paragraph (1–2 sentences, max 3 lines)**.
                        - Starts with: **Action plan:** (exact text, bolded).
                        - Specifies **location, operating hours, pricing, or promotions** directly tied to the analysis.
                        - Avoids bullet points, timelines (“next 2 weeks”), or generic advice.

                ⚠️ Rules for consistency:
                - Always be clear, data-driven, and practical.
                - Do not give generic answers; tailor insights directly to the provided data.
                - Use tables where defined, and bullets only where instructed.
                - Keep tone business-strategic, concise, and data-driven.
                - Do not mix styles between sections.
            """

Notice how we use tables, bullet points, and structured outputs to make things clearer and easier to follow. You can experiment with different prompts to see what works best for you.

Run the AI Agent

def main():
    load_dotenv()

    api_key = os.getenv("SERPAPI_API_KEY")
    while True:
        user_input = input("Enter your business location name or coordinates (e.g., 'Austin, TX' or '30.2957009,-98.0626221') [type 'q' or 'quit' to exit]: ").strip()
        if user_input.lower() in ["q", "quit"]:
            print("Exiting program.")
            return
        if not user_input:
            print("[ERROR] Input cannot be empty. Please enter a valid location name or coordinates.")
            continue
        if not isinstance(user_input, str):
            print("[ERROR] Input must be a string.")
            continue
        break

    global now
    now = lambda: datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    search_params = get_search_params(user_input, api_key)
    local_results = fetch_shops_details(search_params)
    competitors = build_competitor_data(local_results)
    competitor_data_str = format_competitor_data(competitors)
    prompt = build_prompt(competitor_data_str)

    client = OpenAI(
        base_url="http://localhost:11434/v1",  # Local Ollama API
        api_key="ollama"                       # Dummy key
    )
    response = client.chat.completions.create(
        model="gpt-oss:20b",
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    ai_output = response.choices[0].message.content
    logger.debug(f"AI response: {ai_output[:500]}")

    # Save AI output to markdown file with timestamp
    md_filename = f"ai_response_{datetime.now().strftime('%Y%m%d_%H%M%S')}.md"
    with open(md_filename, "w", encoding="utf-8") as md_file:
        md_file.write(ai_output)

    print(ai_output)

if __name__ == "__main__":
    main()

When you run this script, the AI will output a market analysis, sentiment breakdown, prediction, score, and action plan in Markdown format.

The full source code is available in our GitHub repository

What's next?

While this prototype focuses on coffee shops, the same approach could easily apply to other businesses as well, such as:

Restaurants
Gym
Salons
Co-working spaces

Some possible improvements that you can make:

Add charts & visualizations (e.g., competitor density heatmaps)
Deploy as a Flask/FastAPI web app so anyone can test it in their city.
Turn into a dashboard tool for entrepreneurs scouting opportunities.

Conclusion

By combining real-time Google Maps data with AI analysis, we can quickly gain insights into competition, customer sentiment, and market gaps.

Of course, no AI can perfectly predict business success. However, tools like this can provide entrepreneurs with a faster and more data-driven way to explore opportunities.

⚠️ *Disclaimer*: This project is for educational and experimental purposes only. Predictions may be inaccurate. Always validate insights with proper market research before making real business decisions.

Note:

If you'd like to explore the Google Maps API and Google Reviews API for other use cases, be sure to check out our in-depth tutorial below

How To Make a Travel Guide Using SERP data and Python
Gauge Business Popularities using Google Maps