DEV Community

Nilesh Jethwa
Nilesh Jethwa

Posted on Originally published at crawlspider.com

I Built a Simple AI Visibility Tracker in Python. Here’s What Breaks When You Scale It

#webdev #python #ai #architecture

I Built a Simple AI Visibility Tracker in Python. Here’s What Breaks When You Scale It


AI visibility tracking sounds like a fairly simple programming problem.

Ask ChatGPT a question.

Check whether a brand appears in the answer.

Save the result.

Repeat tomorrow.

And honestly, at first, it is that simple.

You can build a primitive AI visibility tracker in a few lines of Python.

from openai import OpenAI
from datetime import datetime

client = OpenAI()

brand = "Acme"

prompts = [
    "What are the best project management tools?",
    "What are good alternatives to Trello?",
    "What project management software is best for small businesses?",
    "Which project management tools have AI features?",
    "What tools can remote teams use to organize projects?"
]

results = []

for prompt in prompts:

    response = client.responses.create(
        model="gpt-5.4-mini",
        input=prompt
    )

    answer = response.output_text

    results.append({
        "prompt": prompt,
        "mentioned": brand.lower() in answer.lower(),
        "response": answer,
        "checked_at": datetime.utcnow().isoformat()
    })

visibility = (
    sum(r["mentioned"] for r in results)
    / len(results)
) * 100

print(f"{brand} visibility: {visibility:.1f}%")
Enter fullscreen mode Exit fullscreen mode

If Acme appears in two of five responses, we could call that 40% visibility.

Done.

Well... not quite.

I recently worked through this problem while building the AI visibility tracking system behind CrawlSpider, and the interesting part wasn't making the LLM API call.

It was everything that happened after that.

The innocent-looking nested loop

Conceptually, an AI visibility tracker looks something like this:

for each brand:
    for each prompt:
        for each model:
            ask the model
            save the response
            find the brand
            find competitors
            calculate metrics
Enter fullscreen mode Exit fullscreen mode

That looks harmless.

But consider:

100 brands
× 50 prompts
× 3 models
× daily scans
Enter fullscreen mode Exit fullscreen mode

That's 15,000 requests every day.

Or 450,000 model responses every month.

Move to 1,000 brands and you're dealing with millions.

And suddenly this:

for prompt in prompts:
    call_llm(prompt)
Enter fullscreen mode Exit fullscreen mode

isn't really the architecture anymore.

You need a queue

The first thing that breaks is the simple loop.

What happens if request #8,742 fails?

What happens when an API starts returning rate-limit errors?

What if one provider slows down?

What if your worker crashes halfway through a batch?

You don't want to restart everything.

So the architecture starts becoming something like:

Scheduler
    ↓
Scan Generator
    ↓
Job Queue
    ↓
Worker Pool
    ↓
LLM Provider
    ↓
Response Store
Enter fullscreen mode Exit fullscreen mode

Now you need:

  • retries
  • exponential backoff
  • concurrency controls
  • job states
  • idempotency
  • dead-letter handling
  • rate-limit management
  • monitoring

We've moved surprisingly far away from our original Python script.

Then brand in response breaks

Our prototype has another wonderfully naive line:

brand.lower() in answer.lower()
Enter fullscreen mode Exit fullscreen mode

Try that with:

brand = "Apple"
Enter fullscreen mode Exit fullscreen mode

Did the model mention Apple Inc.?

Or an apple?

What about abbreviations?

Product names?

Parent companies?

And simply knowing that a brand appeared isn't particularly interesting.

Suppose the response says:

For enterprise teams I'd consider Acme or Monday.com,
while smaller teams might prefer Trello.
Enter fullscreen mode Exit fullscreen mode

Now I probably want something closer to:

{
  "target_brand": {
    "mentioned": true,
    "position": 1
  },
  "competitors": [
    {"name": "Monday.com", "position": 2},
    {"name": "Trello", "position": 3}
  ]
}
Enter fullscreen mode Exit fullscreen mode

The tracker has quietly turned into an entity extraction and classification system too.

The real product is history

Here's another realization I had while working on this.

A single AI response isn't particularly valuable.

Change is valuable.

Imagine seeing this:

"Best project management software for small businesses?"

Week 1    Acme not mentioned
Week 2    Acme #5
Week 3    Acme #3
Week 4    Acme #2
Enter fullscreen mode Exit fullscreen mode

That's interesting.

But now every observation potentially needs:

brand_id
prompt_id
model_id
model_version
timestamp
raw_response
brand_mentioned
brand_position
competitors
sentiment
citations
token_usage
latency
status
Enter fullscreen mode Exit fullscreen mode

Multiply that by millions of responses.

You're not storing API results anymore.

You're building a historical analytics dataset.

Multiple models make things more interesting

Then you decide that monitoring one AI model isn't enough.

Maybe you want:

providers = [
    "openai",
    "anthropic",
    "google"
]
Enter fullscreen mode Exit fullscreen mode

Each has different APIs, response structures, rate limits, model identifiers, errors, citations and pricing.

Eventually you want an abstraction like:

                 ┌── OpenAI Adapter
Prompt Engine ───┼── Anthropic Adapter
                 └── Google Adapter
                         ↓
                 Normalized Response
Enter fullscreen mode Exit fullscreen mode

Otherwise provider-specific logic ends up everywhere.

Scheduling becomes a system of its own

Then users ask for:

Prompt A → Daily
Prompt B → Weekly
Prompt C → Daily
Prompt D → Manual
Enter fullscreen mode Exit fullscreen mode

Now something has to determine what is due.

And prevent duplicate runs.

And recover failed jobs.

And calculate the next run.

And make sure one huge account doesn't consume the entire worker pool.

At this point the "AI visibility tracker" is really a distributed job-processing and analytics application that happens to call LLMs.

AI responses aren't deterministic either

There's another subtle problem.

Run:

What are the best tools for X?
Enter fullscreen mode Exit fullscreen mode

today and your brand might appear.

Run the exact same prompt tomorrow and it might not.

That doesn't necessarily mean the brand suddenly became less visible.

LLM responses vary.

So when a dashboard says:

Visibility

Last week: 42%
This week: 38%
Enter fullscreen mode Exit fullscreen mode

what does that actually mean?

Is something changing?

Or are we observing normal model variation?

This makes prompt consistency, sample size, model versions and historical comparison surprisingly important.

API cost isn't the only scaling problem

It's natural to focus on token costs.

Those certainly matter when you're running hundreds of thousands or millions of requests.

But I found the more interesting cost to be engineering complexity.

At scale you're paying for much more than inference:

LLM inference
+ queues
+ workers
+ databases
+ storage
+ scheduling
+ retries
+ observability
+ analytics
+ provider maintenance
+ engineering time
Enter fullscreen mode Exit fullscreen mode

And every new dimension multiplies the workload:

brands
× prompts
× models
× scan frequency
× time
Enter fullscreen mode Exit fullscreen mode

That's the equation I'd pay attention to when designing one of these systems.

We had a useful head start

One reason we were able to build this into CrawlSpider is that we weren't starting completely from zero.

I'd previously built pieces of this kind of infrastructure for other projects.

InfoCaptor had given us experience with analytics, visualization and AI-driven workflows.

CrawlSpider's existing internal-linking system already dealt with crawling, page analysis, background processing and large collections of URLs.

Other projects had already forced us to solve problems around scheduled jobs, APIs, queues and asynchronous processing.

The AI visibility tracker became less about inventing every component and more about assembling those existing patterns around a new workflow:

Brand
  ↓
Prompts
  ↓
Models
  ↓
Scheduled scans
  ↓
Responses
  ↓
Mentions + competitors
  ↓
Historical metrics
  ↓
Dashboard
Enter fullscreen mode Exit fullscreen mode

That reuse turned out to be extremely valuable.

The interesting lesson

Could you build an AI visibility tracker yourself?

Absolutely.

In fact, I think building the five-prompt Python version is a great weekend project.

The core algorithm can fit on one screen.

But that's also what makes this problem interesting.

There is a huge gap between:

"Call an LLM and see if my brand appears."
Enter fullscreen mode Exit fullscreen mode

and:

"Reliably monitor thousands of brands across
multiple models every day and explain how their
visibility is changing."
Enter fullscreen mode Exit fullscreen mode

The first is an API call.

The second is a platform.

I wrote a much deeper breakdown of the architecture, scaling math, infrastructure and costs while documenting how we approached this at CrawlSpider:

How to Build an AI Visibility Tracker From Scratch

If you're building something similar, I'd be interested in hearing how you're approaching the scheduling, normalization and non-determinism problems.

PS:
I also built a AI Adoption visualization Dashboard , check out!

I also maintain LLM cutoff dates for major providers
https://www.crawlspider.com/llm-knowledge-cutoff-dates/

Lastly there are 50+ brands monitored for their AI Visibility

Top comments (0)