Add an AI-visibility regression test to your CI: fail the build when ChatGPT stops recommending you

Disclosure: I build the AI Search Visibility Tracker Actor used here on Apify. The output below is a real run (October 22, 2026, US, English, HubSpot as the tracked brand, Salesforce and Pipedrive as competitors). The test was run for real against those saved rows with pytest; I did not make new calls for this post.

You test your code on every commit. You probably do not test the thing that now sends a growing share of buyers your way: whether ChatGPT and Perplexity still name your product when someone asks for a recommendation.

This post adds a small pytest file that fails when your brand's mention rate or your own-domain citation rate falls under a threshold. It is meant to run on a schedule (weekly is plenty), not on every push.

What the tracker returns

Each run gives you one check row per prompt per engine, plus summary rows with the rates already computed. The summary row for the tracked brand across both engines looked like this (real values):

{
  "type": "summary", "engine": "all", "brand": "HubSpot", "is_your_brand": true,
  "checks": 2, "answers": 2, "mentions": 2, "mention_rate": 1,
  "share_of_voice": 0.3333, "avg_position": 1, "top_position_rate": 1,
  "citations": 2, "citation_rate": 1,
  "sentiment_positive": 2, "sentiment_negative": 0
}

and the competitor rows have the same shape (is_your_brand: false), so you can also assert that a competitor is not suddenly first. The per-answer check rows carry brand_mentioned, brand_position, competitors_mentioned, your_cited_urls and cited_domains.

The test

"""AI-visibility regression test: fails when AI assistants stop recommending your brand.
Live mode:   APIFY_TOKEN=... pytest -q test_ai_visibility.py      (costs about $0.04 per prompt per engine)
Offline mode: AI_VIS_FIXTURE=tracker_rows.json pytest -q test_ai_visibility.py   (replays saved rows, free)"""
import json, os
import pytest
import requests

BRAND, DOMAIN = "HubSpot", "hubspot.com"
COMPETITORS = ["Salesforce | salesforce.com", "Pipedrive | pipedrive.com"]
PROMPTS = ["What is the best CRM for a small business?"]
ENGINES = ["chatgpt", "perplexity"]
MIN_MENTION_RATE = 0.5      # share of answers that must name the brand
MIN_CITATION_RATE = 0.5     # share of answers that must cite your domain

@pytest.fixture(scope="session")
def rows():
    if os.environ.get("AI_VIS_FIXTURE"):
        return json.load(open(os.environ["AI_VIS_FIXTURE"]))
    r = requests.post(
        "https://api.apify.com/v2/acts/wrenfield-data~ai-search-visibility-tracker/run-sync-get-dataset-items",
        params={"maxTotalChargeUsd": 2.00},     # hard cap: the test can never spend more than $2
        headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
        json={"brands": [BRAND], "domains": [DOMAIN], "competitors": COMPETITORS,
              "prompts": PROMPTS, "engines": ENGINES}, timeout=600)
    r.raise_for_status()
    return r.json()

@pytest.fixture(scope="session")
def mine(rows):
    s = [x for x in rows if x.get("type") == "summary" and x.get("engine") == "all" and x.get("is_your_brand")]
    assert s, "no summary row for your brand: the run failed or returned no answers"
    return s[0]

def test_answers_came_back(rows):
    ok = [x for x in rows if x.get("type") == "check" and x.get("status") == "ok"]
    assert len(ok) >= len(PROMPTS) * len(ENGINES) / 2, "more than half of the checks failed; do not trust the result"

def test_brand_is_named(mine):
    assert mine["mention_rate"] >= MIN_MENTION_RATE, f"{BRAND} named in only {mine['mention_rate']:.0%} of answers"

def test_own_domain_is_cited(mine):
    assert mine["citation_rate"] >= MIN_CITATION_RATE, f"{DOMAIN} cited in only {mine['citation_rate']:.0%} of answers"

Two modes: live (calls the Actor, capped at $2 per run by maxTotalChargeUsd) and offline (replays saved rows for free, which is how you develop the test and how I ran it for this post).

Real output of the offline run:

test_ai_visibility.py::test_answers_came_back PASSED                     [ 33%]
test_ai_visibility.py::test_brand_is_named PASSED                        [ 66%]
test_ai_visibility.py::test_own_domain_is_cited PASSED                   [100%]
============================== 3 passed in 0.07s ===============================

Making it a scheduled job

A GitHub Actions workflow on a weekly cron is enough:

on:
  schedule: [{cron: "0 14 * * 1"}]   # Mondays
jobs:
  ai-visibility:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pytest requests
      - run: pytest -q test_ai_visibility.py
        env:
          APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }}

(I have not run this workflow file; the pytest file above is what I executed.)

Things that will bite you

• Answers are not deterministic. The same prompt gets a different answer next run. One prompt on two engines, as in my example, is a demo; for a real gate use 10 or more prompts across the engines you care about and set the threshold from a few baseline runs, not from a guess.
• Cost. The tracker is $0.04 per prompt per engine at the time of writing (scheduled to become $0.06 on November 2, 2026). Ten prompts on two engines is 20 checks: $0.80 today, $1.20 after the change. Weekly, that is about $3.50 to $5.20 a month, and maxTotalChargeUsd makes the worst case explicit.
• Do not fail the build on one bad week. Alert on two consecutive failures, or on a drop against the previous run, not on an absolute number alone.
• Synchronous calls stop at 5 minutes. Twenty checks finish well inside that; for hundreds of prompts, start the run asynchronously and poll it instead.
• Different from rank tracking. This measures whether assistants name you, not where a page ranks in Google.

Links

• Tool pages, real output rows and MCP setup: https://wrenfielddata.com/search-data-for-ai-agents/
• AI Search Visibility Tracker and AI Answer Scraper on Apify

If you want a field added to the output, say so in the comments.