The monthly agency deck lands, forty keywords are green, and somebody in the room asks the question the deck cannot answer: when a buyer asks ChatGPT or Gemini who does this work in Sydney, do we come up? Ranking well in the ten blue links and being absent from AI answers are now two different states, and a brand can sit in one while assuming the other (RadarKit vs Peec AI vs Profound). That gap is the reason to run your own telemetry. This post is part of our Practical AI in Marketing series, and it covers the smallest thing that works: a scheduled job, a table you own, and a weekly report.
Record presence separately from position
The main design decision is in the schema. Position and AI-surface presence are different measurements and collapsing them into one "visibility score" throws away the signal you actually want.
create table serp_check (
checked_at timestamptz not null,
keyword text not null,
locale text not null, -- gl=au, hl=en-AU, Sydney or Melbourne
engine text not null, -- google, google_ai_mode, chatgpt, perplexity
position int, -- null when absent, never 0
ai_present boolean, -- was an AI answer rendered at all?
cited boolean, -- were we named or linked in it?
competitors text[], -- who was named instead
raw jsonb not null -- the full response, kept forever
);
Two details matter. Use null for absence rather than 0 or 101, because averages over a sentinel value quietly lie to you. And keep the raw payload. The tool market is churning hard enough that August 2026 alone produced round-ups of 17 AI rank trackers plus separate lists of LLM visibility tools and vendor alternatives. Any of those products may not exist in two years. Your history should live in your database, not theirs.
Set the locale properly. Australian result sets differ from US ones, and a national keyword checked from a US data centre will tell you about a market you do not sell into.
The job itself
The architecture is deliberately boring:
- A keyword and prompt list in a config file in the repo, reviewed by whoever owns the site.
- A scheduled runner: cron on a small VM, an Azure Function on a timer, or a GitHub Actions workflow if you want the config and the history in one place.
- A SERP API call per keyword for classic Google, returning positions and any AI Overview block.
- Optional web-grounded calls to the LLM assistants your buyers actually use, one row per prompt per engine.
- An append-only write to Postgres or SQLite, plus a generated weekly markdown or HTML report.
None of this is novel work. A public example repository posted in August 2026 does the whole loop with a prompt bank, four optional engines, JSONL history and a weekly workflow, and Lettertrace ships an MIT-licensed bring-your-own-key version where your data sits in your own Supabase. The tracking layer is commoditised. Fork something, or write it in a day.
What it costs
Per-query pricing is the easy part. SERP APIs bill per successful query in the fractions-of-a-cent range at the volumes a single company needs, so the arithmetic looks like this: 60 keywords checked daily is roughly 1,800 queries a month, which at half a cent a query is under $10 AUD. Add three AI surfaces on a weekly cadence and you are still in tens of dollars, not hundreds. Grounded LLM calls cost more per query than raw SERP fetches, so run those weekly rather than daily.
Verify tier pricing against official sites before you budget. One August 2026 comparison listed Ahrefs at $199 per month per index and was corrected in its own comments to a $50 starting price, which the author acknowledged (9 best LLM visibility tracking tools). Round-up pricing is unreliable in both directions.
The real cost is engineering time: a day or two to build, then an hour or two a month keeping it alive. Budget that hour honestly, because it is the line item people forget.
Rate limits and failure handling
Providers cap concurrency well below what a naive loop will attempt. Practical defaults we use:
- Queue the keywords and run them with bounded concurrency, not a parallel map over the whole list.
- Retry on 429 and 5xx with exponential backoff and a hard attempt limit.
- Add a few minutes of random jitter to the scheduled start so you are not hammering the API at the same second as everyone else's cron.
- Write a row for every failure with a status field. A missing row and a genuine absence must never look the same in the data.
That last point catches the worst failure mode. If your job dies halfway through Tuesday and you silently store nothing, next week's chart shows a ranking collapse that never happened.
Parsers break, and that is normal
The AI surfaces change shape month to month. On 28 August 2026 Google AI Mode added flight price tracking, points and mile rates, and hotel booking. Layout changes of that kind break whatever selector or heuristic you use to decide "AI Overview present?".
Two cheap defences. First, a canary: pick two or three keywords that reliably trigger an AI answer and alert if they all report absent on the same day, because a global drop is a parser bug far more often than a ranking event. Second, put a recurring maintenance task in the backlog rather than waiting for the dashboard to look wrong.
Reading the trend
Daily numbers jitter for reasons that have nothing to do with your site: test buckets, location, index refreshes, and in the case of LLM answers, plain non-determinism between runs. Rules that keep people calm:
- Report a seven-day rolling median position, not yesterday's number.
- Treat AI presence as a rate, not an event: "cited in 9 of 28 checks this month" is a number you can act on.
- Only investigate a move that holds for four or more consecutive checks.
- Alert on the competitor field, not just your own. A rival appearing in six answers where you appear in none is the signal worth a meeting.
Where DIY wins, and what an agency still adds
There are two jobs here and only one of them is a script. Counting how often you appear is a dashboard. Working out why you are missing from AI answers across Google AI Overviews, ChatGPT Search, Perplexity, Copilot and Gemini is diagnosis, and it is where an agency or consultant still earns the fee (AI GEO strategist thread). The method is not the moat either: search engines document their own systems and large agencies publish their manuals for free (how to learn SEO and AI search for free), so what you are buying is judgement and follow-through, not secret knowledge.
Judge your internal dashboard against the three questions agencies are being asked to answer in 2026: does the brand appear in AI-generated answers, which competitors are recommended instead, and what would change that (AI visibility trackers for agencies). A script answers the first two well and the third not at all. If you want a template for splitting the work formally, an August 2026 engineering ticket ran a gap analysis of an in-house visibility surface against a commercial product and produced an explicit build list. That is the right shape of decision: build the collection and the history, licence or outsource the analysis.
Honest limitations
API results are not what a logged-in customer in Parramatta sees on their phone. LLM answers vary between identical runs, so any single check is close to meaningless and only the rate over weeks means anything. Coverage of AI surfaces varies by provider and lags each layout change. You own the maintenance, and telemetry does not connect to revenue on its own. What it does buy you is a defensible measurement you control, at a cost that rounds to nothing, so the conversation with your agency starts from shared numbers instead of their slide deck.
PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.
Originally published at picnet.com.au.
Top comments (1)
Love the null vs 0 point and treating AI visibility as its own metric. One thing I’d add: tag each keyword with intent + funnel stage in your config. Then roll up weekly “AI citation rate by intent” so you know which journeys to fix, not just that you’re absent.