Ranking and being cited are two different problems, and Google Search Console only shows you the first one. You can sit at position 4 on a query, watch an AI Overview answer it with somebody else's page, and see nothing in your reports except a CTR that quietly sags. This post is about closing that gap with evidence: the AI Overview Rewrite Queue on Apify joins your Search Console export with live citation data and hands back a scored list of which pages to rewrite first.
Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.
Why citation tracking alone is not enough
Most AI visibility tools answer one question: were you cited. That is useful and incomplete, because "not cited" on its own does not tell you whether to act. Not cited on a query you rank 40th for is noise. Not cited on a query you rank 6th for, where a competitor is cited instead, is a page Google already understands that somebody else is answering better. Telling those apart needs your ranking data and your citation data in the same row, which is the join this Actor does.
What the rewrite queue returns
The API returns one row per query, sorted so the rewrites worth the most come first, with your Search Console metrics and live citation data side by side.
| Field | Example | Notes |
|---|---|---|
query |
what is a crm |
The keyword |
tier |
A |
A to X, biggest opportunity first |
tier_reason |
One plain sentence | Explains the call without a legend |
position / impressions
|
8.4 / 1900
|
From your export |
ai_overview_present |
true |
Whether an overview appeared at all |
citation_state |
competitor_cited |
Cited, competitor cited, or no references |
reference_domains |
["reddit.com", "ibm.com"] |
Every domain the overview cited, in order |
join_status |
matched |
Or gsc_only / check_only, never dropped |
A whole-run summary lands in the key-value store: counts per tier, the join rate, and the competitor domains cited most often across your keyword set.
The five tiers, and what to do with each
A is the money tier: you rank 5 to 20 and a competitor is cited instead of you. Rewrite these first. B is the one people miss, an answer-shape problem rather than an authority problem: you rank top 4 and the AI still passes you over, so restructure the answer instead of chasing links. If citation simply tracked ranking, tier B would always be empty. C means you are cited and still converting below your own baseline, so the overview is satisfying the search. D is ordinary SEO. X means the check did not complete, which is evidence rather than a conclusion.
Tier C compares each query against your own median CTR at a similar position, computed from the no-overview queries in your export. It never uses an industry benchmark table, because every account's normal is different.
Who this is for
SEO and content people who want a rewrite backlog built from evidence rather than opinion. Anyone tracking AI visibility over time who has noticed that a single check is noise, since the same query can cite you Monday and drop you Thursday. And builders wiring an agent that can pull the tier A list and draft the rewrites in the same session.
The manual way, and where it breaks
You can do this by hand: export Search Console, search each query, note who got cited, paste it into a sheet. It works for twenty queries and falls apart at two hundred. Worse, it quietly produces bad data, because an overview that failed to load looks identical to a query with no overview and both get written down as a zero. That is the failure mode that matters in measurement tools, where a plausible zero reads as an answer instead of a bug, and you end up confidently deprioritizing a page you should have rewritten.
The faster way: run the queue
Open the AI Overview Rewrite Queue, put your domain in target_domains, and give it your Search Console export as a published-Sheet CSV URL or as pasted rows. No Google sign-in, no OAuth, no service account: it reads an export you already have. From the command line:
curl -X POST "https://api.apify.com/v2/acts/johnvc~ai-overview-rewrite-queue/runs?token=YOUR_APIFY_TOKEN" -H "Content-Type: application/json" -d '{ "target_domains": ["example.com"], "search_console_csv_url": "https://docs.google.com/spreadsheets/d/e/EXAMPLE/pub?output=csv", "min_impressions": 50 }'
Run mechanics are in the Apify API docs.
To get the export: open the Performance report, pick a date range, open the Queries tab, then Export. For scheduled runs, send it to Google Sheets and use File, Share, Publish to web, comma-separated values, so every run picks up fresh data. Localized headers, semicolon delimiters, comma decimals, and percent signs on CTR all parse without editing.
Score your queries in Python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("johnvc/ai-overview-rewrite-queue").call(run_input={"target_domains": ["example.com"], "search_console_csv_url": "https://docs.google.com/spreadsheets/d/e/EXAMPLE/pub?output=csv"})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
if row.get("tier") == "A":
print(row["query"], row["position"], row["impressions"], row["reference_domains"])
Filter to tier A and you have this month's rewrite list in a dozen lines.
Read the queue from Claude via MCP
Through the Model Context Protocol the Actor is a callable tool in Claude, Claude Code, and Cursor. Ask for the tier A list and an agent can pull it and start drafting in the same session, which is a better loop than exporting a CSV and pasting it into a chat window. You can read more about Claude at claude.ai.
FAQ about AI Overview citation scrapers
How does this scraper tell a real "no AI Overview" from a failed check?
They are separate fields and are never conflated. ai_overview_present: false means the query was checked and no overview appeared. A null with check_status of retrieval_failed or blocked means it could not be checked, and those land in tier X where they can never be scored as a rewrite priority. A failure that reads as a zero is worse than no data at all.
What does this scraper cost to run?
Pay per event: a one-time setup charge per run covering the export parsing, the citation checks, and your CTR baseline, plus a scored-row event per query returned. One thing to know is that the citation checks run on the Google AI Overview API under your own account and are billed separately under that Actor's pricing. That is the larger part of a run's cost. Use min_impressions to keep the checked set to queries with traffic worth having.
Can I schedule this scraper to track citations over time?
Yes, and you should. A single check is a snapshot, and overviews are volatile enough that one run is not a verdict. Put a published Sheet URL in search_console_csv_url, attach a Schedule to the rewrite queue, and trend the tiers instead of reading one run.
Can Claude drive this scraper over MCP?
Yes. Connect the Apify MCP server and it becomes a callable tool in any MCP client, so an agent can fetch the queue and act on it without leaving the conversation.
What can this scraper not tell me?
Whether you were recommended, as opposed to cited. It reports citation, meaning a link to your domain in the overview's sources. Whether the surrounding text recommends you, merely mentions you, or contradicts you is a text-analysis question this does not answer. It also runs the checks from datacenter addresses, so set gl, hl, and location deliberately or a lost citation is indistinguishable from a different exit node.
Can an AI agent pay for this scraper in USDC with x402?
Yes. The AI Overview Rewrite Queue supports agentic payments via the x402 protocol, so AI agents and MCP clients can pay for runs in USDC on Base with no Apify account or API token. Point your agent at the Apify MCP server and it can discover, pay for, and run the scraper autonomously; the Apify x402 announcement has the details.
More from Truffle Pig Data
The citation checks behind this one, usable on their own for raw overview text and sources: Google AI Overview API. The same visibility question on a different answer engine: Brave AI Mode API. For Korean search: Naver AI Overview API. And to find the question-shaped queries that trigger overviews in the first place: Google Autocomplete API.
Wrapping up
A citation trendline can sit flat and green while your clicks fall, because the overview is answering for you either way. Export Search Console once, point the AI Overview Rewrite Queue at it, and open the tier A rows first.
Top comments (0)