DEV Community

Cover image for How to scrape Google AI Mode: citation pills + place cards
Ricardo Batista
Ricardo Batista

Posted on • Originally published at cloro.dev

How to scrape Google AI Mode: citation pills + place cards

#ai #llm #dataextraction #aimode

Key takeaways

  • AI Mode and AI Overview cite the same URLs only 13.7% of the time, so scraping one tells you nothing reliable about the other.
  • Source URLs live in HTML comments keyed by UUID, not in anchor tags, so a scraper reading <a href> returns an empty citation list rather than an error.
  • Strip #:~:text scroll-to-text fragments before counting, or one article reads as several distinct URLs and inflates your citation numbers.

Google AI Mode does not put its sources in anchor tags. It renders each one as a button, and hides the actual URL in an HTML comment keyed by UUID. A scraper that reads <a href> finds nothing. That one design choice is most of the work in this post.

The rest is that nothing is in the initial HTML at all. The shell loads, then the answer streams in over a separate call to /async/folwr. Parse on page load and you get an empty container.

  1. Build the URL with udm=50 and aep=11 to force the interface.
  2. Register a network interceptor on https://www.google.com/async/folwr before navigating.
  3. Navigate, and treat a non-200 as a CAPTCHA rather than a dead page.
  4. Poll until the intercepted response lands, with a hard timeout.
  5. Read the answer text off data-session-thread-id, not off a snapshot.
  6. Collect the citation buttons, then rejoin them to their HTML-comment metadata by UUID.
  7. Rewrite the pills into anchors before converting anything to markdown.

Steps 6 and 7 are where the work is. The rest is plumbing.

Why bother with this surface specifically

AI Mode is not AI Overview with a different skin. Per ALM Corp's analysis of 1.3 million AI Mode citations, the two cite the same URLs only 13.7% of the time on semantically similar queries, and google.com self-cites in 17.42% of AI Mode answers, more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. Scraping one tells you nothing reliable about the other.

It is also where the comparison queries go. Seer Interactive's 2026 trigger data puts AI surfaces on 95.4% of comparison queries and 85.9% of question-format ones, and AI Mode passed 75 million daily active users by late 2025 per Alphabet's Q3 2025 earnings call. The query fan-out behind it, where Google splits your question into subtopics and runs many searches at once, is why one answer stitches together so many cited pages.

Surface Trigger Citation style Difficulty
AI Mode udm=50 Pills, metadata in HTML comments High: async streaming, drifting selectors
AI Overview Organic query, varies by class Inline source list above the SERP Medium: single fetch
Gemini gemini.google.com Grounded sources panel High: separate auth domain
ChatGPT Search chatgpt.com with search on Numbered citations Highest: SSE plus Cloudflare

Force the interface

udm=50 is the switch. aep=11 goes with it.

search_url = build_url_with_params(
    "https://www.google.com/search",
    {
        "udm": 50,        # AI Mode
        "aep": 11,
        "q": prompt,
        "hl": google_params["hl"],
        "gl": google_params["gl"],
    },
)
Enter fullscreen mode Exit fullscreen mode

Wait for the stream, then parse

We drive Playwright and capture the async call, then poll until it lands. Sixty seconds, then fail loudly. A hard timeout beats a silent partial parse, because a half-streamed answer looks like a real answer with fewer citations.

page_interceptor = PlaywrightInterceptor(do_not_block_resources=True)
page_interceptor.add_capture_urls(["https://www.google.com/async/folwr"])
await page_interceptor.setup_page_interceptor(page)

for _ in range(120):                      # 500ms intervals, 60s cap
    if len(page_interceptor.captured_responses):
        break
    await sleep(500)
else:
    raise Exception("Never received AI Mode response after 60 seconds")
Enter fullscreen mode Exit fullscreen mode

The answer text hangs off data-session-thread-id. Target that and read its parent, because AI Mode rewrites its own container several times while streaming and a locator survives that where a snapshot does not.

thread_element = page.locator("[data-session-thread-id]")
text = await thread_element.locator("..").text_content() or ""
Enter fullscreen mode Exit fullscreen mode

The citation pills

Here is the shape you are working against. A UUID ties the visible button to a comment block carrying the real URLs.

<!--Sv6Kpe[["uuid-12345",["label","description"],["https://example.com","source2"]]]-->
<button data-icl-uuid="uuid-12345" data-amic="true">[1]</button>
Enter fullscreen mode Exit fullscreen mode

So: find the buttons, read their UUIDs, regex the comments back out of the page HTML, and filter Google's own domains out of the URL list. One pill can carry several sources.

async def extract_aimode_citation_pills(page: Page) -> Dict[str, List[LinkData]]:
    citation_pills: Dict[str, List[LinkData]] = {}
    pill_locators = page.locator('button[data-icl-uuid][data-amic="true"]')
    page_html = html.unescape(await page.content())

    for i in range(await pill_locators.count()):
        pill_button = pill_locators.nth(i)
        if not await pill_button.is_visible():
            continue
        uuid = await pill_button.get_attribute("data-icl-uuid")
        if not uuid:
            continue

        pattern = rf'<!--Sv6Kpe\[\["{re.escape(uuid)}".*?]]-->'
        current_pill: List[LinkData] = []

        for content in re.findall(pattern, page_html, re.DOTALL):
            desc_match = re.search(
                rf'"{re.escape(uuid)}"\s*,\s*\[\s*"[^"]+"\s*,\s*"([^"]+)"', content
            )
            url = next(
                (u for u in re.findall(r'"(https://[^"]+)"', content)
                 if not any(skip in u for skip in
                            ["google.com", "gstatic.com", "encrypted-tbn"])),
                None,
            )
            if url:
                url = url.split("#:~:text")[0]
                url = url.replace("\\u003d", "=").replace("\\u0026", "&")
                current_pill.append(LinkData(
                    position=len(current_pill) + 1,
                    label=f"Source {len(current_pill) + 1}",
                    url=url,
                    description=desc_match.group(1) if desc_match else None,
                ))

        if current_pill:
            citation_pills[uuid] = current_pill

    return citation_pills
Enter fullscreen mode Exit fullscreen mode

The #:~:text strip matters more than it looks. Google appends scroll-to-text fragments, so the same article comes back as several distinct URLs and your citation counts inflate quietly.

The sources panel needs its own selector per layout. Web-results pages expose it one way, the dialog another. Playwright's actionability checks handle the waiting, so log and continue on timeout rather than killing the run.

sources_selector = (
    '[data-container-id="rhs-col"] [role="dialog"] a'
    if not is_web_results_page
    else "a.ZbQNgf"
)
Enter fullscreen mode Exit fullscreen mode

Converting to markdown without losing the sources

Hand the raw HTML to html2text and every pill becomes a dead button. Rewrite the pills into real anchors first, using the UUID map you already built, then convert.

soup = BeautifulSoup(html_content, "html.parser")
for button in soup.find_all("button", attrs={"data-icl-uuid": True, "data-amic": "true"}):
    uuid = button.get("data-icl-uuid")
    for link_data in reversed(citation_pills.get(uuid, [])):
        anchor = soup.new_tag("a", href=link_data.get("url"))
        anchor.string = link_data.get("label")
        button.insert_after(anchor)
    button.decompose()

h = html2text.HTML2Text()
h.ignore_links = False
h.body_width = 0
markdown = h.handle(str(soup)).strip()
Enter fullscreen mode Exit fullscreen mode

What breaks at scale

A non-200 is usually a CAPTCHA, not a dead page, so solve before you give up. One blocked request should not tank a batch.

response = await page.goto(search_url, timeout=20_000)
if not is_http_success(response.status):
    if not await solve_captcha(page, page_interceptor):
        raise Exception(f"HTTP error: {response.status} (probably captcha)")
Enter fullscreen mode Exit fullscreen mode

The real running cost is selector drift. AI Mode's DOM changes without notice, and a scraper that worked last month returns empty rather than throwing. Alert on success rate, not on exceptions. A sudden drop is a layout change, and you will find it weeks late if you are only watching for errors. You will also want a proxy pool and fingerprint rotation before this runs at any volume.

Or skip it

cloro maintains this pipeline as a managed endpoint, which is the disclosure to weigh this section against. One POST, parsed text and sources back, P50 under 8 seconds.

response = requests.post(
    "https://api.cloro.dev/v1/monitor/aimode",
    headers={"Authorization": "Bearer sk_live_your_api_key"},
    json={"prompt": "What do you know about Tesla's latest updates?",
          "country": "US",
          "include": {"markdown": True}},
)
result = response.json()["result"]
print(f"{len(result['sources'])} citations, fan-out: {result['searchQueries']}")
Enter fullscreen mode Exit fullscreen mode

Built in-house, this runs $5,000 to $10,000 a month once you count engineering time, browser instances and proxies. Below a few thousand queries a month that maths does not favour us, and you should build it. Above that it does, and cloro's AI Mode endpoint absorbs the selector drift.

If you want the dashboard rather than the pipeline, the tools that track this surface are compared separately.

Top comments (0)