DEV Community: BotFumble

Can AI agents find links in hover dropdown menus? 6 menu variants tested with Playwright

BotFumble — Mon, 26 Oct 2026 18:11:25 +0000

An AI agent reading your page through Playwright doesn't see your nav the way your customers do. That "Product" dropdown a person opens by moving the mouse across the header? In the snapshot the model gets, the links inside it are missing until something hovers, and nothing marks "Product" as having children. It might be why a model asked to find your pricing or signup link reports that there isn't one.

Short answer: it depends on how the submenu is hidden. With display:none until :hover, playwright-cli find Pricing returned "No matches found" and get_by_role("link", name="Pricing") counted 0 until something hovered "Product". A visibility:hidden submenu was also missing from the role tree at load. With opacity:0 plus pointer-events:none, the link was in the tree but a normal click timed out. Of six menu variants, the one with an extra always-rendered footer link was the only one where a role search found and clicked Pricing at load.

The snapshot matters because the Playwright MCP README says the server lets "LLMs interact with web pages through structured accessibility snapshots", and playwright-cli describes itself as "run playwright mcp commands from terminal". So the snapshot below is the page as a model using those tools gets it. No LLM ran in this post. Everything here is a fixed script or a CLI command.

In the pricing-page post I wrote that a "Plans" item inside a hover menu won't match get_by_role. I hadn't shown it. This is the check.

The test page

One nav, one hover submenu, nothing else:

<html><head><title>exp4</title><style>.sub{display:none}.menu:hover .sub{display:block}</style></head><body>
<nav><ul><li class="menu"><a href="/product">Product</a><ul class="sub"><li><a href="/pricing">Pricing</a></li><li><a href="/docs">Docs</a></li></ul></li><li><a href="/blog">Blog</a></li></ul></nav></body></html>

Save it as page.html and serve the folder with python3 -m http.server 8765.

Step 1: what playwright-cli returns

playwright-cli 0.1.22, run on 2026-10-26:

$ playwright-cli open http://localhost:8765/page.html
$ playwright-cli snapshot
- navigation [ref=e2]:
  - list [ref=e3]:
    - listitem [ref=e4]:
      - link "Product" [ref=e5] [cursor=pointer]:
        - /url: /product
    - listitem [ref=e6]:
      - link "Blog" [ref=e7] [cursor=pointer]:
        - /url: /blog
$ playwright-cli find Pricing
### Result
No matches found for "Pricing".
$ playwright-cli hover e5
$ playwright-cli find Pricing
### Result
Found 2 matches for "Pricing":
- navigation [ref=e2]:
  - list [ref=e3]:
    - listitem [ref=e4]:
      - link "Product" [ref=e5] [cursor=pointer]:
        - /url: /product
      - list [ref=e8]:
        - listitem [ref=e9]:
          - link "Pricing" [ref=e10] [cursor=pointer]:
            - /url: /pricing
        - listitem [ref=e11]:
          - link "Docs" [ref=e12] [cursor=pointer]:
            - /url: /docs

(Output trimmed to the snapshot and find results. The only console error was a 404 for favicon.ico.)

Before the hover, nothing in the snapshot hints that "Product" has children. A model that wants pricing has to decide on its own to hover a plain link. (find printed "2 matches" after the hover; the tool's output shows one Pricing link.) "Pricing" and "Docs" exist in the HTML and are absent from what the model gets.

Step 2: the same check from Python

# exp4: is a display:none-until-hover submenu in the accessibility tree? Fixed script, not an agent.
import pathlib, datetime
from playwright.sync_api import sync_playwright
url = pathlib.Path('page.html').resolve().as_uri()
with sync_playwright() as p:
    b = p.chromium.launch(); pg = b.new_page(); pg.goto(url)
    print("== aria snapshot before hover\n" + pg.locator('body').aria_snapshot())
    print("get_by_role link Pricing count:", pg.get_by_role('link', name='Pricing').count())
    pg.hover('text=Product')
    print("== aria snapshot after hover\n" + pg.locator('body').aria_snapshot())
    print("get_by_role link Pricing count:", pg.get_by_role('link', name='Pricing').count())
    b.close()
print(datetime.datetime.now().strftime('%F %H:%M'))

Output with Playwright 1.64.0 (Python 3.13.5), snapshots trimmed to the counts:

get_by_role link Pricing count: 0
get_by_role link Pricing count: 1
2026-10-26 20:06

A bug of my own first: the script hard-codes page.html, and I had saved the page under a different name, so the first rerun died with net::ERR_FILE_NOT_FOUND. If you copy this, keep the file name and the script in step.

Step 3: six menu variants and what changed

I wrote a second script, exp4b.py, that builds six versions of the same nav and checks each one at load, after one Tab key press, and with a direct click on the Pricing link by role (1.5 s timeout). Chromium 156.0.8078.4. "In DOM" is a CSS selector count, a[href='/pricing'].

# exp4b: hover-only submenu vs five variants (fixes and other hiding styles). Fixed Playwright script, no LLM, no agent.
# Run: python3 exp4b.py   (writes exp4b-*.html next to itself, Chromium only)
import pathlib, datetime
from playwright.sync_api import sync_playwright

NAV = '<li class="menu">{trigger}<ul class="sub" id="sub"><li><a href="/pricing">Pricing</a></li><li><a href="/docs">Docs</a></li></ul></li><li><a href="/blog">Blog</a></li>'
LINK = '<a href="/product">Product</a>'
BTN = '<button aria-expanded="false" aria-controls="sub" onclick="const o=this.getAttribute(\'aria-expanded\')===\'true\';this.setAttribute(\'aria-expanded\',String(!o));document.getElementById(\'sub\').style.display=o?\'none\':\'block\'">Product</button>'
PAGES = {
  "1-hover-only":   ('.sub{display:none}.menu:hover .sub{display:block}', LINK, ''),
  "2-focus-within": ('.sub{display:none}.menu:hover .sub,.menu:focus-within .sub{display:block}', LINK, ''),
  "3-disclosure":   ('.sub{display:none}', BTN, ''),
  "5-visibility-hidden": ('.sub{visibility:hidden}.menu:hover .sub{visibility:visible}', LINK, ''),
  "6-opacity-0":    ('.sub{opacity:0;pointer-events:none}.menu:hover .sub{opacity:1;pointer-events:auto}', LINK, ''),
  "4-footer-link":  ('.sub{display:none}.menu:hover .sub{display:block}', LINK, '<footer><a href="/pricing">Pricing</a></footer>'),
}
here = pathlib.Path(__file__).resolve().parent
with sync_playwright() as p:
    b = p.chromium.launch()
    print("chromium", b.version)
    for name, (css, trigger, extra) in PAGES.items():
        f = here / f"exp4b-{name}.html"
        f.write_text(f'<html><head><title>{name}</title><style>{css}</style></head><body><nav><ul>{NAV.format(trigger=trigger)}</ul></nav>{extra}</body></html>')
        pg = b.new_page(); pg.goto(f.as_uri())
        role = lambda: pg.get_by_role('link', name='Pricing').count()
        print(f"\n== {name}")
        print("  css a[href='/pricing'] in DOM:", pg.locator("a[href='/pricing']").count(),
              "| get_by_role link Pricing at load:", role())
        try:
            pg.get_by_role('link', name='Pricing').first.click(timeout=1500)
            print("  click by role at load: clicked")
        except Exception as e:
            print("  click by role at load:", type(e).__name__, str(e).splitlines()[0][:70])
        pg.goto(f.as_uri())  # run-1 bug: a successful click navigated away, so reload before the keyboard check
        pg.keyboard.press('Tab')
        print("  after 1x Tab, focused:", pg.evaluate("document.activeElement.outerHTML.slice(0,40)"), "| role count:", role())
        if name == "3-disclosure":
            print("  button line before Enter:", [l.strip() for l in pg.locator('nav').aria_snapshot().splitlines() if 'button' in l])
            pg.keyboard.press('Enter')
            print("  button line after Enter: ", [l.strip() for l in pg.locator('nav').aria_snapshot().splitlines() if 'button' in l], "| role count:", role())
        pg.close()
    b.close()
print("\nrun at", datetime.datetime.now().strftime('%F %H:%M'), "Asia/Jerusalem")

Its first run had a second bug of mine. In the footer variant the click worked, the page navigated to /pricing, and the Tab check then ran on a page that no longer had a nav. The pg.goto reload before the Tab check fixes that, and the table below is from the run after the fix.

Variant	In DOM	Pricing by role at load	Click by role at load	By role after 1x Tab
1. `display:none` until `:hover`	1	0	TimeoutError	0
2. Same plus `:focus-within`	1	0	TimeoutError	1
3. `<button aria-expanded>` that toggles the menu	1	0	TimeoutError	0 (1 after Enter)
4. Variant 1 plus a plain footer link	2	1	clicked	1
5. `visibility:hidden` until `:hover`	1	0	TimeoutError	0
6. `opacity:0; pointer-events:none` until `:hover`	1	1	TimeoutError	1

What each row shows:

Rows 1 and 5. The link is in the HTML and missing from the role tree at load. A keyboard user tabbing to "Product" doesn't open the menu either. (Hover was tested for row 1 in Step 2, not for row 5.)
Row 2. Adding .menu:focus-within .sub{display:block} opens the menu on keyboard focus. It does nothing for a snapshot taken at load.
Row 3. The disclosure button opened the menu on Enter. Its snapshot line was - button "Product" before and - button "Product" [expanded] after. In this Playwright version the closed state printed no expanded=false, so the closed button gives the model no hint that a menu is there.
Row 4. One always-rendered link made Pricing findable and clickable at load.
Row 6 fails the other way. The link is in the role tree from the start and is_visible() returned True, but the click timed out with <li class="menu">… intercepts pointer events repeated in the call log. A snapshot-reading tool finds a link that a normal Playwright click can't reach. (force=True or dispatch_event would skip that check; that changes what the test means, see the previous post.)

What I'd change on a real site

Put the two or three links that matter (signup, pricing, docs) somewhere that is always rendered: top-level nav or footer. Row 4 was the only variant where a role search found and clicked Pricing at load.
Add :focus-within to CSS hover menus. It costs one selector and opens the menu for keyboard users (row 2).
If you build the menu with a button, use the W3C disclosure navigation pattern so keyboard and screen-reader users can open it. In row 3, Enter on the button opened it and the role count went to 1. The closed button still showed no hint in the snapshot, so pair it with fix 1.
Avoid hiding menus with opacity:0 plus pointer-events:none alone. In row 6 the link was in the snapshot and a normal click on it timed out.

To test your own signup link: open the page with playwright-cli, run find "Sign up" before touching anything, then try get_by_role("link", name="Sign up").click(). If the find is empty or the click times out, a model using those tools has to guess an interaction first.

Limitations

Synthetic pages, Chromium only, one Playwright version and one playwright-cli version.
No LLM agent ran. I don't know how often a real agent decides to hover "Product" when the search comes back empty.
The link is in the raw HTML in every variant (CSS selector count 1 or 2), so an agent that reads HTML instead of the accessibility tree may behave differently. HTML-reading and screenshot-based agents were not tested.

FAQ

Can AI agents see links in hover dropdown menus?
Not at load, if the agent reads Playwright's accessibility snapshot and the submenu is hidden with display:none. playwright-cli find Pricing returned no match until "Product" was hovered. A visibility:hidden submenu was also absent from the role tree at load.

Does get_by_role find hidden menu links?
Not while they are hidden with display:none or visibility:hidden. It counted 0 at load in variants 1, 2, 3 and 5. It counted 1 after a hover (variant 1, Step 2), after Tab focus (variant 2) and after Enter on the button (variant 3).

Is opacity: 0 a safe way to hide a menu from agents?
It keeps the link in the snapshot, but with pointer-events:none a Playwright click timed out because the parent <li> intercepted pointer events.

What is the smallest fix?
One always-rendered link to the page that matters. In row 4 a footer link made Pricing findable and clickable at load.

Both scripts and the test page are above. If your browser or menu library gives a different row, put it in the comments.

I write these while building BotFumble, a tool for seeing where AI agents get stuck on real signup and checkout flows. It is not live yet. I post shorter notes from these runs on X as @VerigateAI.

Playwright docs say dispatch_event('click') equals element.click(). On a disabled button it didn't.

BotFumble — Mon, 26 Oct 2026 11:00:08 +0000

We gave a script one simple task: press three buttons on a tiny signup form. A disabled "Continue", a "Create account" pushed 5000px off-screen, and a "Next" with visibility:hidden. A mouse user can press none of them.

Playwright's Locator docs say dispatch_event("click") "is equivalent to calling element.click()". On the disabled button they behaved differently. dispatch_event ran the button's onclick. element.click() did not.

In our run (playwright-python 1.64.0, Chromium 156), the four calls gave three different outcomes on the same disabled button, and two of them returned normally while its onclick never ran. If a test or an agent tool counts "no exception" as "the click worked", it can pass when nothing happened, and pass when a handler ran that no user could trigger.

Why this matters for agent tools

An agent tool that clicks through dispatch_event or a JavaScript element.click() hands the model whatever that call does. We have not checked which tools use which. We did see the effect in one of them earlier the same day: agent-browser (its package.json calls it a "Browser automation CLI for AI agents"), driven by fixed commands, which makes that run a script. Version 0.39.1 printed ✓ Done with exit code 0 on all three buttons, and document.title stayed empty, so no onclick ran. Version 0.39.2 refused all three with a reason, for example Element '#cont' is disabled, so the click would not reach it. That change came in PR #2173.

An agent that reads "Done" plans its next step on a page that never changed. So we checked what plain Playwright does with the same buttons.

The setup

The test page. Save it as exp3-page.html, since the script loads it by that name:

<!doctype html><html><body><h1>Sign up</h1>
<form onsubmit="return false"><input aria-label="Email" value="a@b.co">
<button id="cont" disabled onclick="document.title='clicked-disabled'">Continue</button>
<button id="off" style="position:absolute;left:-5000px" onclick="document.title='clicked-off'">Create account</button>
<button id="hid" style="visibility:hidden" onclick="document.title='clicked-hidden'">Next</button></form></body></html>

Every onclick writes to document.title, so an empty title after the call means the handler did not run. The script opens a fresh page for each button and method, tries one click with a 2 second timeout, and records whether the call raised and what the title became. There is no LLM in the loop. This is the exact file we ran:

# exp6: same 3 unclickable buttons (exp3-page.html), 4 ways to "click" in Playwright (Python).
# For each: did the call report success (no exception) and did the onclick actually run (document.title)?
import json, pathlib, datetime, playwright
from playwright.sync_api import sync_playwright
page_url = pathlib.Path("exp3-page.html").resolve().as_uri()
methods = {
 "click()": lambda l: l.click(timeout=2000),
 "click(force=True)": lambda l: l.click(force=True, timeout=2000),
 "dispatch_event('click')": lambda l: l.dispatch_event("click", timeout=2000),
 "evaluate(el.click())": lambda l: l.evaluate("e => e.click()", timeout=2000),
}
res=[]
with sync_playwright() as p:
  b=p.chromium.launch()
  for bid in ["cont","off","hid"]:
    for name,fn in methods.items():
      pg=b.new_page(); pg.goto(page_url)
      err=None
      try: fn(pg.locator("#"+bid))
      except Exception as e: err=str(e).splitlines()[0][:160]
      title=pg.title()
      res.append({"button":bid,"method":name,"reported_success":err is None,"onclick_ran":title.startswith("clicked"),"title":title,"error":err})
      pg.close()
  ver=b.version; b.close()
out={"run_at":datetime.datetime.now().astimezone().isoformat(),"tool":"playwright-python "+__import__("importlib.metadata").metadata.version("playwright"),"chromium":ver,"page":"exp3-page.html","kind":"SCRIPT, no LLM","results":res}
json.dump(out,open("exp6-results.json","w"),indent=1)
for r in res: print(r["button"],"|",r["method"],"| success:",r["reported_success"],"| onclick ran:",r["onclick_ran"],"|",r["error"] or "")

We ran it twice and got identical results both times.

Results

Button	`click()`	`click(force=True)`	`dispatch_event("click")`	`evaluate(e => e.click())`
disabled	timeout, no run	returned, no run	returned, ran	returned, no run
off-screen (`left:-5000px`)	timeout, no run	error: outside of the viewport	returned, ran	returned, ran
`visibility:hidden`	timeout, no run	returned, no run	returned, ran	returned, ran

"Returned" means the call finished without an exception. "Ran" means the onclick changed the title. We measured only the onclick, not which input events reached the page.

What each column shows

click() waits for Playwright's actionability checks (visible, stable, receives events, enabled). It raised Timeout 2000ms exceeded on all three buttons.

click(force=True) disables what the docs call the non-essential actionability checks. On the disabled and hidden buttons it returned normally and nothing ran, which is what a real user would also get from those buttons. A test that only checks for "no exception" passes here. On the off-screen button force still raised Element is outside of the viewport.

dispatch_event("click") fires a DOM event at the element. The docs say it is dispatched regardless of visibility, which matches the hidden and off-screen rows. The disabled row is the one that contradicts "equivalent to element.click()": the onclick ran.

evaluate(e => e.click()) is plain JavaScript run through Playwright's evaluate. It ran the handler on the hidden and off-screen buttons and did nothing on the disabled one.

Limitations

The off-screen button is still focusable, so a keyboard user could probably Tab to it. We did not test that. The disabled and hidden buttons are the ones no user can press.
We only detected the onclick through document.title. Form submission and other side effects were not checked.
Playwright's dispatch_event creates the event and dispatches it on the element. Whether a disabled button's listeners then fire is up to the browser, so Firefox and WebKit may give a different disabled row. We only ran Chromium.
One synthetic page, one Chromium build, one Playwright version.
There is no test runner and no assertions in this run, only calls that did or did not raise.

FAQ

Does Playwright force=True click a disabled button?
The call returned without an error in our run, and the button's onclick did not run.

Is dispatch_event("click") the same as a user click?
Not on a disabled button in Chromium: it ran an onclick that no user could trigger, and element.click() on the same button ran nothing.

What we changed in our own checks

Assert on what the click should cause: a URL, a title, a network request, a new element. With Playwright's expect, for example expect(page).to_have_title("Welcome"), or page.wait_for_url(...).
Treat force=True, dispatch_event and evaluate(... .click()) in a user-flow test as a review question. Each one removes part of what makes the test mean "a person could do this".
If you build tools for agents, return what changed on the page after the action. agent-browser 0.39.2 refuses the click and gives the reason, and that reason is something the model can act on.

The page and script above are the full repro. If your browser or Playwright version gives a different table, we'd like to see it in the comments.

I write these while building BotFumble, a tool for seeing where AI agents get stuck on real signup and checkout flows. It is not live yet.

How to test if an AI agent can find your pricing page (Playwright, no LLM)

BotFumble — Sun, 25 Oct 2026 18:08:11 +0000

If you want to know whether an AI agent can get from your homepage to your prices, you can check most of it with a plain Playwright script and no model at all. The script looks for a link whose accessible name contains "Pricing", clicks it, waits for a price to render, and reads the prices from the page text. If any step fails, a browser agent driving your site through the accessibility tree or the DOM will probably struggle at the same spot.

I ran a first version of this on five browser-infrastructure sites today (Steel, Browser Use, Anchor Browser, Hyperbrowser and Browserless). All five passed. The script itself failed three times before it gave honest answers, and those failures are the useful part of this post.

What the check covers

Three questions, in order:

Is anything covering the page on load? The script counts visible elements with role="dialog" or aria-modal="true". A cookie wall or signup modal is the first thing an agent has to get past.
Is there a visible link named "Pricing"? It uses get_by_role("link", name=...), which matches the accessible name, the same thing a screen reader announces. An icon with no label, or a "Plans" item hidden inside a hover menu, won't match.
Are the prices plain text? It pulls every dollar amount out of innerText. Prices drawn in an image, a canvas, or hidden behind "Contact sales" don't show up.

It does not tell you how GPT, Claude or any specific agent behaves. It tells you whether the path is discoverable from the DOM, which is the floor every agent has to clear.

The script

import asyncio, json, re, sys
from playwright.async_api import async_playwright

PRICE = re.compile(r"\$\s?\d[\d,]*(?:\.\d+)?")

async def check(browser, url):
    ctx = await browser.new_context(viewport={"width": 1280, "height": 800}, locale="en-US")
    page = await ctx.new_page()
    r = {"site": url}
    try:
        await page.goto(url, wait_until="domcontentloaded", timeout=30000)
        try:  # busy sites with analytics may never go idle
            await page.wait_for_load_state("networkidle", timeout=10000)
        except Exception:
            pass
        # 1. Anything covering the page?
        r["open_dialogs"] = await page.locator("[role=dialog]:visible, [aria-modal=true]:visible").count()
        # 2. Is there a link an agent can find by its accessible name?
        links = page.get_by_role("link", name=re.compile("pricing", re.I))
        r["pricing_links"] = await links.count()
        visible = None
        for i in range(r["pricing_links"]):
            if await links.nth(i).is_visible():
                visible = links.nth(i)
                break
        r["visible_in_first_viewport"] = visible is not None
        if visible is None:
            r["result"] = "FAIL: no visible link named Pricing"
            return r
        await visible.click(timeout=8000)
        try:  # client-side routers change the URL after the click returns
            await page.wait_for_url(re.compile("pricing", re.I), timeout=10000)
        except Exception:
            pass
        try:  # wait for a price to render instead of sleeping a fixed time
            await page.get_by_text(PRICE).first.wait_for(timeout=10000)
        except Exception:
            pass
        r["landed_on"] = page.url
        # 3. Are the prices plain text, or locked in images / canvas / a sales form?
        text = await page.inner_text("body")
        r["prices_in_text"] = sorted(set(PRICE.findall(text)))[:12]
        r["contact_sales"] = bool(re.search(r"contact (sales|us)", text, re.I))
        r["result"] = "PASS" if r["prices_in_text"] else "WARN: no prices in page text"
        await page.screenshot(path=re.sub(r"\W+", "_", url) + ".png")
    except Exception as e:
        r["result"] = f"ERROR: {str(e)[:150]}"
    finally:
        await ctx.close()
    return r

async def main(urls):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        results = [await check(browser, u) for u in urls]
        await browser.close()
    print(json.dumps(results, indent=1))

asyncio.run(main(sys.argv[1:]))

Run it with pip install playwright && playwright install chromium, then python pricing_check.py https://yoursite.com/. Each site gets a JSON row and a screenshot of where the script ended up.

Three ways the script lied to me

1. It read the wrong page. My first version clicked "Pricing" and read the text right away. On steel.dev the result said it had landed on https://steel.dev/, the homepage, because the site's client-side router updated the URL after click() had already returned. Steel also lists prices on its homepage, so the check would have passed for the wrong reason. The fix is wait_for_url on a pattern before reading anything.

2. It timed out on a site that was fine. Waiting for networkidle with a 10-second limit threw a timeout on browserless.io and marked the whole site as an error. Some pages keep a request open the whole time, so "idle" never arrives. Treat networkidle as a best effort and move on.

3. It reported "no prices" on a page full of prices. After fixing the first two, Browserless came back as WARN: no prices in page text. An earlier version of the same check, which slept 3.5 seconds after each click, had found seven dollar amounts there. The pricing table renders after the document loads, and I was reading too early. Waiting for the first element whose text matches the price pattern fixed it, and Browserless passed.

My earlier version used fixed sleeps, so its timings measured my own waits more than the sites. That's why this post reports pass or fail and no load times.

The same trap applies to signup checks

We have an open bug in our own checker that's the signup version of mistake number 1. A rule that says "the agent found signup if it found an email input" counts a newsletter box in the footer as a successful signup. If you write a signup check like this, assert on something that only the real flow produces: the URL after submit, a "check your inbox" message, or a new row in your test database.

What a pass looks like

On today's run, all five sites had at least two links whose accessible name matched "Pricing", one of them visible without scrolling, no open dialog on load, and dollar amounts in the page text. Anchor Browser's link goes to a #pricing section on the homepage instead of a separate page, which still passes because the prices are readable text.

FAQ

Does a pass mean ChatGPT's agent will find my prices? No. It means the path is there in the DOM with names a machine can match. Real agents also deal with bot checks, slow pages and their own mistakes.

Why match on "pricing" only? Because it's the word most sites use. If your nav says "Plans", change the regex to pricing|plans and make sure the label is real text or an aria-label, not only an icon.

Should I run this in CI? It's cheap enough to run on every deploy against staging. A redesign that moves pricing into a hover menu would turn it red.

I'm building BotFumble, which runs checks like this on signup and checkout flows and records where an agent gets stuck. It isn't open yet. If you want to see the next runs, including the ones that fail, follow @VerigateAI on X, where I post each one with the script output.