Creator of GithubStarsManager — an open-source, local-first tool that organizes GitHub stars with AI: auto-categorization, semantic search, release tracking, and an MCP server for star library.
The allowSubmit=false default is the part I'd steal first — one wrong click sending half-checked data live is exactly the failure mode that makes people distrust browser agents. And the React _valueTracker detail is the kind of thing you only learn by watching a fill vanish on blur. Curious about your open question: did SUBMIT_LABEL_RE ever misclassify a safe button in your testing, like a 'Save draft' or 'Continue' that also triggers a submit? Nice write-up, the local benchmark numbers make the trade-offs easy to follow.
Full-stack developer in Vietnam. I build browser automation and MCP tools for AI agents (turbo-browser-mcp) and Postgres RLS checks for Supabase (supabase-rls-check).
Location
Vietnam (UTC+7)
Education
IT student, Industrial University of Ho Chi Minh City (IUH)
Yes, that exact case bit me early on. Multi-step wizards (like Workable and Greenhouse forms) often render "Continue", "Next", or "Save draft" as <button type="submit"> inside a <form>, so blocking every submit-type button broke pagination and draft saving.
I ended up splitting the check into three regexes in src/dom-engine.mjs: SUBMIT_LABEL_RE (publish, submit, send, pay, delete), SAFE_SUBMIT_TYPE_RE (next, continue, back, search, filter, save draft, preview, cancel), and COMPOUND_DANGER_RE. If a <button type="submit"> matches SAFE_SUBMIT_TYPE_RE, the form guard temporarily marks form.__tbmAllowedSubmit = true for that click unless the label also contains a final-action phrase like "Continue to payment" or "Save and submit".
Creator of GithubStarsManager — an open-source, local-first tool that organizes GitHub stars with AI: auto-categorization, semantic search, release tracking, and an MCP server for star library.
That three-way split is a neat fix — I hadn't thought about the wizard case where 'Continue' is technically a submit button but semantically just pagination. The COMPOUND_DANGER_RE escape hatch for 'Continue to payment' is the part I'd steal back, honestly. Thanks for walking through the actual regexes, that makes the trade-off a lot clearer than the usual 'just block submits' advice.
the react _valuetracker hack is so painfully real. i've spent hours trying to figure out why .value = '...' wasn't sticking on react forms until i realized i had to bypass the prototype. but to answer your question: custom virtualized dropdowns (like react-select) are the absolute worst for agents. they literally don't exist in the dom until you click them, so standard snapshots miss them completely. how are you handling those?
Full-stack developer in Vietnam. I build browser automation and MCP tools for AI agents (turbo-browser-mcp) and Postgres RLS checks for Supabase (supabase-rls-check).
Location
Vietnam (UTC+7)
Education
IT student, Industrial University of Ho Chi Minh City (IUH)
react-select and Radix/Headless UI portals are definitely the trickiest ones because the <div role="option"> nodes only mount after focus or pointerdown.
Right now smart_fill_form handles the standard [role="combobox"] / [aria-haspopup="listbox"] pattern in two steps inside the same call: it clicks the combobox trigger first, waits briefly for the portal/listbox to mount in the document, and then matches the visible [role="option"] text. Where that still breaks is heavy windowing (like react-window inside a 5,000-row select) where the target option isn't mounted until you type into the inner search input or scroll the list container. For those, I currently fall back to batch_actions (click trigger -> type filter text -> click option).
oh man, react-window inside a select is the absolute final boss of browser automation. if it's not in the dom, you literally can't click it, so the batch_actions fallback (click -> type -> click) is actually the smartest way to handle it. it mimics what a real human would do anyway. trying to manually scroll a virtualized list via script is a special kind of hell lmao.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
onlyChanged is where I'd expect the first bug: a per-line signature diff reports a deleted row as nothing changed, and a re-sorted list as a batch of new controls, so the agent still clicks f0:e12 that now points at a different button. Test it with two snapshots of the same 140-button page where one li is removed and the rest are reordered - if the second delta comes back empty, the ref map is lying about what is on screen.
Full-stack developer in Vietnam. I build browser automation and MCP tools for AI agents (turbo-browser-mcp) and Postgres RLS checks for Supabase (supabase-rls-check).
Location
Vietnam (UTC+7)
Education
IT student, Industrial University of Ho Chi Minh City (IUH)
You were right about deletions. The delta listed only new or changed lines, so a removed row looked like "nothing changed". It now ends with a Removed: f0:e12, … line. Reordering was already fine: refs live as an attribute on the element, not by position, so a re-sorted list keeps its refs and adds nothing to the delta. Your exact case is now a test: 140 buttons, one removed, the rest reordered; the delta lists only the removed ref, and clicking a surviving ref still hits the same button. It's on main, not released yet.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Good that a removed row now says so — the piece I would still pin down is ref reuse. If a later snapshot ever hands f0:e12 to a different element, an agent holding the stale ref clicks the wrong button and the delta looks clean, so a monotonic ref counter that never recycles plus a test asserting a stale ref comes back "gone" rather than resolving to a fresh element is what makes that Removed line trustworthy.
Full-stack developer in Vietnam. I build browser automation and MCP tools for AI agents (turbo-browser-mcp) and Postgres RLS checks for Supabase (supabase-rls-check).
Location
Vietnam (UTC+7)
Education
IT student, Industrial University of Ho Chi Minh City (IUH)
You were right to push on this. Full snapshots used to renumber from e1, so a stale ref could land on a new element. Now refs never recycle within a document: survivors keep theirs, new elements get numbers above a per-page high-water mark, and a removed element's ref fails as not found. Your 140-button case is a test. On main, not released yet.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Per-page high-water is the right move, and the case I would still test is navigation: a new document resets the ref namespace, so an agent holding a pre-navigation ref has to come back not-found rather than land on the same-numbered element in the new page. A counter scoped to the session instead of the document gets you that for free.
Full-stack developer in Vietnam. I build browser automation and MCP tools for AI agents (turbo-browser-mcp) and Postgres RLS checks for Supabase (supabase-rls-check).
Location
Vietnam (UTC+7)
Education
IT student, Industrial University of Ho Chi Minh City (IUH)
That's a real gap. The counter is per document, so after a navigation a pre-navigation ref only fails when the new page has fewer elements than its number. On a bigger page it would resolve to whatever got that number. I'm moving the counter to the tab so it carries across navigations.
Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.
Resolving raw { x, y } coordinates against SUBMIT_LABEL_RE before firing a mouse click is such a smart detail. A label-based submit guard that an agent can accidentally bypass with a coordinate click is just security theater, so keeping allowSubmit=false by default and checking the underlying element at the target coordinates gives real human-in-the-loop safety before anything irreversible goes live.
Resetting React's _valueTracker alongside native prototype setters in smart_fill_form also fixes one of the most annoying silent failure modes in browser automation, where .value = "..." looks completely fine in the DOM until blur or submit wipes the controlled state clean.
To answer your question on what breaks most often in daily workflows: portaled dropdowns and virtualized comboboxes (like Radix or Headless UI menus that mount their listbox options at document.body instead of inside the form container). When the trigger is inside a nested frame or card and the options render at the root, or only mount while focused—scoped snapshots and onlyChanged deltas can easily miss the newly mounted option nodes.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The 48 KiB PDF number is the one that made me wince — 44,728 tokens just to hand a file to the model, versus 85 for a disk path. That's not an optimisation, that's a whole failure mode hiding in plain sight. Haven't run turbo-browser-mcp myself, but the upload fix alone would've saved me a few very long afternoons.
Full-stack developer in Vietnam. I build browser automation and MCP tools for AI agents (turbo-browser-mcp) and Postgres RLS checks for Supabase (supabase-rls-check).
Location
Vietnam (UTC+7)
Education
IT student, Industrial University of Ho Chi Minh City (IUH)
Thanks. That one surprised me too when I measured it. The server only passes the file path to Playwright's setInputFiles, so the bytes never go through the model at all.
On the open question about SUBMIT_LABEL_RE misclassifying buttons: I moved that check to the request layer a while ago and never looked back. Continue in a Workable wizard and Submit on the last step both sit on a submit button, but only one ends in a URL change or a mutating fetch. If you record the URL and in-flight requests for about a second after the click, draft and save buttons give themselves away: they fire an XHR and stay on the same page. Labels lie in every language, so classifying by aftermath sidesteps the naming problem entirely.
For the portal dropdowns, keyboard events saved me more than pointer ones. Focus the combobox and send ArrowDown then Enter. react-select and Radix open their menu on focus anyway, so you never have to wait for the option list to mount, and react-window stays somebody else's problem.
Full-stack developer in Vietnam. I build browser automation and MCP tools for AI agents (turbo-browser-mcp) and Postgres RLS checks for Supabase (supabase-rls-check).
Location
Vietnam (UTC+7)
Education
IT student, Industrial University of Ho Chi Minh City (IUH)
Took most of this. Since today a click that starts a POST, PUT, PATCH or DELETE within 800 ms comes back with possibleSubmit plus the method and host. It reports and doesn't block, and the label check stays as the first gate. The first false positive was an ad beacon firing on dev.to's own Publish button, so analytics hosts are excluded now.
We're a place where coders share, stay up-to-date and grow their careers.
The allowSubmit=false default is the part I'd steal first — one wrong click sending half-checked data live is exactly the failure mode that makes people distrust browser agents. And the React _valueTracker detail is the kind of thing you only learn by watching a fill vanish on blur. Curious about your open question: did SUBMIT_LABEL_RE ever misclassify a safe button in your testing, like a 'Save draft' or 'Continue' that also triggers a submit? Nice write-up, the local benchmark numbers make the trade-offs easy to follow.
Yes, that exact case bit me early on. Multi-step wizards (like Workable and Greenhouse forms) often render "Continue", "Next", or "Save draft" as
<button type="submit">inside a<form>, so blocking every submit-type button broke pagination and draft saving.I ended up splitting the check into three regexes in
src/dom-engine.mjs:SUBMIT_LABEL_RE(publish, submit, send, pay, delete),SAFE_SUBMIT_TYPE_RE(next, continue, back, search, filter, save draft, preview, cancel), andCOMPOUND_DANGER_RE. If a<button type="submit">matchesSAFE_SUBMIT_TYPE_RE, the form guard temporarily marksform.__tbmAllowedSubmit = truefor that click unless the label also contains a final-action phrase like "Continue to payment" or "Save and submit".That three-way split is a neat fix — I hadn't thought about the wizard case where 'Continue' is technically a submit button but semantically just pagination. The COMPOUND_DANGER_RE escape hatch for 'Continue to payment' is the part I'd steal back, honestly. Thanks for walking through the actual regexes, that makes the trade-off a lot clearer than the usual 'just block submits' advice.
the react _valuetracker hack is so painfully real. i've spent hours trying to figure out why .value = '...' wasn't sticking on react forms until i realized i had to bypass the prototype. but to answer your question: custom virtualized dropdowns (like react-select) are the absolute worst for agents. they literally don't exist in the dom until you click them, so standard snapshots miss them completely. how are you handling those?
react-selectand Radix/Headless UI portals are definitely the trickiest ones because the<div role="option">nodes only mount after focus or pointerdown.Right now
smart_fill_formhandles the standard[role="combobox"]/[aria-haspopup="listbox"]pattern in two steps inside the same call: it clicks the combobox trigger first, waits briefly for the portal/listbox to mount in the document, and then matches the visible[role="option"]text. Where that still breaks is heavy windowing (likereact-windowinside a 5,000-row select) where the target option isn't mounted until you type into the inner search input or scroll the list container. For those, I currently fall back tobatch_actions(clicktrigger ->typefilter text ->clickoption).oh man, react-window inside a select is the absolute final boss of browser automation. if it's not in the dom, you literally can't click it, so the batch_actions fallback (click -> type -> click) is actually the smartest way to handle it. it mimics what a real human would do anyway. trying to manually scroll a virtualized list via script is a special kind of hell lmao.
onlyChanged is where I'd expect the first bug: a per-line signature diff reports a deleted row as nothing changed, and a re-sorted list as a batch of new controls, so the agent still clicks f0:e12 that now points at a different button. Test it with two snapshots of the same 140-button page where one li is removed and the rest are reordered - if the second delta comes back empty, the ref map is lying about what is on screen.
You were right about deletions. The delta listed only new or changed lines, so a removed row looked like "nothing changed". It now ends with a Removed: f0:e12, … line. Reordering was already fine: refs live as an attribute on the element, not by position, so a re-sorted list keeps its refs and adds nothing to the delta. Your exact case is now a test: 140 buttons, one removed, the rest reordered; the delta lists only the removed ref, and clicking a surviving ref still hits the same button. It's on main, not released yet.
Good that a removed row now says so — the piece I would still pin down is ref reuse. If a later snapshot ever hands f0:e12 to a different element, an agent holding the stale ref clicks the wrong button and the delta looks clean, so a monotonic ref counter that never recycles plus a test asserting a stale ref comes back "gone" rather than resolving to a fresh element is what makes that Removed line trustworthy.
You were right to push on this. Full snapshots used to renumber from e1, so a stale ref could land on a new element. Now refs never recycle within a document: survivors keep theirs, new elements get numbers above a per-page high-water mark, and a removed element's ref fails as not found. Your 140-button case is a test. On main, not released yet.
Per-page high-water is the right move, and the case I would still test is navigation: a new document resets the ref namespace, so an agent holding a pre-navigation ref has to come back not-found rather than land on the same-numbered element in the new page. A counter scoped to the session instead of the document gets you that for free.
That's a real gap. The counter is per document, so after a navigation a pre-navigation ref only fails when the new page has fewer elements than its number. On a bigger page it would resolve to whatever got that number. I'm moving the counter to the tab so it carries across navigations.
Resolving raw
{ x, y }coordinates againstSUBMIT_LABEL_REbefore firing a mouse click is such a smart detail. A label-based submit guard that an agent can accidentally bypass with a coordinate click is just security theater, so keepingallowSubmit=falseby default and checking the underlying element at the target coordinates gives real human-in-the-loop safety before anything irreversible goes live.Resetting React's
_valueTrackeralongside native prototype setters insmart_fill_formalso fixes one of the most annoying silent failure modes in browser automation, where.value = "..."looks completely fine in the DOM untilblurorsubmitwipes the controlled state clean.To answer your question on what breaks most often in daily workflows: portaled dropdowns and virtualized comboboxes (like Radix or Headless UI menus that mount their listbox options at
document.bodyinstead of inside the form container). When the trigger is inside a nested frame or card and the options render at the root, or only mount while focused—scoped snapshots andonlyChangeddeltas can easily miss the newly mounted option nodes.The 48 KiB PDF number is the one that made me wince — 44,728 tokens just to hand a file to the model, versus 85 for a disk path. That's not an optimisation, that's a whole failure mode hiding in plain sight. Haven't run turbo-browser-mcp myself, but the upload fix alone would've saved me a few very long afternoons.
Thanks. That one surprised me too when I measured it. The server only passes the file path to Playwright's setInputFiles, so the bytes never go through the model at all.
On the open question about SUBMIT_LABEL_RE misclassifying buttons: I moved that check to the request layer a while ago and never looked back. Continue in a Workable wizard and Submit on the last step both sit on a submit button, but only one ends in a URL change or a mutating fetch. If you record the URL and in-flight requests for about a second after the click, draft and save buttons give themselves away: they fire an XHR and stay on the same page. Labels lie in every language, so classifying by aftermath sidesteps the naming problem entirely.
For the portal dropdowns, keyboard events saved me more than pointer ones. Focus the combobox and send ArrowDown then Enter. react-select and Radix open their menu on focus anyway, so you never have to wait for the option list to mount, and react-window stays somebody else's problem.
Took most of this. Since today a click that starts a POST, PUT, PATCH or DELETE within 800 ms comes back with possibleSubmit plus the method and host. It reports and doesn't block, and the label check stays as the first gate. The first false positive was an ad beacon firing on dev.to's own Publish button, so analytics hosts are excluded now.