The browser is where agent work starts to feel useful—and where a vague idea of control becomes dangerous. An agent can open a page, read it, click through a flow, and fill ordinary fields. Then a login page, a 2FA challenge, or a payment form appears. At that moment I do not want a polite convention saying “the human is in charge.” I want the runtime to enforce it.
That is why I made APX stop reading a tab when I take the browser wheel. A handoff is not a message to the agent. It is a control-state change that blocks agent calls.
This sounds like a small implementation detail. It changed how I think about agent browser work.
Watching is not the same as controlling
APX V2 can give an agent an embedded browser tab. On desktop, I can see that tab while the agent works. The agent operates through a narrow set of browser tools: it opens and navigates tabs, takes an accessibility snapshot, reads visible text, clicks element references from that snapshot, and types into a referenced field.
That visible surface matters, but it is not enough. Seeing an agent move a cursor does not answer the important question: what happens when I intervene?
A weak answer would be “the agent should wait.” That leaves too much to timing and interpretation. A model might already be reading the page while I am entering a one-time code. A queued click might run after I decide to take over. A screenshot taken during a login flow can reveal information I intended to keep inside my own browser session.
So APX tracks control on each tab. A tab can be idle, controlled by an agent, in a temporary handoff, or controlled by the user. When I choose Take Control, the desktop host changes that tab to user. Every subsequent agent attempt to read or operate it fails with paused_by_user. This applies before a page read, not only before a click.
That last part is the point. A browser is both an action surface and an observation surface. If the human is typing a password, a 2FA code, card data, or private account details, blocking clicks alone is incomplete. The agent should not be able to inspect the page state either.
Login is a handoff, not a chat request
I also did not want an agent to ask for credentials in conversation. The browser tool exposes a browser_handoff action for login, password, 2FA, or CAPTCHA. It records a short reason, marks the browser card as handoff, and waits for the owner to finish for up to five minutes.
The agent gets only one of two answers: the owner finished, with the current URL; or the owner did not finish in time, so the agent stops. It does not receive the password, the code, or a stream of the human’s interaction.
This is intentionally less clever than trying to automate every authentication path. Auth flows are different across sites, often intentionally hostile to automation, and frequently involve secrets or identity decisions that do not belong in an agent transcript. A visible handoff gives the person one clear job and gives the agent one clear boundary.
There is a second, separate path for a site where I am already signed in through my own browser. APX can ask me to approve copying cookies for a specific domain into an APX browser profile. That operation is human-only. The runtime can inspect source names and domain counts, but cookie values stay on the desktop. The agent cannot turn this into a general “read my browser” permission.
Keeping these paths separate matters. Logging in myself is a direct interaction. Importing an existing session is a scoped transfer with a specific domain. Neither should silently become permission for unrelated tabs or tools.
Separate profiles prevent accidental shared steering
Control alone is not the full boundary. APX gives agents browser profiles and checks ownership before an operation. An agent may touch tabs from its own profile, but it cannot drive a tab assigned to another agent. If another agent is currently controlling a tab, APX rejects the request rather than letting two runs steer one page.
That is less dramatic than a large security feature, but it removes a surprisingly common source of ambiguity: a tab is not merely a URL. It has a profile, cookies, current state, and an operator. Treating it as shared global state would make a browser session hard to explain after anything unexpected happens.
The same design shapes the normal tool flow. Clicks reference an element from the latest accessibility snapshot. Typing is an external action and requires confirmation after untrusted page content has been read. Browser text is marked untrusted. The agent is pushed toward checking the new page state before claiming that a form was submitted or a message was sent.
None of that makes browser automation safe by declaration. It gives the system places to stop, ask, and show evidence.
The design rule I am keeping
My original instinct was to make the browser feel seamless: agent opens page, I help at login, agent continues. The better rule became more precise: seamless is fine only when ownership stays obvious.
A good handoff has four properties:
- The human can take a tab without negotiating with the agent.
- Agent reads stop as well as agent actions.
- Sensitive input stays out of chat and tool output.
- Returning control is explicit, so the agent does not guess when work may resume.
This adds state to the runtime. It also removes the kind of invisible state that causes trouble: an agent that assumes it still owns a page while a person believes they have taken over.
I am building APX around the idea that agents should be capable without becoming opaque. In a browser, the practical version of that idea is simple. If I take the wheel, the agent waits—and it cannot watch me type.
Top comments (0)