DEV Community: Ritabrata Maiti

How I built Browy, an AI agent that lives in your browser

Ritabrata Maiti — Mon, 19 Oct 2026 19:58:27 +0000

When you send Browy its first message, Chrome puts up a bar saying that Browy
started debugging this browser. No extension can hide it, Browy included. It
appears because Browy drives your tabs through chrome.debugger, which lets
an extension speak the Chrome DevTools Protocol, the same protocol your own
DevTools use.

I've grown to like that bar, because it describes Browy better than I usually
manage to. Browy is an AI agent that lives in your browser, and I mean that
literally. It works in the Chrome, Edge or Brave window you already have open,
signed in to everything you're signed in to.

You chat with it in the side panel, or type at it in a DevTools panel, and it
reads pages, clicks, types, fills forms and runs JavaScript in your tabs while
you watch. The models come through your own GitHub Copilot subscription. It's
open source under Apache-2.0, and I've
been building it on my own since April. This is the long version of how it
works, with the bugs left in.

What happens when you hit send

Three programs on your computer take part, and they talk over pipes.

The first is your browser, running Browy's extension: a side panel for
chatting, a DevTools panel for people who'd rather type commands, and a
settings page. None of them talk to the model. They talk to the extension's
service worker, which keeps one connection open to the second program. The
frontend architecture page walks through the extension file
by file.

That connection is Chrome's native messaging. Browy's installer registers a
manifest under the name com.browy.host, and when the service worker calls
chrome.runtime.connectNative('com.browy.host'), Chrome starts the host and
wires the extension to its stdin and stdout. Each message is a four-byte
length followed by JSON. The open port also keeps the service worker alive,
which Manifest V3 would otherwise put to sleep.

The host is a Node program, and it's where the agent lives. It wraps GitHub's
Copilot SDK, which starts the third
program, the Copilot CLI, in headless
mode and talks JSON-RPC to it over another pair of pipes. The CLI makes the
model calls on your Copilot subscription and stores each chat. Browy's code
covers the browser half: the tools, and the picture of the page the model gets
on every turn. The backend architecture page has the
details.

That picture is rebuilt each time you send a message. The host appends a
<browser_context> block to your text, with the active tab's title and URL,
the viewport, the scroll position and any text you've selected, followed by a
snapshot of the page and the agent's last few tool calls. Your words go first,
so the chat's summary in the history list shows what you asked.

When the model calls a tool, the SDK runs a handler in the host. If the tool
needs the page, the host sends a cdp.send message back up the native port,
naming a DevTools Protocol method and its parameters. The service worker picks
the tab, attaches chrome.debugger if it isn't attached yet, runs the command
and sends the result back down. Network and console events travel the same
route. The host coordinates, and every action happens in your browser.

The DevTools panel is the same agent with a terminal in front: slash commands,
a model picker, and a /js mode that runs each line in the inspected page
without calling the model. Each panel works on the tab it inspects. The
DevTools CLI page covers how the REPL is built.

The page as a numbered list

Ask a model to click something by CSS selector and it will happily invent one.
Even a correct selector only says where an element sat at one moment, and a
re-render can move it.

So Browy hands the model a numbered list. One Runtime.evaluate call walks
the page, shadow roots and same-origin iframes included, and keeps each
element that's interactive, visible and within 2,000 pixels of the viewport.
It sorts them in reading order, drops any that sit almost entirely inside
another, stops at 150 and stamps each one with a data-browy-id attribute.
The model sees something like this:

[1]<a href=/expenses>My expenses</a>
[4]<input type=text name=title placeholder="e.g. Client dinner" aria-name=Title />
[5]<input type=number name=amount aria-name="Amount (SGD)" />
[6]<input type=checkbox name=billable unchecked aria-name="Billable to client" />
[*7]<button>Attach receipt</button>
[~12]<button type=submit>Submit for approval</button>

A ~ marks an element that's off screen, which the click tools scroll into
view first. A * marks one that wasn't in the previous snapshot, which is how
the model notices that a dialog opened or a new part of the form appeared. The
aria-name is what a screen reader would announce, usually the label you'd
read yourself.

For click_index(12), the host finds the stamped element and sends real mouse
events to its center with Input.dispatchMouseEvent. Before and after, it
records the URL, the title, a hash of the visible text, the number of open
dialogs and the element's ARIA state. Half a second after the click it
compares them and reports what changed, with flags like urlChanged,
domChanged, modalAppeared and elementStateChanged. If nothing changed,
the reply says so and suggests what might be in the way, such as an overlay or
a page that's still rendering.

I didn't invent this. The indexed snapshot is modeled on
browser-use's DOM service and
serializer, down to the rule for dropping nested elements, and I'm grateful to
that team for working it out in the open.

Turning tools off

Browy has 54 browser tools, from click_index and fill_form to
get_network_requests and replay_request. Most have a switch in Browy's
settings. Turning one off removes it from two places, the tool definitions
sent to the model and the SDK's availableTools allowlist, so the model never
learns it exists.

Two of them run code the model writes. evaluate_js runs JavaScript in
the page, for the jobs the indexed tools can't express, and it's on.
run_script runs Node.js on your computer, so it's off until you switch it
on. It was on by default until version 0.1.10, which was the wrong default
for a tool that reaches your machine.

The Copilot CLI also brings tools of its own, like bash and web_fetch,
which act on your computer instead of the tab. Browy keeps those out of the
allowlist until you switch them on, one at a time, the same as run_script.

This matters because Browy reads pages written by strangers while it's signed
in as you. Any page can carry text aimed at the model, and Browy's security
model assumes some pages will. The switches decide what a page like that could
talk the agent into.

Why your own browser

Plenty of agents start a fresh browser, often on a server, with an empty
profile that isn't signed in to anything. For public pages at scale, that's
the right design. Most of my repetitive work happens on pages that only exist
because I'm signed in, though: the pull request I'm reviewing, my Gmail
filters, an expense claim, a journal's submission portal. A fresh browser sent
to any of them lands on a login screen. I wrote about the form-filling side of
this in an earlier post.

Browy sidesteps that by staying in your browser. Ask about the pull request in
front of you, and it reads the pull request in front of you, in the tab where
you're already signed in. Your logins stay where you made them, in your
browser profile, next to your cookies and your password manager. When a page
wants a password or a 2FA code, the agent is meant to call await_user, which
pauses and hands the job to you. That tool's own description tells the model
to use it instead of typing into password fields.

Your chat history is stored on your machine too, under ~/.copilot/. What
leaves it is what the model needs to see: your messages and the page content
the agent reads, sent through your Copilot subscription under GitHub's terms,
the same way the Copilot CLI sends them from a terminal.

The trade is scale. Browy works in one browser on one machine, and only when
you ask. It won't run a hundred jobs overnight, and I'm fine with that.

The second install, and other costs

Browy needs two programs that live outside the browser, Node for the agent and
the Copilot CLI for the model. An extension can't install those, and native
messaging only reaches a program that's already registered on the machine. So
Browy installs in two steps: the extension from the Chrome Web Store, then a
one-line command in PowerShell or a terminal. The side panel shows the command
for your system, with a copy button, and picks up the host by itself once it's
registered. In my testing that took 3.6 seconds. The
install guide has both commands.

Having two halves also means two update schedules: the Web Store updates the
extension, and you update the host, which the side panel prompts you to do
when it falls behind.

Then there's size. The Windows download is about 140 MB, most of it the
Copilot CLI. In August I cut a 106 MB native binary out of every download,
because the SDK ran the CLI as a Node script and nothing ever executed the
binary. In October the CLI itself moved into that same package as a native
executable. Switching to it cut the CLI's startup on Windows from 5.8 seconds
to 0.6, and pushed the Windows download from about 80 MB back up to about 140.
You download Browy once and wait for it to start far more often, so I'll take
that trade.

Browy is also Chromium only: Chrome, Edge and Brave. All of the above depends
on chrome.debugger, and Firefox and Safari don't give extensions an
equivalent.

And there's the debugging bar from the top of this post. It stays up while
Browy holds a tab, and clicking Cancel on it detaches the agent.

The system prompt that never arrived

In October I noticed that Browy still bundled version 1.0.36 of the Copilot
CLI, while the current release was 1.0.95, the native binary from the last
section. Updating meant checking each SDK call Browy makes against the new
CLI, and that turned up a bigger problem.

The SDK builds its session.create request from named fields, and the system
prompt goes in one called systemMessage. Browy had been passing its prompt
as systemPrompt since the first commit. Nothing complained. JavaScript
doesn't mind an extra key on an options object, and the SDK never looked for
that one. For five and a half months, every Browy chat ran on the Copilot
CLI's default instructions plus Browy's tool list. The instructions that tell
the model it's Browy, how to use the indexed tools and when to hand a sign-in
back to you never reached it.

The SDK's TypeScript types would have caught it, since SessionConfig has a
systemMessage field and no systemPrompt. But Browy loads the SDK with
require() and holds the client as any, so the compiler never compared the
two.

The part that stings is a comment about thirty lines higher in the same
function, shipped in the very first release. It notes that an earlier
includeDefaultTools: false flag had never been a real SDK option either. I
had caught this exact kind of bug once and written it down, and the second one
sat thirty lines below my note.

I suspect Browy mostly behaved anyway because the tool descriptions did reach
the model, and a lot of the guidance lives there.

The fix was one line, systemMessage: { content: SYSTEM_PROMPT }, with a
catch. The prompt I'd written in April told the model to fill a form and then
submit it, and once the prompt started arriving, that rule would have taken
effect for the first time. So the same release rewrote it. Browy now submits
only when you ask it to submit, send, post or buy, and otherwise stops and
tells you the form is ready to check. Asked its name, it finally says "I'm
Browy."

The chats that were on disk all along

In May I removed my development copy of the extension to test a clean
install, the way a new user would get it. When I opened the new copy, my older chats were
missing from the history list. Uninstalling an extension wipes its storage, so
I assumed they were gone.

They weren't. The Copilot SDK had written every turn of every chat to its
session store under ~/.copilot/, on my own disk, and the host already knew
how to list and read those sessions. It just didn't recognize the old ones as
Browy's. It looked for session ids starting with sp- (side panel) or dt-
(DevTools), and my older chats predated that scheme.

Version 0.1.4 recognizes a Browy session by any of three marks: the id prefix,
the working directory every Browy session is created with, or the
<browser_context> block Browy adds to messages. I widened the rule for
listing chats but left the narrow one in charge of automatic cleanup, so
nothing of uncertain origin could be deleted by accident.

Chats looked lost once more in August, when "No past chats yet" turned out to
be a four-second timeout on a host that needed between 7.7 and 40 seconds to
answer. Both times, the chats were fine. That's an upside of a local agent I
hadn't planned for: when the interface said my chats were gone, I could open
the folder and check.

The form it said it had filled

On every turn, Browy shows the model its own last five tool calls in a
<recent_actions> block, with each tool's arguments, whether it worked, how
long it took and the start of its result. That's how the model checks whether
its last click did anything. The same idea shows up in browser-use and in
research agents like ReAct and SeeAct.

The list lived on the host's agent object, and one agent serves every chat. So
a new chat's first message arrived carrying the previous chat's last five
calls.

I found out on October 13, while filming the demo video that's now on Browy's
homepage. In a brand-new chat, I asked Browy to fill the same form it had just
filled in another chat. It replied that the form was filled. It hadn't called
a single tool.

The model wasn't making things up, exactly. It was reading. Its context showed
fill_form coming back ok moments earlier, in what it had every reason to
think was its own chat, and it believed the record.

The fix keys the list by chat, in
a small module of its own,
and a new test fails if a fresh chat ever starts with another chat's tool
calls again. Version 0.1.9 shipped the same day.

The demo video came out of that same filming session. It runs 65 seconds in
real time: one sentence typed into the side panel, a paper open in one tab,
and a 26-field paper submission form in the other. By the end every field is
filled, and the submit button is still waiting for me.

All posts Index of every post on the Browy blog.
Architecture: native host How the host wraps the Copilot SDK and routes tool calls back to the extension.

AI browser agents compared: Browser Use, Skyvern, Nanobrowser, Browy

Ritabrata Maiti — Mon, 19 Oct 2026 19:58:25 +0000

I built Browy, the first of the seven agents on this page.

All seven can read a web page and click through it for you. The differences
that matter are whose browser they drive and who pays for the model.

Seven browser agents side by side

	Best for	Where it runs	Whose browser and logins	How you pay for the model	Open source	Interface
Browy	Help in the tabs you're signed into, next to DevTools	Inside your browser, with a small local host	Your own Chrome, Edge or Brave, with your sessions	Your GitHub Copilot plan, Free included	Yes, Apache-2.0	Side panel chat and a DevTools panel REPL
Browser Use	Agents you script and run at scale	Your machine or Browser Use Cloud	A fresh browser, your Chrome profile, or a cloud browser	Your own API key, or Browser Use credits	Yes, MIT	Python library, CLI, cloud API
Skyvern	Weekly jobs behind a login	Your machine (pip or Docker) or Skyvern Cloud	Its own browser, or your running Chrome	Your own key when self-hosted; Cloud has free and paid plans	Yes, AGPL-3.0	Web app with a workflow builder, SDKs, API
Nanobrowser	Any model provider, local models included	Inside your browser	Your Chrome or Edge	Your own API keys, or a local model through Ollama	Yes, Apache-2.0	Side panel chat
Stagehand	Browser steps inside your own code	Your machine or Browserbase	A browser it launches or attaches to, or a cloud one	Your own key, or their model gateway	Yes, MIT	SDK for Python, Go and TypeScript
Claude in Chrome	Claude users, especially for scheduled tasks	Inside Chrome	Your Chrome	A paid Claude plan	No	Side panel, Claude Code, Claude Cowork
ChatGPT	Handing off a task and walking away	OpenAI's cloud, the desktop app, or Chrome	A cloud browser, the app's browser, or your Chrome	Your ChatGPT plan; the cloud browser needs a paid plan other than Go	No	ChatGPT on web, desktop and mobile

Browy, the agent I built for your signed-in browser

Browy works in your own Chrome, Edge or Brave, in the tabs you already have
open and the sessions you're already signed into. Ask it to review the pull
request in front of you or set up a Gmail filter, and it starts from the page
you're on, not from a login screen. It reads the page's accessibility tree and
numbers each control, then clicks and types by number. When it fills a form,
it stops before submitting unless you ask it to submit.

It runs on the GitHub Copilot plan you already have, the same one your editor
and terminal use. You sign in with GitHub, then choose from that plan's
Claude, GPT and Gemini models and switch between them mid-chat with /model
or in Settings. Copilot Free works too.

It's also the only agent on this page with its own panel inside Chrome
DevTools. Open DevTools on any tab and the Browy panel gives you a REPL with
slash commands and a JavaScript mode, next to the Network and Console tabs you
already use. The agent can read that tab's network log and console itself, so
you can ask which API call is slowest and get the URL, status and timing back.

Browy is Apache-2.0, and the agent loop is one file you can read. Since
v0.1.10, the tools that reach your machine rather than the page, from bash
to Browy's own run_script, stay off until you switch them on in Settings.

Getting started takes three steps: add the extension, run one install command,
and sign in with GitHub. It suits developers who already pay for GitHub
Copilot and want an agent in the tabs they're signed into, including the ones
for sites they're building.

Browser Use, for agents you run from code

Browser Use is the open-source project most people mean when they say
"browser agent," and it earned that spot. Magnus Müller and Gregor Žunič
built it as a Python library: you hand an Agent a task and a model, it opens
a browser, and it works through pages until it has an answer. It now comes in
three forms: the MIT-licensed library, a command-line tool that gives a coding
agent like Claude Code or Codex control of your browser, and Browser Use
Cloud, which runs the agent and the browser for you.

It suits developers who want an agent inside their own program, with custom
tools and structured output, running whichever model they prefer: OpenAI,
Anthropic, Browser Use's own BU2 model, or a local model through Ollama.

Where it shines is work that runs while nobody is watching. The library runs
inside your own program or on your own servers, and when you need many
browsers at once, Browser Use Cloud sells them at $0.02 an hour with proxies
and CAPTCHA handling built in. The library can also open your real Chrome
profile with its logins (you may have to quit Chrome first).

If you'd rather ask for help in the tabs you're already signed into than write
a script, that's the job Browy is built for, and it runs on the Copilot plan
you already pay for. Browser Use's code is on GitHub at
browser-use/browser-use.

Skyvern, for workflows you run every week

Skyvern drives websites with language models and computer vision, so it can
work through a site it has never seen. Its workflow builder chains steps into
something you can rerun: log in, filter the invoices, loop over them, download
each one. You can run it yourself with pip or Docker and your own model key,
or use Skyvern Cloud, which starts with a free tier of 5,000 credits and has
paid plans from $29 a month. The code is AGPL-3.0, except for the anti-bot
features, which live only in the cloud.

It's built for teams with the same job to do behind a login, again and again.
The examples in its README include invoice portals, job applications,
government forms and insurance quotes.

It shines at repetition. Skyvern handles 2FA codes, stores credentials, solves
CAPTCHAs in its cloud and connects to Zapier, Make and n8n. It can also drive
your own Chrome over remote debugging, with your cookies and logins.

For one-off work in the tabs you're already signed into, Browy is the lighter
fit, because you open it beside the page and ask. Skyvern's code is on GitHub
at Skyvern-AI/skyvern.

Nanobrowser, the extension that takes any model

Nanobrowser is the closest open-source relative of Browy. It's an Apache-2.0
extension with a chat in the side panel, and it runs a pair of agents in your
browser, a planner and a navigator, each of which can use a different model.
Its README credits Browser Use as one of the projects it builds on. Version
0.2.0 came out on October 2.

It suits people who want an agent in the browser they already use, signed in
where they're already signed in, and who'd rather choose the model provider
themselves.

Its strength is model choice. It works with OpenAI, Anthropic, Gemini,
DeepSeek, Grok, OpenRouter and more, and with local models through Ollama, so
a page's text never has to leave your machine. You add it from the Chrome Web
Store, paste an API key into its settings and pick a model for each agent. It
officially supports Chrome and Edge.

If you already pay for GitHub Copilot, Browy gives you the same kind of side
panel with that plan's models behind it, and adds a panel inside DevTools.
Nanobrowser's code is on GitHub at
nanobrowser/nanobrowser.

Stagehand, the SDK from Browserbase

Stagehand is here because it's the main alternative to Browser Use when you're
writing code rather than chatting. It's an MIT-licensed SDK from Browserbase
for TypeScript, Python and Go, and it wraps a Playwright-style browser in a few
AI calls. act performs an action you describe in plain English, observe
finds elements and hands back real selectors, and extract pulls data out
against a schema you define. It can launch Chrome on your machine, attach to a
Chromium browser you already have running, or use Browserbase's cloud
browsers.

It's a good fit for developers building a product or a pipeline that needs a
browser, who want control over single steps rather than a whole agent loop.

It's at its best as a building block. It lives in your codebase, so it gets
tested and deployed like the rest of your code, and on Browserbase it can cache
repeated actions. The login example in its README keeps passwords away from the
model by filling fields through selectors. Browserbase has a free plan, and its
paid plans start at $20 a month.

If you want an agent to use rather than one to build, Browy works in the tabs
you're signed into, on the Copilot plan you already pay for. Stagehand's code
is on GitHub at
browserbase/stagehand.

Claude in Chrome, Anthropic's agent in your own Chrome

Claude in Chrome is here because it's the closest thing to Browy from a model
company: an extension that works in your own signed-in Chrome from a side
panel. It reads the page you're on and works through it for you, forms
included, and it also serves as the browser for Claude Code and Claude Cowork.
It's available on every paid Claude plan, from Pro to Enterprise, and it
supports desktop Chrome only.

It suits people who already pay for Claude, and developers using Claude Code
who want it to check its work in a real browser, console errors and network
requests included.

Where it shines is unattended work. It keeps working in the background while
you use other tabs, and it can run tasks on a schedule, daily or weekly. Its
classic side panel can also learn a workflow you record once. For control, you
get site-by-site permissions and a safety check before it acts, and Team and
Enterprise admins can switch it on or off for everyone.

If your subscription is GitHub Copilot rather than Claude, or you work in Edge
or Brave, Browy gives you the same kind of side panel in your own signed-in
browser, with Claude among the models your Copilot plan offers. Anthropic's
page for it is Claude in Chrome.

ChatGPT's cloud browser and Chrome extension, from OpenAI

ChatGPT is here because OpenAI has retired its Atlas browser and moved browser
work into ChatGPT itself, which is where a lot of people will try a browser
agent first. On paid plans other than Go, ChatGPT Work can run a task in a
cloud browser on a separate computer. That task keeps going after you leave
the conversation, and it pauses when it needs you to sign in or confirm
something. The ChatGPT desktop app for macOS and Windows has a browser of its
own, and OpenAI's ChatGPT extension for Chrome lets the app work in your
existing Chrome profile and signed-in sessions. The extension supports Chrome
only and works together with the desktop app.

It suits people who already live in ChatGPT, and anyone who wants to hand off
a task and come back later to a result.

It shines at delegation. You can start a cloud browser task from your phone,
and it carries on without you, and you type passwords into a sign-in form that
the model never sees.

If you'd rather watch the work happen in your own tabs, next to DevTools,
Browy is built for exactly that, and it runs on the Copilot plan you already
pay for. OpenAI explains the cloud browser in
Using cloud browser in ChatGPT,
and the extension is
ChatGPT on the Chrome Web Store.

How I'd pick

If you already pay for GitHub Copilot and want an agent in the tabs you're
signed into, start with Browy. That's who I built it for, and it's the only
agent here that also lives inside DevTools.

The others are built for different jobs:

For agents that run unattended and at scale, from your own code, use Browser Use, or Stagehand if you'd rather assemble smaller building blocks.
For workflows behind a login that someone reruns every week, with 2FA and stored credentials, look at Skyvern. If you pay for Claude, Claude in Chrome can run tasks on a schedule.
For local models, try Nanobrowser.
To hand a task off and walk away, use ChatGPT's cloud browser.

Here's Browy filling a 26-field form from a paper in another tab, in real time:

<iframe
src="https://www.youtube.com/embed/bnellxgwJwY"
width="710"
height="399"
style="width: 100%; height: auto; aspect-ratio: 16 / 9;"
allowfullscreen
loading="lazy">

One sentence, in real time: Browy reads the authors from a paper in another tab, fills all 26 fields, and stops before submit. Try the same form.

The install guide takes you from here to your first chat in about
two minutes.

Your browser already knows who you are. The form does not.

Ritabrata Maiti — Tue, 13 Oct 2026 14:34:34 +0000

The fifth journal portal of the year wants the author list again. Same eight
names, same order, same affiliations, same ORCID identifiers, typed into eight
separate pairs of first-name and last-name boxes because the form was designed
in 2011 and has been load-bearing ever since. You have the data. It is in the
last submission, which is in the portal you are looking at, three clicks away
in a tab you are already signed into.

This is the shape of most form work. Not one enormous form once, but the same
medium-sized form over and over, on a site that knows exactly who you are, with
data that already exists somewhere the browser can reach.

It is also the thing browser agents are worst at, and the reason is structural.

Two different problems wearing the same word

"Automate this form" splits into two problems that have almost nothing in
common.

The first is filling a form on a public page. A signup, a contact form, a
checkout. Anything can do this. Playwright has done it for years, headless
browsers in the cloud do it at scale, and it works because the page is the same
for everybody.

The second is filling a form that only exists because you are signed in. A
manuscript submission. An expense claim. A CRM record. A conference review. A
grant portal. Here the page is assembled from your session: your saved drafts,
your institution's custom fields, the co-authors you added last time, a
half-finished record from three weeks ago.

Almost every browser agent solves the first problem and quietly cannot touch
the second. They launch a fresh browser, usually in a data centre. That browser
has no cookies, no session, no extensions, and no history. Pointed at a portal,
it lands on a login screen. You can hand it credentials, which means putting
your institutional password into an agent's config, and then it meets your
second factor and stops.

The gap is not intelligence. A model that could reason its way through the form
still never gets to see it.

Running inside the session you already have

Browy takes the other route. It is a Chromium extension, so it drives the
browser you are actually using: your profile, your cookies, your logged-in
tabs, your password manager, your 2FA already satisfied an hour ago. There is
no credential handoff because there is nothing to hand off. The page the agent
sees is the page you see.

That choice costs something, and it is worth naming. Browy cannot run a hundred
of these overnight while you sleep. It is attached to a real browser on a real
machine, so it is bounded by that. If you need volume against public pages,
Browser Use or Skyvern will serve you better and I would point you at them.
What this buys instead is access to the pages that only exist after login,
which is where the repetitive work actually lives.

Indices, not selectors

The second structural problem is how an agent refers to a field.

The common approach is CSS selectors, either written by hand or generated by
the model. Both are brittle in the same way. A selector like
div.form-group:nth-child(7) > input describes where something sat in the DOM
at the moment it was written, and a React re-render or a conditionally shown
field moves it. Stagehand's answer to this is self-healing selectors, which is
a real approach to a real problem.

Browy does something different. Before acting, it builds an indexed snapshot of
the page from the accessibility tree, the same structure a screen reader
consumes, and gives every interactive element a number:

[12]<input type="text" name="author_1_given" aria-label="Given name">
[13]<input type="text" name="author_1_family" aria-label="Family name">
[14]<select name="author_1_role" selected="Author" options=[Author, Editor, Reviewer]>
~[27]<button>Add another author</button>

The model works in those indices. type_index(13, "Maiti") targets the element
that snapshot resolved, not a path guessed from HTML the model half-remembers.
Elements below the fold are marked with a tilde and scrolled into view
automatically. Elements that appeared since the last snapshot are flagged, which
matters more than it sounds: when a form reveals a conditional section, the new
fields are the interesting ones and the agent is told which those are.

The accessibility tree also carries the labels. Not name="f_7" but "Given
name", because that is what the browser exposes to assistive technology. A form
that is accessible is, by accident, a form that is legible to an agent.

Reading the form before writing to it

The part that changes the economics is doing the whole form in one pass.

extract_form returns the structure: every field, its label, its type, whether
it is required, its current value, and for a select, the options it will accept.
That is one call and it produces something the model can plan against rather
than discover field by field.

Then fill_form takes an array and fills all of it in a single call:

[
  { "index": 12, "value": "Ritabrata" },
  { "index": 13, "value": "Maiti" },
  { "index": 14, "value": "Author" },
  { "index": 18, "value": "0000-0002-1825-0097" }
]

One tool call, not twenty. This matters for a boring reason: every tool call is
a round trip through the model, and a thirty-field form filled one field at a
time is thirty round trips of latency and tokens, with thirty chances for the
page to shift underneath. Reading the schema once and writing once collapses
that.

Type detection is handled per field, so checkboxes, radios and selects do not
need separate calls. submit_form is deliberately separate, and it reports
whether the URL changed afterwards, because "did that actually submit" is a
question worth answering rather than assuming.

Where it still goes wrong

Three places, honestly.

Custom widgets that are not really form controls. A rich-text editor built on a
contenteditable div, or a date picker that is fourteen nested divs and no
<input>, will not appear as a fillable field. The escape hatch is
evaluate_js, which runs in page context and can do anything the page can do,
but at that point you are writing code rather than asking for something.

Multi-step wizards where the next step depends on server validation of the
current one. Browy handles these, but each step is a fresh snapshot and a fresh
round trip, so a six-step wizard feels slow in a way a single page does not.

And uploads still need a real file path. upload_index bypasses the OS file
dialog, which is the hard part, but you have to know where the file is.

The thing that made me build it

The honest origin is that I was uploading a manuscript, for the fourth time
that month, to a portal that had all of my previous submissions stored, and
asking me to retype every author anyway. The data was sitting in the same
browser session. It just had no way to move six inches across the screen.

That is not a hard problem in the sense of requiring a better model. It is a
plumbing problem: something that can read the page as structure rather than
pixels, act on stable handles rather than guessed paths, and do it from inside
the session where the data already lives.

Browy is open source, Apache-2.0, and runs on a GitHub Copilot subscription you
probably already have, so there is no second bill for the tokens. If you want to
see the form tools specifically, they are in the
tool reference, and
the install guide covers the two-part
setup.

All posts Index of every post on the Browy blog.
Tool reference Every browser tool the agent can call, including extract_form and fill_form.

Every $20 AI subscription costs about $100 to serve. The bill is coming.

Ritabrata Maiti — Thu, 24 Sep 2026 08:33:39 +0000

A piece called
AI subscriptions are a ticking time bomb for enterprise
made the rounds this week, and the headline is the right shape of the
story. Every major AI lab is running an industry-wide loss-leader at a
scale that does not really have a precedent. Your company's $20 Claude
Pro seats and $20 ChatGPT Plus seats are being served at something
like five times the cost the lab is collecting for them, and that
arrangement is not stable.

The price tag has not moved in three years. The product has changed
completely.

The unit economics

A Claude Pro seat is $20 a month and gives you Sonnet 4.6, Opus 4.6,
file creation, code execution, web search. On the API, those same
models cost $3 input and $15 output per million tokens for Sonnet,
$5 input and $25 output for Opus. A knowledge worker running Claude
for a few hours a day, uploading documents, drafting reports, easily
moves through enough tokens that the API-priced equivalent of that
seat sits somewhere between $200 and $400 a month.

Microsoft was reportedly losing more than $20 a month on every GitHub
Copilot seat. Power users were hitting $80. One widely-cited analysis
found Anthropic was burning about $8 of compute for every $1 of
subscription revenue. The $20 sticker has been frozen since 2022 and
the models in that window picked up image generation, code execution,
voice, agentic reasoning, web search, and a generational capability
jump. The number stayed put.

That is the whole story. Everything from here is mechanism.

What the subsidy was buying

Cheap inference at consumer prices, deployed broadly to enterprises,
was buying integration depth. The labs were not trying to make money
on the seat. They were trying to make sure the seat existed in the
first place, then make sure the company's marketing draft and the
engineer's pull request review and the analyst's quarterly summary
all happened through it. Once those workflows are load-bearing, the
price can move. The dependency is the asset.

You can see this in the language coming out of OpenAI. Nick Turley,
their VP of product, described the subscription pricing as something
they
stumbled into,
and has floated phasing out unlimited plans entirely, comparing them
to "unlimited electricity." Sam Altman said publicly that OpenAI now
needs to become "an AI inference company," which is the polite version
of admitting the consumer subscription was a customer-acquisition line
item, not a P&L.

The KPMG Q1 2026 pulse has U.S. organizations projecting average AI
spending of $207 million over the next twelve months, roughly double
the year before. A Goldman Sachs survey of large companies has most
of them overrunning their AI budgets by orders of magnitude.
Chandrasekaran, who runs AI and data at KPMG North America, told
Marketplace
the quiet part: "Even a quarter or two ago nobody bothered about LLM
consumption costs." It is the bother stage now.

Agents are what broke the math

The reason the subsidy held as long as it did is that AI was a chatbot.
You typed, it answered, you read the answer, repeat. A normal session
was a few thousand tokens. Heavy use ran into the tens of thousands.
At those volumes, $20 a seat was uncomfortable for the lab but not
catastrophic.

Agents do not look like that.

A Claude Code session runs autonomously for an extended period. It
reads files, writes files, runs commands, looks at the output, decides
what to do next, repeats. Users have been
exhausting five-hour rate-limit windows in under ninety minutes.
Multiple agents in parallel on a single project multiply that. A
developer running three or four concurrent coding agents is consuming
something close to an order of magnitude more tokens than the same
person in chat, and the subscription price on the seat is unchanged.

GitHub took the obvious next step. On June 1, 2026, Copilot
moves to usage-based billing,
specifically because the flat-fee model collapsed under agentic
workloads. The announcement spelled it out: agentic usage is becoming
the default, the inference demand is qualitatively different, the
pricing has to follow.

Everyone else will do the same thing on a delay.

The enterprise position

This is where it gets ugly for organizations that have not done the
work.

Over the past two years, thousands of companies have woven $20 AI
subscriptions deep into operations. Marketing drafts copy through
ChatGPT Plus. Engineering writes and reviews code through Claude Pro.
Research synthesizes documents. Customer success summarizes tickets.
Finance models scenarios. The line items are budgeted at subsidized
prices because that is what the bill currently says. The actual cost
of the same workloads at API rates, if the lab were charging it, is
fifteen to twenty times higher.

When prices adjust, two things happen at once. The bill goes up, and
the workflows are already too embedded to rip out. The subsidy creates
the dependency, the dependency makes the price increase unavoidable.
That is the trap, and there is no clever way out of it.

The companies that survive this transition cleanly will be the ones
that did the bookkeeping. Track per-team token consumption. Know which
workflows are genuinely high-value and which are running Claude on
something a 2018 script could have done. Have a sense of which
subscriptions can move to per-seat billing if it has to, and which
ones become structurally expensive overnight.

The companies that don't survive cleanly will discover the bill in
the third week of whichever month the subsidy ends, with no time to
re-budget and no leverage to renegotiate.

What it means for agent tools

The shake-out hits the agent tool market harder than it hits the labs.
The labs are losing on the seat, but they own the inference. They can
move the price. They can change the plan. They can introduce
usage-based tiers and call them "for power users." The seat is still
there at the end.

The agent tool, the wrapper, the IDE plugin, the browser extension
that uses the lab's API or subscription on your behalf, is in a worse
position. If it has its own per-seat subscription, it is selling you
something whose underlying cost just doubled, and it has to either eat
that or pass it on. If it bills on its own meter on top of the lab's,
it is asking enterprises to swallow a usage line item that already had
no budget code.

The agent tools that survive the next twelve months will be the ones
that don't sell their own meter. Tools that piggyback on the
subscription the user already has. Tools that don't introduce a second
billing surface for finance to police. Tools whose cost curve is
shaped by the lab's pricing, not by the tool's overhead.

This is not a clever positioning argument. It is what happens to every
software-on-top-of-software market when the underlying utility starts
charging real prices. The tools that own the price stack survive. The
tools that resell the utility at a markup get squeezed.

A note on what I work on

I build Browy, an open-source AI agent that lives
in a Chrome side panel and a DevTools REPL. It drives the real browser
tabs you have open. The thing it does not have is its own subscription.
It uses your existing
GitHub Copilot subscription for
the model. The model call goes from your machine to GitHub, the answer
comes back, the rest happens locally.

When Copilot moves to usage-based billing on June 1, you pay GitHub
the new rate, the same as you would have anyway. Browy doesn't sit
between you and that bill. It doesn't add a per-seat charge of its
own. It doesn't run a metered tier on top. That is not a special
business decision on our end, it is the only decision that survives
the shake-out described above. The tools that try to live on top of a
collapsing subsidy by adding their own subsidy get squeezed twice.

That is most of the story I wanted to put down. The original piece is
here.
The 30-second video version is at the top of this post.

All posts Index of every post on the Browy blog.
AI slop killed the open-source bug bounty Earlier essay on the other end of the same arc: cheap inference, not yet expensive.

Google quietly declared war on the open web.

Ritabrata Maiti — Thu, 24 Sep 2026 08:32:53 +0000

At Google I/O this week, the Search team made AI Overviews the default
answer surface for English users. The
official keynote post
buries the change under a lot of "agentic" and "personal intelligence"
language, but the practical effect is small and definite: Search now
answers from a Google-hosted synthesis. The blue links are a fallback,
not the product.

This was the inevitable next step of a series of changes that started
with featured snippets and accelerated with AI Overviews. The
10 percent inaccuracy rate
on AI Overviews is now everyone's problem, because Overviews are
everyone's default answer.

The
piece on tante.cc
called this "declaring war on the web," and the wording is exactly
right. What follows is what I think the change actually is, the deal
that broke, and what to do about it.

The deal that built the web

Sites let Google crawl them. Google sent traffic back. Both sides
won. The crawler got fresh content; the site got readers, and the
readers had a clear path back to the source when they wanted to dig
deeper.

This contract held for about twenty years. It built the open web as
a publishing surface, the indie blog as a thing that paid rent, the
SEO industry, the long-tail of niche reference sites, and most of
what we now call "documentation on the internet." It was an
asymmetric trade, but both sides understood it. You give Google the
catalog. Google sends you the foot traffic.

AI Overviews quietly cancelled the trade.

The crawler still runs. The catalog still gets ingested. The output
on the user's screen is now a Google-synthesised answer, with the
sources demoted to a row of tiny chip-shaped citations the median
user does not click. The reader gets an answer, often correct,
sometimes wrong, and never has to leave the search results page.

What this is, named plainly

The framing in the original piece is sharp and worth quoting at
length:

Your work, your writing or art do matter a bit still: As (unpaid)
raw material for their synthetic text extruders. You get to work
for free so Google can have tight control on the flow of
information and make sure that the responses people get are in
line with what they need them to be.

This is the right frame. The web is not the product anymore. The
web is the raw material. The product is the synthesised view on top
of it, hosted on Google's infrastructure, branded with Google's UI,
and accountable to Google's editorial choices.

A useful analogy: the relationship between a wire service and a
newspaper. The wire produces the underlying reporting; the newspaper
picks, edits, and presents. The reader sees the newspaper. For a
long time the web was the newspaper. Google was the index, the table
of contents that pointed at the newspaper. AI Overviews flipped that.
Google is the newspaper now. Your site is the wire.

The thing that makes this disorienting is that nobody negotiated the
new arrangement. The publishers did not agree to be wire
contributors. The wire contributors did not get paid. The newspaper
masthead never asked.

Why this is bigger than search-traffic math

The temptation is to treat this as an SEO problem. Traffic numbers
will drop, sites will adapt, the market will sort it out. That misses
the scale.

Google holds the largest single browser deployment on the planet,
the largest single mobile OS, the largest single web video platform,
the dominant share of web standards committee participation, the
dominant share of ad-tech, and now the dominant search-answer
surface. A single company sets the abstraction layer at every point
between the publisher and the reader. When that company decides the
abstraction layer should not include outbound links, there is no
counter-party large enough to disagree at scale.

This is the part the SEO frame misses. The market does not have a
way to sort out a question of this scale because the market does not
have a participant of this scale. The closest analogue is regulatory.
The EU has begun acting. The DOJ case is grinding along. Neither is
fast enough to matter at the speed Google can ship.

The web is going to survive this. The independent publisher, the
solo blog, the niche reference site, the corner of the internet
written by one human for fifteen others, will get squeezed in the
process. Some will respond by exiting. Some will paywall. Some will
move to Substack or to mailing lists, which is just running their own
distribution stack. The shape of the open web will change a lot,
faster than the SEO market expects.

What you can do as a user

The advice older than the problem still applies. Run your own tools.
Pick your own sources. Don't outsource your reading to a system whose
incentives are to keep you on its surface.

Concretely:

Use a search engine that links out. Kagi, Marginalia, Mojeek, Searx, DuckDuckGo, Brave Search. None are perfect, all are linking out by default. The web still indexes; the index is just not the default surface anymore.
Keep an RSS reader. Feedbin, Inoreader, NetNewsWire on the Mac, Readwise Reader if you want web archiving. Subscribing directly to the publishers you trust routes you around the answer-surface entirely.
Build a personal allowlist of go-to sources. The handful of sites you check first when you want a real answer. For me that's about thirty sites; the search engine is where I go when none of them have it. This is roughly how everyone used the web in 2005.
Read source material when the stakes are high. AI Overviews are wrong 10 percent of the time. That rate is fine for "what is the recipe for pancakes." It is not fine for "what does this medication interact with."

The pattern is the same in each case: keep the human in the loop
when the cost of wrong is high, and let the machine condense when the
cost of wrong is low.

What this has to do with what I build

I work on Browy, an open-source AI agent that
lives in a Chromium extension. It drives the real browser tab you
have open. You ask it a question; it inspects the page, follows
links, reads other tabs, and writes you an answer with the sources
visible inline. It does not synthesise from a hosted index. It
operates on the web you would see if you were doing the work yourself.

That is not a coincidence. It is the response to exactly the
disintermediation move described above. If the answer-surface is
hosted by a counter-party whose incentives are not yours, the answer
you want is the one you can produce yourself, against the sources
you trust, on the page that loaded in front of you.

The web is not ending. The middleman is trying to swallow it. Run
your own tools.

All posts Index of every post on the Browy blog.

GitHub got pwned through one VSCode extension. 3,800 repos.

Ritabrata Maiti — Thu, 24 Sep 2026 08:32:12 +0000

GitHub confirmed this week that roughly 3,800 of its internal
repositories were exfiltrated after one of its own employees installed
a trojanised version of the Nx Console extension for VSCode. The
malicious extension came through the official VS Code Marketplace, ran
with the editor's permissions, and quietly handed back the contents of
the employee's repo cache. GitHub has since linked the campaign to the
TanStack npm supply chain attack
from earlier this month and removed the poisoned extension from the
marketplace.

The TeamPCP group is asking $50,000 for the dump. As far as anyone
outside GitHub can tell, no customer data has moved.

That is the news. The interesting part is the design that made it
possible.

The editor has no sandbox

A VSCode extension runs as Node.js code inside the editor's process.
It has the editor's permissions. That means it can read every file in
every workspace you have ever opened, every token in your
~/.aws/credentials, every entry in your SSH agent, and every byte
that ever passed through git push. The Marketplace itself has no
sandbox model. There has been
a feature request open since 2018
asking Microsoft to ship one. The request is still open.

This is not a VSCode-specific complaint. It is the model. Cursor
inherits the architecture. Windsurf inherits it. JetBrains plugins
run with the IDE's permissions, just packaged differently. Vim and
Emacs have had this problem since plugins were invented; the only
reason it never blew up at this scale is that nobody put a marketplace
in front of them.

The Marketplace is the part that scales the blast radius. Twelve
months ago, a set of malicious extensions disguised as legitimate
tools racked up
9 million installs
before takedown. A different set posed as
cryptominer-laden helpers.
A WhiteCobra-flooded batch of twenty-four extensions, one of them with
basic ransomware capability,
made it onto the marketplace before anyone noticed. And in January, a
pair of malicious AI-coding-assistant extensions with
1.5 million installs
silently shipped developer data to servers in China.

Add it up. Across the visible-to-the-public campaigns, this is more
than 11 million installs of code that should not have been there. The
GitHub breach is one well-placed install away from "any of the above
got an SRE."

The bug, named honestly

The threat model in this story is not a clever attacker. The threat
model is a checkbox the operator did not read carefully because they
were trying to get work done.

A developer at GitHub had an editor open. The editor said: install
this extension. It looked like Nx Console; the developer needed Nx
Console. They clicked install. Three thousand eight hundred internal
repos went out the back door. There is no version of "be more
careful" that fixes this. The marketplace is the install surface and
the marketplace cannot, in its current shape, distinguish the real
extension from the malicious one with the same name.

The fix is at the layer the operator doesn't have to think about.
That is either:

The editor ships a sandbox. Extensions ask for the file paths, network endpoints, and OS capabilities they need. The user reviews a manifest, the same way they review extension permissions in a browser. Microsoft has been promising this since 2018.
The marketplace adds an integrity layer that an attacker cannot forge. Reproducible builds, source mirroring, signed releases. Nothing on the marketplace today provides this end to end.
The user installs nothing they have not personally read or that does not come from a vendor whose source they can audit.

Option 3 is the only one available today. Options 1 and 2 require
Microsoft to ship work that has been promised for seven years.

Why this matters for browser extensions

I work on a browser extension, so I want to be clear about what does
and does not transfer from this story.

Chromium MV3 extensions are not VSCode extensions. The permission
model is real. An extension declares the URLs it can touch
(host_permissions) and the capabilities it can use (debugger,
storage, nativeMessaging, and so on) in manifest.json. The
browser enforces those declarations. An extension that says
"host_permissions": ["https://gmail.com/*"] literally cannot reach
bank.example.com. The Chrome Web Store review process is not
perfect, but it does look at the manifest before publish.

That makes the browser-extension surface meaningfully better than the
editor-extension surface for the same class of attack. It does not
make it safe. Things to watch:

<all_urls> is a real permission Chrome will grant if the user agrees. Many general-purpose extensions ask for it. Browy asks for it (we have to; you point us at any tab you visit). What stops abuse is the next thing on the list.
The CRX you download is signed. It is not necessarily the source you read. Chrome Web Store does not require reproducible builds. If the vendor publishes obfuscated code, you are trusting the vendor, not the code.
Native messaging is the escape hatch. Browy has a native host that runs on your machine. So does any "remote control" extension, any password manager native bridge, and many "AI assistants." A malicious extension can ship a clean source repo and a malicious native binary, and the user has no easy way to know.

The list of things that would help Chrome users defend themselves is
roughly the same as for VSCode users: a sandbox the user reviews, a
marketplace integrity layer the attacker cannot forge, and a habit of
installing nothing whose source you have not read.

What I do about this in Browy

Browy is open-source under Apache-2.0. The
source ships in the release. The
agent loop is in one file
and the
tool registry is in another.
A reviewer who reads either file in twenty minutes has read the
attack surface. There is no Browy server. The model call goes from
your machine to GitHub Copilot via the local Copilot CLI; everything
else is local.

The host-touching tools (bash, read_file, write_file, grep,
glob, web_fetch) are off by default. To use any of them you have
to open Settings, find the tool, and turn it on. The browser-driving
tools are on by default but each can be turned off the same way. We
ship a debugger banner whenever a session is live, because Chrome
makes that mandatory and we are happy with that.

None of this prevents a future me from going rogue and shipping a
malicious release. It does mean that anyone who cares can pin a
known-good build, audit any future build before installing, and run
their own from source. The fix for the marketplace problem is not a
better marketplace. It is a habit of looking at what you install.

The short version: when the editor has no sandbox, the only sandbox
left is your attention. Spend it before you click install, not after.

All posts Index of every post on the Browy blog.
AI slop killed the open-source bug bounty The other end of the same arc: cheap inference, asymmetric inbox, maintainer attention as the limiting factor.

AI slop killed the open-source bug bounty

Ritabrata Maiti — Thu, 24 Sep 2026 08:31:31 +0000

Turso, the SQLite-compatible database written in Rust, retired its bug
bounty program this week. The post explaining why is titled, dryly,
The wonders of AI. For
about a year the project had been paying $1,000 for each critical
vulnerability someone reported in the codebase. Budget wasn't the
problem. The problem was that most of the reports landing in the
maintainer's inbox were written by an LLM, and the maintainer was
spending most of his bounty hours reading text that no human had really
intended for him to read.

The examples are funny in a bleak way. One submission claimed to have
found a critical flaw that let an attacker execute arbitrary SQL
statements. Against a SQL database. Another report on the same project
described a buffer overflow whose reproduction steps included editing
Turso's own source code, recompiling with a forced volatile write past
the end of a vector, and running the modified binary. The "vulnerable
code paths" did not exist. The "exploits" did not reproduce.

So the maintainers ended the program. That is a small story about one
database, and also a very large story about every queue on the internet
that involves humans on one side and an open submission form on the
other. Every maintainer who has read Turso's post this week has
recognised their own inbox.

The contract that worked

The economics of a bug bounty are tidy. The project says: if you find a
real problem that meets this severity bar, we pay you $X. Both sides
win. The hunter gets paid for genuinely useful work. The project gets
a security audit it could not have afforded any other way.

That contract rested on a quiet assumption. The cost of submitting a
report was supposed to be in the same neighbourhood as the cost of
producing one. Producing a real exploit is hard. Writing one up takes
hours. So submission volume self-throttled. The queue stayed roughly
the size of the real-findings rate, which is the rate a small team can
actually triage.

Cheap inference broke that assumption in about eighteen months.

What changed

It now costs roughly a tenth of a cent to ask a frontier model to
produce a vulnerability report against a public codebase. The model
will produce one. The report will look real enough that a human has to
spend about ten minutes reading it before concluding that it isn't.
Submission is effectively free. Triage costs the most expensive thing
a software project has, which is the unbroken attention of the one
person who understands the code.

You don't need many people running that arbitrage to break a small
program. Once the submission rate climbs past the triage rate, the
queue does not stabilise. It grows until the maintainer either gives
up reading or gives up the program. Turso gave up the program.

The same shape, everywhere

The bug bounty story is just one shape of a pattern that is now visible
in a dozen places.

Drive-by pull requests are flooding popular GitHub repos. Most are
LLM-generated typo fixes or hallucinated refactors. Maintainers either
auto-close everything and lose the legitimate ones, or burn out on
triage. Open issue trackers are seeing the same flood of "I think there
might be a bug" reports with no reproduction, often filed by a chatbot
middle layer that promised a confused user it would "let the developers
know." Code review comments are now arriving thirty at a time on small
PRs, mostly tautologies and hedges. The comment threads on technical
blog posts are heading the same way.

The shape is always the same. Producing the input costs nothing.
Processing the input still costs a person.

We have automated finding bugs. We have automated submitting bugs. This
year we are automating rejecting bugs. Nobody is automating fixing them.

The bottleneck was never the writing

A lot of the early excitement about LLMs in software was that they
could write code very fast. That turned out to be true, and it turned
out to be the less interesting half of the problem. Writing the code
was never the bottleneck. Reading the code is the bottleneck. Reviewing
the code, understanding the code, deciding whether to merge the code,
deciding whether to ship the code, deciding what to do when the code
breaks at 2am. The reading-and-deciding loop is where engineering
actually happens, and it has always been bounded by how many things a
human can hold in their head at once.

LLMs add to the writing side of that loop and add to the input side of
every queue feeding into it. They don't move the reading-and-deciding
bottleneck. They make it the limiting factor for almost everything.

This is why "just put an AI in the triage loop" doesn't fix the Turso
problem. If there's a model deciding which reports a human sees,
you've created a new game: produce reports that pass the model. That
game is also cheap to play. The judgment step that the attacker is
trying to commodify is the exact step that has to stay with a person,
because that's what was being attacked in the first place.

What might actually hold

A handful of patterns are starting to circulate in maintainer
conversations. None are clean.

A refundable submission deposit is the most-discussed option. You put
down a small amount, say $20. You get it back if the report holds up.
You lose it if it's slop. The economics work. The downside is that it
filters out the hobbyist newcomer who doesn't have a card on file,
which is exactly the population the open programs were designed to
find.

Reputation gating goes the other way. First-time submitters have to
clear a higher bar, including a full reproduction and a vouching
introduction. Established hunters skip the bar. This rebuilds an
apprenticeship model that hasn't really existed in software security
for a generation, but it raises the cost of discovering new talent.

A few projects have started running honeypots. The clearest example is
UnsafeLabs/Bounty-Hunters,
which publishes a bounty-shaped repo that exists mainly to attract
automated scanners. The submissions feed a public
leaderboard of automated submitters.
You can read it as petty, or you can read it as the first attempt at
something a search-engine spam team would recognise: a shared blocklist
for the actor side of an asymmetric inbox problem.

Some teams are moving programs off public surfaces entirely and onto
verified-identity platforms like HackerOne or Bugcrowd, where the
platform handles attribution. Small projects can't usually afford the
platform fees. Bigger ones lose the discovery effect of an open
program.

The most interesting idea I've seen circulating among maintainers is
what someone called proof of code. Before you can submit a bounty
report, you have to land a small, accepted PR somewhere in the
project. The PR doesn't have to be security-related. It just has to be
real engineering by a person who understood the surrounding code. That
converts a cheap text-generation game into a more expensive engineering
game. It doesn't solve the problem. It tilts the asymmetry the other
way.

This is bigger than bounties

The same dynamic is coming for everything that depends on an open
queue with humans on the receiving end. Customer support tickets.
Conference paper submissions. Open peer review. Public consultation
periods on regulation. Government FOIA requests. Hiring inboxes. The
comment thread under this post, probably.

The economic concept is older than the internet. It is the tragedy of
the commons applied to attention instead of grazing land. What's new
is the rate at which the cost of producing input is falling. Most of
the institutions that grew up around the old cost curve don't have
time to redesign before the next drop.

The next decade of public-facing system design is going to be about
adding friction back in on purpose. Identity. Deposits. Reputation.
Proof of work. Vouching. We spent twenty years stripping friction out
of every form on the internet because friction was the enemy of
growth. Now friction is the only thing protecting the people on the
receiving end of those forms. The companies that figure out how to add
it without losing the magic are going to eat everyone else.

A note on what "AI agent" means

There are two things people now mean by "AI agent," and they are
nearly opposite.

One is the slop pipeline that broke Turso. Cheap inference pointed at
any open queue, with no human in the loop, generating volume that
other humans then have to filter. The person whose name is on the
submission usually isn't watching what's submitted. Often they don't
even know what was submitted.

The other is the tool a person uses to do their actual work. An agent
that drives the browser tab they're sitting in front of, that reads
the codebase they're working on, that runs the command they would
have typed. The person is at the keyboard. They see every action.
Nothing gets sent to anyone else's queue without them watching it
happen.

These two things share a phrase and almost nothing else. The first one
is a tax on every public surface on the internet. The second one is a
power tool. The fact that they have the same name in the press is
going to cost the second category a lot of goodwill before things
sort themselves out.

I work on the second kind. Browy is an
open-source AI agent that lives in a Chrome side panel and a DevTools
REPL. It drives your real browser tabs through chat. There is no Browy
server in the loop. There is no inbox you can flood by talking to it.
There is a person, you, watching every click and every form fill it
makes. Whatever happens to the public commons over the next few years,
the part of the internet that's a tool you operate yourself is still
going to be a good place to live.

The 30-second video version of this argument is at the top of this
post. Turso's own write-up is
here.

All posts Index of every post on the Browy blog.
Mullvad gave you 8 trillion exit IPs. 9 servers found you. Same researcher-pattern thinking, different domain: a VPN whose deterministic IP picker leaks identity across servers.

Mullvad gave you 8 trillion exit IPs. 9 servers found you.

Ritabrata Maiti — Thu, 24 Sep 2026 08:30:05 +0000

Mullvad is one of the few VPN providers that hands you a different public
IP for each of its servers. With 578 servers in the fleet, a naive
calculation gives you over 8 trillion possible combinations of exit IPs
across the network. The pitch is that you melt into a very large crowd.

A researcher who writes as tmctmt sat down with a script, rotated through
3,650 WireGuard keys, and watched which exit IPs each key was assigned
across nine servers. They expected to see thousands of distinct
combinations. They saw 284. The
full write-up is here.

The cause is unglamorous and the impact is not.

How the picker works

When you connect to a Mullvad server, the exit IP you get is not random.
It is picked deterministically from a per-server pool, seeded by your
WireGuard public key. The pool sizes vary by server. Sydney exposes 60
IPs, Helsinki 66, Los Angeles 91, Santiago 11. The key rotates every
one to thirty days if you use the official app, and never if you bring
your own WireGuard client.

So far so reasonable. Per-key stickiness avoids hammering a single
external IP with users who reconnect every few minutes, and it lets
sites that ratelimit by IP behave sanely for a Mullvad user inside one
session.

The problem is the picker itself.

One float, many pools

The picker seeds a standard library RNG with the pubkey, then draws a
single floating point number in the range zero to one. It scales that
float to the size of the current server's pool to produce an index.
That part is plausible. The part that breaks privacy is that the float
is the same float on every server, because the RNG is seeded with the
same pubkey and only the upper bound changes between calls.

The consequence is that your exit IPs across servers land in the same
percentile of each pool. If your float happens to be 0.82, you sit at
position 49 of 60 in Sydney, position 9 of 11 in Santiago, position 54
of 66 in Helsinki. Different IPs, same relative slot.

If you have ever wondered why two of the smaller Mullvad servers seem
to give you the "same" index, this is why. Santiago and Johannesburg
both have pools of 11. With the same float scaled to the same size,
they hand you the same position.

Why 8.2 trillion collapsed to 284

This is the headline of the post. The pool sizes multiply to 8.2 trillion
on paper. In practice the float is one number, so the entire vector of
exit IPs across all servers is parameterised by that one number. The
researcher's 3,650 sampled pubkeys produced only 284 distinct
combinations across nine servers because the space being sampled is
not thirteen-dimensional. It is one-dimensional, then projected.

The 284 figure is the resolution of that projection at nine servers.
With more servers the resolution rises, because each additional pool
size carves the unit interval into finer buckets. With fewer servers
it drops, but it does not drop as much as you would hope.

What that means for a user

The author built a small tool that takes a set of observed exit IPs and
back-solves for the float interval that produced them. Nine well-chosen
exit IPs squeeze the interval down to roughly 0.0034 wide. At an
estimated 100,000 active Mullvad users that is around 340 people.

Three hundred and forty is a lot of people in a crowd. It is not a lot
of people in a deanonymisation attack. If a forum has IP logs and
suspects two accounts are the same user, and the two accounts both
connected through Mullvad on different servers, the overlap of the
float intervals derived from those exits gives a very high probability
that the accounts share a pubkey. Pair that with timing, user agent,
language headers, or the kind of low-effort browser fingerprint that
ad networks already collect, and the pool of suspects shrinks to one.

The interesting attack is not a moderator with one forum's logs. It is
the joining of logs across services that already happens in litigation,
in subpoenas, and in stolen breach data. Mullvad's per-server IP
diversity stops being a defence the moment two unrelated services
contribute their exit IP records to the same correlation.

The response

Mullvad's co-founder posted a response on the morning the article went
up. The short version: some of the described behaviour was intentional,
some was not, the unintended part is being patched on a subset of the
fleet first, and the intended part is now under internal review. He
also asked, gently, that future researchers consider notifying the
vendor before publishing.

That is about the best response a small vendor can give to a Tuesday
morning surprise, and it points at the part of this story that is most
worth sitting with.

The bug under the bug

The proximate bug is a seeded RNG whose first draw was treated as a
fresh random number per call, when it is the same number every call.
A defensible code review could miss this. The Rust documentation does
not put the behaviour on a billboard, and the test that would have
caught it, "do my exit IP percentiles correlate across pools," is not
the test anyone writes.

The deeper bug is the design assumption that per-key stickiness is
compatible with per-server independence. It is not, for any picker
that derives its index from a single seeded float. To get true
independence you have to either reseed per call with a per-server salt,
or pre-mix the pubkey with the server identity before drawing, or
abandon determinism and accept the operational cost.

There is a clean lesson here about cryptographic determinism that I
will not labour, but the practical one for anyone shipping
privacy-flavoured software is worth saying out loud. Deterministic
behaviour is the thing that makes user experience predictable and the
thing that makes users linkable across observations. The instinct to
make a system feel coherent across sessions is the same instinct that
hands an analyst a join key.

What a user can do today

The author lists two mitigations. Both reduce the value of the
fingerprint without removing it.

The first is to avoid switching servers while your pubkey is unchanged.
If you only ever connect from one Mullvad server, an observer sees one
exit IP and gets one of the pool's possible float intervals, not the
intersection of many. That interval covers a much larger share of the
user base.

The second is to force-rotate your pubkey, which the Mullvad app will
do if you log out and log back in. A new pubkey produces a new float
and a new vector of exit IPs. Past observations of you, before the
rotation, still link to whatever activity happened under the old key.

Neither of these is satisfying. Both are what the user can do until
the patch lands.

The bigger picture for privacy tools

VPNs are sold as a tool for blending in. The category has spent ten
years competing on server count, jurisdiction, and "we don't log."
None of that helps if the tunnel itself emits a stable signal that
correlates across exits. Mullvad is one of the better-regarded
operators in the category, run by people who care about this stuff
and respond like it. If this design slipped past them, it is worth
assuming similar designs are sitting unexamined in operators who
care less.

The healthier framing for anyone using a privacy tool is that the tool
moves the boundary of who can see you. It does not make you invisible
to the people inside the new boundary. A VPN protects you from your
ISP and from the cafe Wi-Fi. It does not protect you from the website
you are visiting, which still sees your browser, your timing, and
whatever your tunnel happens to leak about your identity across
sessions. Pretending otherwise is how people end up surprised.

What this has to do with browsers

A lot of the leak surface in modern web sessions is in the browser,
not in the tunnel. Canvas fingerprinting, font enumeration, audio
context fingerprinting, third party cookies that you thought were
blocked, redirect chains that smuggle identifiers in the URL, the
clipboard, the way your tabs talk to each other, the way an extension
you forgot you installed talks to its origin. The interesting thing
about the Mullvad story is that it is one of the rare cases where the
leak is below the browser, and the browser would not see it without
help.

I work on Browy, an open-source AI agent that
lives in a Chrome side panel and a DevTools REPL. It drives the real
tab you are looking at. You can point it at a page and ask, in
English, what the page is sending, what it stores, what it loads from
where, what it is trying to identify you with. It will open the
Network tab and the Application tab and read them for you, and it
will tell you what it found in the same chat where you asked the
question. It does not send your browsing anywhere. The model call
goes out and the answer comes back, the rest happens on your machine.

The Mullvad bug is a server-side problem. Browy will not see it. But
the broader habit, watching what your browser actually emits while
you are using it, is what every category of privacy leak in the last
decade has rewarded. The 30-second video version of the Mullvad story
is at the top of this post. The full research, with the seed
estimator tool, is
at tmctmt's site.

All posts Index of every post on the Browy blog.
AI slop killed the open-source bug bounty The same shape in software security: a real database project just retired its bounty program because AI submissions broke the queue.

AnyModal: Train Multimodal LLMs in PyTorch

Ritabrata Maiti — Tue, 19 Nov 2024 11:13:42 +0000

Today, I want to introduce an open-source framework I’ve been working on: AnyModal.

Introduction

During my work on machine learning projects, I struggled to find flexible solutions for training multimodal LLMs. While there are plenty of great tools for specific tasks—like image classification or audio processing—there was no straightforward way to combine these modalities with large language models (LLMs). The process was often tedious, involving boilerplate code, custom integration, and a lot of trial and error to make different components work together.

This frustration led me to build AnyModal, a framework designed to reduce the complexity of multimodal AI development. It provides a modular, reusable structure that makes it easier for developers and researchers to combine diverse data types and experiment with new ideas without reinventing the wheel every time.

The Goal

AnyModal is built with the following objectives in mind:

Reduce Boilerplate Code

Combining modalities like images or audio with LLMs typically involves repetitive steps—preprocessing, encoding, tokenizing, and integrating. AnyModal minimizes this boilerplate by providing reusable modules for common tasks, letting developers focus on building smarter systems faster.

Enable Seamless Integration

Whether you're working with images using a Vision Transformer (ViT) or audio spectrograms, AnyModal offers plug-and-play components that simplify the integration process. This makes it easy to handle multiple data types within a single framework.

Encourage Experimentation and Customization

AnyModal supports rapid prototyping while offering the flexibility to customize components like feature encoders, projection layers, and tokenizers. It’s versatile enough for both quick experiments and production-level deployments.

Example Usage: Integrating Images with LLMs

Here’s a detailed example of how AnyModal simplifies the integration of image data into LLMs:

1. Install Dependencies

pip install torch transformers datasets torchvision tqdm

2. Initialize Vision Components

from transformers import ViTImageProcessor, ViTForImageClassification

# Load a pre-trained Vision Transformer
processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')
vision_model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')

# Define a Vision Encoder to extract feature embeddings
from vision import VisionEncoder
vision_encoder = VisionEncoder(vision_model)

3. Initialize Tokenizer and LLM

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load a pre-trained LLM and its tokenizer
llm_tokenizer = AutoTokenizer.from_pretrained("gpt2")
llm_model = AutoModelForCausalLM.from_pretrained("gpt2")

4. Define a Projection Layer

from vision import Projector

# Create a projection layer to map vision embeddings to LLM token space
vision_tokenizer = Projector(
    in_features=vision_model.config.hidden_size, 
    out_features=768
)

5. Combine Everything with AnyModal

from anymodal import MultiModalModel

# Build the multimodal model
multimodal_model = MultiModalModel(
    input_processor=None,
    input_encoder=vision_encoder,
    input_tokenizer=vision_tokenizer,
    language_tokenizer=llm_tokenizer,
    language_model=llm_model,
    input_start_token='<|imstart|>',
    input_end_token='<|imend|>',
    prompt_text="Describe this image: "
)

6. Training and Inference

Training involves processing batches of image-text pairs and optimizing the model:

from torch.utils.data import DataLoader
from datasets import load_dataset

# Load a sample dataset
dataset = load_dataset("image_caption_dataset", split="train")

# Prepare DataLoader
train_loader = DataLoader(dataset, batch_size=2, shuffle=True)

# Training Loop
optimizer = torch.optim.AdamW(multimodal_model.parameters(), lr=3e-4)

for epoch in range(10):
    for batch in train_loader:
        optimizer.zero_grad()
        logits, loss = multimodal_model(batch)
        loss.backward()
        optimizer.step()

# Generate captions
sample_input = dataset[0]['image']
generated_caption = multimodal_model.generate(sample_input, max_new_tokens=30)
print("Generated Caption:", generated_caption)

Current Status

AnyModal is currently in its early stages, with the latest version supporting tasks like:

LaTeX OCR
Chest X-Ray Captioning (in progress)
Image Captioning

Future planned features include support for visual question answering and audio captioning.

As the framework evolves, I’m focusing on expanding its functionality, refining the codebase, and addressing community feedback to move towards a stable release.