DEV Community

Cover image for Pilotfish: Redesigning Browser Agents
Nitin
Nitin

Posted on Edited on

Pilotfish: Redesigning Browser Agents

An agent opens an invoice. The network is quiet. The DOM has stopped changing. The balance says $8,400, under a banner that reads "Refreshing balances".

The wrong optimization is to return that number faster.

The useful one is to cut repeated page text and model round trips without losing track of what the agent knows. The rule that makes the rest work: a diff must start from the page the caller actually received, not merely the one the browser captured.

I work on Pilotfish. This is a shorter engineering companion to my interactive article, using a fictional receivables portal. The customer, invoice and amounts below are illustrative.

Contents

Keep the row intact

A page serializer can drop generated CSS classes without dropping the relationship between an invoice and its balance:

Customer | Invoice | Outstanding
Acme Fasteners | [e5]INV-1042 | $8,400.00
Enter fullscreen mode Exit fullscreen mode

The important part is the row, not the character count. If serialization separates the amount from its invoice, the model gets fewer tokens and a worse problem.

Pilotfish's plain-row path keeps cells in order and the link's action reference. It does not flatten every table indiscriminately: form controls, stateful cells and nested tables can fall back to the normal representation. The row serializer spells out those boundaries.

Viewport-only output needs another contract: say what was left out and how to read more. Offscreen is not the same as irrelevant. A compact view can still miss the control the task needs, so keep a fuller read available.

Diff against the page the model received

Typing a customer name changes a field and a few suggestions. Resending the unchanged invoice table is wasteful. A diff helps, provided the model can apply it.

The base must be a full observation that reached the caller, not the server's latest capture.

Illustrative protocol sequence: A is withheld, diff A to B is rejected, full B recovers the base, then diff B to C succeeds

Illustrative protocol model. This is not a recording of a Pilotfish run.

Capture A can exist on the server while the caller still has no page at all. If A is withheld, a later diff from A to B is unusable. The recovery is a full B, not a cleverer diff.

In the implementation described here, a full observation commits as the base on delivery. A withheld reply leaves no base; the next observation must be full again. See the delivery/base rules.

A diff also needs page order. Two invoices swapping balances must not look like an unchanged bag of lines. The ordered diff uses surrounding context, counts replaced lines and names removed interactive refs.

When the document changes, the base gets old, or the diff becomes too large or ambiguous, send a full observation. Resynchronization is part of the design, not an embarrassing fallback.

Run the failure and recovery

Save this as delivered-base.mjs and run node delivered-base.mjs. It uses only Node's built-in assertions. No browser, account, API key or model call is needed.

This is a deliberately small protocol model, not Pilotfish's implementation or an integration test. Each balance stands in for a full page. Delivery is synchronous; a real transport must also handle dropped responses, acknowledgements and resets. Its job is to make the broken-base case reproducible.

import assert from 'node:assert/strict';

// A small protocol model, NOT the Pilotfish implementation.
const capture = (id, balance) => ({ id, balance });
const a = capture('A', 8400);
const b = capture('B', 5150);
const c = capture('C', 5000);
const client = { base: null };

function receive(reply) {
  if (reply.kind === 'diff') {
    assert.equal(reply.from, client.base?.id, 'unknown diff base');
    client.base = { id: reply.to, balance: reply.balance };
  } else {
    client.base = reply.page;
  }
}

// Failure: the server captured A, but A never reached the client.
assert.throws(() => receive({ kind: 'diff', from: a.id,
  to: b.id, balance: b.balance }), /unknown diff base/);
console.log('FAILURE CAUGHT: diff A -> B has no delivered base');

let delivered = null;
function replyFor(page, withheld = false) {
  if (withheld) return null; // no delivery, no committed base
  const reply = delivered
    ? { kind: 'diff', from: delivered.id, to: page.id,
        balance: page.balance }
    : { kind: 'full', page };
  receive(reply); // synchronous delivery in this toy model
  delivered = page;
  return reply;
}

assert.equal(replyFor(a, true), null);
assert.equal(replyFor(b).kind, 'full');
assert.deepEqual(client.base, b);
console.log('RECOVERED: full B, balance 5150');
assert.equal(replyFor(c).kind, 'diff');
assert.deepEqual(client.base, c);
console.log('NEXT: diff B -> C, balance 5000');
console.log('PASS: withheld capture never became the diff base');
Enter fullscreen mode Exit fullscreen mode

Checked output:

FAILURE CAUGHT: diff A -> B has no delivered base
RECOVERED: full B, balance 5150
NEXT: diff B -> C, balance 5000
PASS: withheld capture never became the diff base
Enter fullscreen mode Exit fullscreen mode

Try changing the withheld branch to if (withheld) { delivered = page; return null; }. Recovery now fails with unknown diff base: the server sends a diff even though the caller still has no base. That is the bug this model isolates.

Quiet does not mean ready

Back to the invoice. A partial payment has arrived, but the balance panel has not caught up.

There are two checks:

Check What it tells you
The action settled The requests and rendering caused by this action have finished, within the wait's limits.
The needed value is ready The page no longer presents it as loading or interim.

Whole-page network quiet is a poor shortcut. Analytics and polling can keep a useful page busy. A quiet page can still display a loading balance.

Pilotfish's adaptive settle step follows work started by the action rather than waiting for all traffic. Write requests get particular attention: an unanswered write is not an accepted write.

A separate loading-indicator option waits up to six seconds for signals such as aria-busy or an indeterminate progress bar. If they remain, the reply says so.

Neither check proves every website is correct. Loading signals can be absent or misleading. The point is to preserve uncertainty instead of quietly converting it into a reported balance.

Batch the typing, not the judgment

Now the balance is confirmed at $5,150. The invoice is not disputed, and no recent reminder exists. The workflow allows a reminder draft, pending human review.

The recipient, subject, message and internal note are known. These fills can share one model decision:

decide once
  -> fill email   -> check
  -> fill subject -> check
  -> fill message -> check
  -> fill note    -> check
stop before Send
Enter fullscreen mode Exit fullscreen mode

The browser steps still run in order. Batching removes the decisions and tool calls between them, not the checks.

What if the email validator is still spinning after step one? Stop. Known text does not make a changing page safe to keep acting on.

The batch executor also stops for failures, navigation, dialogs, new tabs, refused or unanswered writes, loading, and sign-in/check states. An invalid field is reported, but is not by itself a stop condition in this implementation. That distinction belongs in the contract.

What to test

Before celebrating smaller replies, test the awkward cases:

  • Swap two rows' amounts. Does the diff preserve ownership?
  • Withhold a full reply. Does the next action recover a usable base?
  • Leave a loading indicator visible after requests finish. Is the value still treated as interim?
  • Leave a write unanswered. Is it distinguishable from success?
  • Trigger validation during a batch. Do later actions stop where promised?

Then measure the whole task, including recovery reads and failures. A tiny reply followed immediately by another model call may cost more than a useful first observation.

Fewer tokens are a result worth measuring. They are not the correctness argument.

What the published runs show

On 11 November 2026, the project ran the same WebVoyager task set separately on each tool: Pilotfish open MCP 0.1.39 and Playwright MCP 0.0.84. After matched exclusions, Pilotfish completed 65 correctly out of 71 eligible tasks; Playwright completed 61 correctly out of the same 71 eligible tasks*. Counts and medians were recomputed from the published result rows.

Matched tasks only* Pilotfish open MCP Playwright MCP
Correct / eligible tasks 65 / 71 61 / 71
Incorrect / eligible tasks 6 / 71 10 / 71
Median input tokens 29,526 90,342
Median tool calls 5 10
Median seconds 100.0 115.7

* Exclude a task from both columns if either tool was site-blocked (24 task pairs), then exclude the 5 remaining pairs with a pre-turn error in either run. The resulting 71-task denominator contains identical task IDs for both tools, each run separately. All medians use that same set. These results describe eligible attempts, not overall success or site-access coverage; excluded records remain in the linked data.

This is one project-run, whole-system comparison: GPT-6 Luna drove and judged both arms on one Windows machine in India, with five tasks running concurrently. It is not independent validation or proof of accuracy superiority. The open MCP comparison does not test the managed service. Method and reproduction details.

Connect this to a browser

Project links, grouped by what you want to use:

For Claude Code, the documented registration command is:

claude mcp add pilotfish -- npx -y @gaiventures/pilotfish-mcp
Enter fullscreen mode Exit fullscreen mode

This registers the open MCP, not the TypeScript SDK or managed relay. The command is from the setup docs; the local test above runs only the protocol model. Start browser experiments on a disposable local page, then test response withholding and page resets.

Video and source notes

The 62-second LinkedIn explainer gives a visual overview of Pilotfish. It is not a recording of the fictional invoice task. The original interactive article has the longer walkthrough; implementation and benchmark links here pin commit b990f0f.

Top comments (0)