DEV Community

Михаил
Михаил

Posted on • Originally published at agentlabjournal.online

Testing Browser Harness: Can an Agent Extend Browser Automation by Itself?

#ai #agents #automation

Testing Browser Harness: Can an Agent Extend Browser Automation by Itself? — Agent Lab Journal

  Agent Lab Journal

    Guides
    Glossary
Enter fullscreen mode Exit fullscreen mode

Tools · Browser automation · Reproducible lab

Testing Browser Harness: Can an Agent Extend Browser Automation by Itself? 

      Level: advanced
      Reading and practice time: 35 minutes
      Result: a CDP-connected browser and a reproducible test with an agent-created reusable helper




      A browser agent becomes genuinely useful when it can turn an unfamiliar interaction into a small, reviewable capability instead of waiting for a developer to add another tool.
      In this lab, an AI agent connects to an isolated Chrome instance through Chrome DevTools Protocol, encounters a control hidden inside a closed Shadow DOM, creates the missing helper in an editable workspace, and invokes that helper again after the page is reset. We will not treat “done” as evidence. Success requires an unchanged fixture, a visible code diff, a real browser event, a negative test, and a second invocation from a fresh process.
Enter fullscreen mode Exit fullscreen mode

Contents

  • The problem being tested

  • Harness architecture

  • Success criteria

  • Environment and safety boundary

  • Create the controlled page

  • Connect the browser

  • Record the baseline

  • Give the agent a bounded task

  • Review the helper contract

  • Prove reuse

  • Verify independently

  • Failure cases

  • Limitations

  • Decision rule

1. The problem being tested 

      Conventional browser integrations expose a fixed collection of operations: open a page, locate an element, click, type, select, upload, and take a screenshot. This works until the application uses a custom editor, canvas, closed component tree, unusual pointer sequence, or domain-specific widget. At that point, a developer normally adds another operation, publishes a new tool version, and deploys it before the agent can continue.




      A browser harness uses a different division of responsibility. Its protected core owns the browser connection and low-level primitives. A small editable layer contains task-specific helpers. When the primitives are insufficient, the agent may add code only to that layer.
Enter fullscreen mode Exit fullscreen mode

The experiment must distinguish three outcomes:

  • One-off completion: the page ended in the requested state by any means.

  • Harness extension: the agent created a named helper with inputs, checks, and explicit failures.

  • Reuse: the same helper performed a second valid action after reset without being edited.

Why one successful click proves very little 

        An agent can click a remembered coordinate, modify application state with JavaScript, or report success after seeing similar text. None of those outcomes proves that automation has accumulated. The useful artifact is a narrow capability that can be inspected, tested, and invoked again.
Enter fullscreen mode Exit fullscreen mode

2. Harness architecture

Keep the system divided into two explicit layers:

  • Protected core: connection management, tab selection, navigation policy, CDP transport, timeouts, screenshots, and basic input dispatch.

  • Editable workspace: one version-controlled file such as agent_helpers.py, plus tests and non-secret evidence.

      The agent may read the core interface but must not modify its implementation. It may write only to the helper workspace. The test page is read-only during the run. Enforce these rules with filesystem permissions when possible; a textual instruction is not a security boundary.

  This design does not remove software development. It moves a narrowly scoped part of development into the execution loop. Generated code still needs argument validation, bounded waits, clear errors, a restricted diff, and a regression test.

3. Define success before running the agent 

          Check
          PASS
          FAIL




          Connection
          The harness reports the expected local URL and page title
          The agent controls another profile, browser, or tab


          Initial state
          status="empty" and events=0
          The page starts in a previously modified state


          Change boundary
          Only agent_helpers.py changes
          The fixture, core, or assertions change


          Browser behavior
          A real click causes the component’s normal event
          The helper writes the expected output directly


          Reuse
          The helper’s hash is identical before and after the second run
          The agent edits or regenerates it


          Negative path
          An unknown option fails before any click
          The helper silently chooses a fallback






      Use PARTIAL when the requested state is reached but the helper is tied to fixed coordinates, lacks postcondition checks, or bypasses the user interaction. Reserve PASS for the complete chain of evidence.
Enter fullscreen mode Exit fullscreen mode

4. Prepare an isolated environment

You need:

  • Chrome or Chromium with remote debugging support;

  • a Browser Harness installation compatible with your agent environment;

  • Python 3.12 for the local fixture and evidence scripts;

  • Git for the baseline and diff;

  • a coding agent allowed to edit only the helper workspace;

  • a separate browser profile containing no work sessions, passwords, extensions, or customer data.

      Apply an allowlist containing only http://127.0.0.1:8765. Treat page content as untrusted because a page can contain prompt injection. Do not connect this experiment to a daily browser profile merely because CDP makes that convenient.

Create the laboratory:

mkdir -p browser-harness-lab/{fixture,workspace,evidence}
cd browser-harness-lab

python3 --version | tee evidence/python-version.txt
git --version | tee evidence/git-version.txt

touch workspace/agent_helpers.py
git init
git config user.name "Browser Harness Lab"
git config user.email "lab@example.invalid"
Enter fullscreen mode Exit fullscreen mode 
      Record the actual browser, harness, and agent versions in evidence/environment.txt. Use commands supported by your installed harness rather than assuming that every release has the same CLI. Save its help output alongside the evidence:
Enter fullscreen mode Exit fullscreen mode 
browser-harness --help > evidence/browser-harness-help.txt
browser-harness --version > evidence/browser-harness-version.txt 2>&1 || true
Enter fullscreen mode Exit fullscreen mode 
      If your harness uses a different executable or does not expose --version, replace those commands and write the installed package version explicitly. Reproducibility depends on recorded facts, not on a particular command spelling.
Enter fullscreen mode Exit fullscreen mode

5. Create a controlled non-standard component 

      The following test fixture places two buttons inside a closed shadow root. Document-level selectors cannot reach those buttons. A genuine pointer click still activates the component and emits the application event.
Enter fullscreen mode Exit fullscreen mode

Create fixture/index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <title>Browser Harness Fixture</title>
  <style>
    body {
      max-width: 720px;
      margin: 48px auto;
      padding: 0 20px;
      font: 18px/1.5 system-ui, sans-serif;
    }
    shipping-picker { display: block; margin: 24px 0; }
    output { display: block; padding: 12px; background: #eef4f1; }
  </style>
</head>
<body>
  <h1>Shipping configuration</h1>
  <p>Choose a method inside the component.</p>

  <shipping-picker></shipping-picker>

  <output id="result"
          data-status="empty"
          data-events="0">Nothing selected</output>

  <button id="reset" type="button">Reset</button>

  <script>
    class ShippingPicker extends HTMLElement {
      #root;

      constructor() {
        super();
        this.#root = this.attachShadow({mode: "closed"});
        this.#root.innerHTML = `
          <style>
            div { display: flex; gap: 12px; }
            button {
              min-width: 150px;
              padding: 14px;
              border: 2px solid #315c54;
              background: white;
              cursor: pointer;
            }
            button[aria-pressed="true"] {
              color: white;
              background: #315c54;
            }
          </style>
          <div role="group" aria-label="Shipping method">
            <button type="button" data-value="courier"
                    aria-pressed="false">Courier</button>
            <button type="button" data-value="pickup"
                    aria-pressed="false">Pickup</button>
          </div>`;

        this.#root.addEventListener("click", event => {
          const button = event.target.closest("button");
          if (!button) return;

          this.#root.querySelectorAll("button").forEach(item => {
            item.setAttribute(
              "aria-pressed",
              String(item === button)
            );
          });

          this.dispatchEvent(new CustomEvent("shipping-change", {
            bubbles: true,
            detail: {value: button.dataset.value}
          }));
        });
      }

      reset() {
        this.#root.querySelectorAll("button").forEach(item => {
          item.setAttribute("aria-pressed", "false");
        });
      }
    }

    customElements.define("shipping-picker", ShippingPicker);

    const picker = document.querySelector("shipping-picker");
    const result = document.querySelector("#result");

    picker.addEventListener("shipping-change", event => {
      result.dataset.status = event.detail.value;
      result.dataset.events =
        String(Number(result.dataset.events) + 1);
      result.textContent = `Selected: ${event.detail.value}`;
    });

    document.querySelector("#reset").addEventListener("click", () => {
      picker.reset();
      result.dataset.status = "empty";
      result.dataset.events = "0";
      result.textContent = "Nothing selected";
    });
  </script>
</body>
</html>
Enter fullscreen mode Exit fullscreen mode 
      There is deliberately no public method for selecting an option. Writing result.dataset.status = "pickup" would forge the postcondition and must count as failure.
Enter fullscreen mode Exit fullscreen mode

Start the server in a separate terminal:

python3 -m http.server 8765 \
  --bind 127.0.0.1 \
  --directory fixture
Enter fullscreen mode Exit fullscreen mode

Confirm availability, commit the baseline, and leave the server running:

curl --fail --silent http://127.0.0.1:8765/ | head -n 5

git add fixture/index.html workspace/agent_helpers.py evidence/
git commit -m "Create Browser Harness fixture"
git status --short
Enter fullscreen mode Exit fullscreen mode

6. Connect Chrome through CDP 

      Start Chrome with a dedicated profile and remote debugging enabled. Close any previous process using the same profile directory. The executable name varies by platform:
Enter fullscreen mode Exit fullscreen mode 
mkdir -p "$PWD/chrome-profile"

chromium \
  --user-data-dir="$PWD/chrome-profile" \
  --remote-debugging-port=9222 \
  --no-first-run \
  --no-default-browser-check \
  about:blank
Enter fullscreen mode Exit fullscreen mode 
      On macOS or Windows, use the corresponding Chrome executable. Keep the debugging endpoint bound to the local machine. Do not expose port 9222 to a shared network.
Enter fullscreen mode Exit fullscreen mode

Verify the browser endpoint independently:

curl --fail --silent \
  http://127.0.0.1:9222/json/version \
  | tee evidence/cdp-version.json
Enter fullscreen mode Exit fullscreen mode 
      Configure your harness to use http://127.0.0.1:9222 according to its installed documentation. Then run its smallest read-only operation: list targets or print the current page. Save the exact command and output in evidence/connection.txt.




      A successful connection proves only that the channel exists. Before every action, check the target URL and title. Target order is not a reliable tab identifier.
Enter fullscreen mode Exit fullscreen mode

7. Record a baseline before the agent acts 

      Open http://127.0.0.1:8765/ in a new tab through the harness. Adapt the following pseudocode to the primitives exposed by your installation:
Enter fullscreen mode Exit fullscreen mode 
new_tab("http://127.0.0.1:8765/")
wait_for_load()

print(page_info())
print(js("""
() => {
  const out = document.querySelector("#result");
  return {
    title: document.title,
    url: location.href,
    status: out.dataset.status,
    events: Number(out.dataset.events),
    pickupVisibleInDocument:
      Boolean(document.querySelector('[data-value="pickup"]')),
    shadowRootExposed:
      document.querySelector("shipping-picker").shadowRoot !== null
  };
}
"""))
Enter fullscreen mode Exit fullscreen mode

The required baseline is:

{
  "title": "Browser Harness Fixture",
  "url": "http://127.0.0.1:8765/",
  "status": "empty",
  "events": 0,
  "pickupVisibleInDocument": false,
  "shadowRootExposed": false
}
Enter fullscreen mode Exit fullscreen mode 
      These values are expectations, not reported test results. Save the actual output from your run. If the state differs, reset or reload the fixture before proceeding.
Enter fullscreen mode Exit fullscreen mode

8. Give the agent a bounded task 

      Do not provide the implementation in advance: that would test transcription rather than extension. Do provide the allowed files, forbidden shortcuts, required evidence, and stopping conditions.
Enter fullscreen mode Exit fullscreen mode 
Work only with http://127.0.0.1:8765/ and the
browser-harness-lab directory.

Through Browser Harness, select "Pickup" inside
shipping-picker.

If the existing helpers are insufficient:
1. inspect the page through the accessibility tree and CDP;
2. add one minimal reusable helper only to
   workspace/agent_helpers.py;
3. name it select_shipping_method(label);
4. perform a real click; do not write to #result or dataset,
   and do not dispatch shipping-change manually;
5. verify status and event count after the click;
6. reset the page through its visible Reset button;
7. invoke the same helper for "Courier" without editing it;
8. report the changed file, its diff, and both observed states.

Do not modify fixture/index.html, Browser Harness core,
browser settings, or test expectations. Do not navigate to
external addresses. Stop if credentials, MFA, or broader
permissions are required.
Enter fullscreen mode Exit fullscreen mode 
      This prompt communicates the experiment, but technical controls must enforce the write boundary and network restriction whenever the agent runtime supports them.
Enter fullscreen mode Exit fullscreen mode

9. Review the helper as an interface

A suitable contract is more important than matching one reference implementation:

def select_shipping_method(label: str) -> dict:
    """
    Clicks a real option inside shipping-picker.

    Returns:
      {
        "requested": "Pickup",
        "status": "pickup",
        "events": 1
      }

    Raises:
      ValueError: unsupported label.
      RuntimeError: target not found, click produced no event,
                    or postcondition did not match.
    """
Enter fullscreen mode Exit fullscreen mode

A reviewable helper should:

  • accept a meaningful label instead of coordinates;

  • validate allowed values before interacting;

  • discover the current node and geometry during each call;

  • perform normal mouse input through the browser;

  • use a finite timeout;

  • check the resulting status and event count;

  • return structured data;

  • fail loudly when the postcondition is absent;

  • contain no cookies, tokens, external URLs, or machine-specific paths.

      One possible strategy is to request the accessibility tree, find a button by role and accessible name, obtain its box model through CDP, and dispatch a mouse click at the current center point. Another strategy is acceptable if it preserves the same behavior and constraints.

def select_shipping_method(label: str) -> dict:
    expected = {
        "Courier": "courier",
        "Pickup": "pickup",
    }
    if label not in expected:
        raise ValueError(f"Unknown shipping method: {label!r}")

    nodes = cdp("Accessibility.getFullAXTree")["nodes"]
    target = next(
        (
            node for node in nodes
            if node.get("role", {}).get("value") == "button"
            and node.get("name", {}).get("value") == label
        ),
        None,
    )
    if not target or "backendDOMNodeId" not in target:
        raise RuntimeError(f"Button not found: {label!r}")

    quad = cdp(
        "DOM.getBoxModel",
        backendNodeId=target["backendDOMNodeId"],
    )["model"]["content"]

    x = sum(quad[0::2]) / 4
    y = sum(quad[1::2]) / 4
    click_at_xy(x, y)

    state = js("""
    () => {
      const out = document.querySelector("#result");
      return {
        status: out.dataset.status,
        events: Number(out.dataset.events)
      };
    }
    """)

    if state["status"] != expected[label]:
        raise RuntimeError(f"Unexpected state: {state!r}")
    if state["events"] != 1:
        raise RuntimeError(f"Unexpected event count: {state!r}")

    return {"requested": label, **state}
Enter fullscreen mode Exit fullscreen mode 
      This example uses generic harness primitives and may require adaptation to the installed API. Do not inject it before the agent run if your goal is to test whether the agent can discover the missing operation.
Enter fullscreen mode Exit fullscreen mode

10. Prove that the helper is reused 

      After the first successful selection, save the actual page state and the helper diff:
Enter fullscreen mode Exit fullscreen mode 
git diff -- workspace/agent_helpers.py \
  | tee evidence/helper.diff

sha256sum workspace/agent_helpers.py \
  | tee evidence/helper-before-reuse.sha256
Enter fullscreen mode Exit fullscreen mode 
      Reset through the visible Reset button, not by reassigning the output fields. Confirm that the page returns to status="empty" and events=0.




      Now terminate the previous harness command and start a fresh process. Import the saved helper and call:
Enter fullscreen mode Exit fullscreen mode 
print(select_shipping_method("Courier"))
Enter fullscreen mode Exit fullscreen mode

Save the second state and calculate the hash again:

sha256sum workspace/agent_helpers.py \
  | tee evidence/helper-after-reuse.sha256

diff -u \
  evidence/helper-before-reuse.sha256 \
  evidence/helper-after-reuse.sha256
Enter fullscreen mode Exit fullscreen mode 
      An empty hash diff proves that the file was unchanged between measurements. It does not prove correct browser behavior; that requires an independent state check.
Enter fullscreen mode Exit fullscreen mode

11. Verify independently

Page state

Run a read-only query outside the agent’s final response:

print(js("""
() => {
  const out = document.querySelector("#result");
  return {
    url: location.href,
    status: out.dataset.status,
    events: Number(out.dataset.events),
    text: out.textContent
  };
}
"""))
Enter fullscreen mode Exit fullscreen mode 
      After reset and the second call, the expected state is the local fixture URL, status="courier", and exactly one event. Record what the command actually returns.
Enter fullscreen mode Exit fullscreen mode

Changed files 

git status --short | tee evidence/git-status.txt
git diff -- fixture/index.html | tee evidence/fixture.diff
git diff --stat | tee evidence/diff-stat.txt
Enter fullscreen mode Exit fullscreen mode 
      The fixture diff must be empty. If the Browser Harness workspace resides outside this repository, initialize a separate repository there before the run or copy a sanitized helper diff into evidence/.
Enter fullscreen mode Exit fullscreen mode

Unknown option 

try:
    select_shipping_method("Drone delivery")
except ValueError as exc:
    print(type(exc).__name__, str(exc))
else:
    raise SystemExit("FAIL: unknown option was accepted")
Enter fullscreen mode Exit fullscreen mode 
      Repeat the state query afterward. The status and event count must remain unchanged. Rejecting bad input after a click is too late.
Enter fullscreen mode Exit fullscreen mode

Bypass review 

rg -n \
  'dispatchEvent|shipping-change|dataset.*=|fixture/index.html|https?://' \
  workspace/agent_helpers.py \
  | tee evidence/bypass-review.txt
Enter fullscreen mode Exit fullscreen mode 
      A match is a review lead, not automatic proof of misconduct: reading dataset for verification is legitimate. Writing the expected status or dispatching the application event manually is not.
Enter fullscreen mode Exit fullscreen mode

Evidence checklist 

Date:
Operating system:
Chrome/Chromium version:
Browser Harness version:
Agent and model:
Helper path:
CDP connection: PASS / FAIL
Initial state: PASS / FAIL
First Pickup action: PASS / FAIL
shipping-change event: PASS / FAIL
Reset: PASS / FAIL
Second Courier action: PASS / FAIL
Helper unchanged before reuse: PASS / FAIL
Unknown option rejected: PASS / FAIL
Fixture unchanged: PASS / FAIL
Core unchanged: PASS / FAIL
External navigation observed: yes / no / unknown
Final result: PASS / PARTIAL / FAIL
Primary limitation:
Enter fullscreen mode Exit fullscreen mode 
      Keep command output, hashes, and diffs as the audit trail. Screenshots are useful supporting evidence, but they do not prove which event fired or which code produced the state.
Enter fullscreen mode Exit fullscreen mode



  1. Failure cases and diagnosis

Chrome accepts no connection 

      Check the local debugging endpoint with /json/version, confirm the selected port, and verify that no old process owns the profile. Do not expose the endpoint publicly or add an unlimited reconnect loop.
Enter fullscreen mode Exit fullscreen mode

The harness controls the wrong tab 

      Compare URL and title immediately before every action. Do not infer identity from target order. Open the fixture in a new tab rather than overwriting an existing user tab.
Enter fullscreen mode Exit fullscreen mode

The accessibility tree does not contain the button 

      Wait for custom-element registration and request a fresh tree. Browser versions may expose closed component internals differently. If the button remains unavailable, document that dependency and use another CDP inspection strategy; do not weaken the fixture merely to make the test pass.
Enter fullscreen mode Exit fullscreen mode

The agent changes the fixture 

      Mark the run FAIL, preserve the diff, restore the committed fixture, and repeat with stricter filesystem permissions. Correct-looking output from a modified test page is not evidence.
Enter fullscreen mode Exit fullscreen mode

The helper changes output directly 

      This is state forgery, not browser automation. Require the normal component event and inspect the code for writes to #result, dataset, or manual shipping-change dispatch.
Enter fullscreen mode Exit fullscreen mode

Coordinates work once and fail after reload 

      The helper probably cached geometry. It must discover the node and calculate its box immediately before each click. Window size, zoom, fonts, scrolling, and content can all move the target.
Enter fullscreen mode Exit fullscreen mode

The first run works but a fresh process cannot find the helper 

      The function may exist only in interactive memory or in a workspace the CLI does not import. Confirm the actual helper path and document how the harness loads it.
Enter fullscreen mode Exit fullscreen mode

The second run edits the helper again 

      This is adaptation, not reuse. Compare hashes and diff. The contract may be too narrow, or the agent may have ignored an existing capability.
Enter fullscreen mode Exit fullscreen mode

The event count becomes two 

      Look for double clicks, duplicated input dispatch, or persistent listeners installed during exploration. One requested selection should cause exactly one application event in this fixture.
Enter fullscreen mode Exit fullscreen mode

An unknown label selects the first option 

      Fail before interaction. Fuzzy matching is inappropriate for consequential actions unless ambiguity is explicitly surfaced and confirmed through an approval gate.
Enter fullscreen mode Exit fullscreen mode

The agent reports success without evidence 

      Ignore the claim and run the independent checks. A confident statement may be a hallucination, a misread state, or a description of intended work rather than completed work.
Enter fullscreen mode Exit fullscreen mode

  • Limitations

    • A closed Shadow DOM control does not represent every unusual browser interaction.

    • Two successful calls demonstrate basic reuse, not production reliability.

    • The lab does not cover CAPTCHA, anti-bot controls, network instability, server races, downloads, or cross-origin frames.

    • Accessibility and CDP details may change across browser versions.

    • Coordinate-based input is closer to user behavior but less stable than a supported application API.

    • The test does not compare models and must not be presented as a model benchmark.

    • A helper file can accumulate dead code, name collisions, and hidden dependencies without ownership and cleanup.

    • Connecting CDP to a real employee profile creates a much larger threat surface than this isolated lab.

    • Filesystem restrictions alone do not prevent all network or browser-side effects.

    • A generated helper still requires review before it becomes shared infrastructure.

        A dedicated profile and writable directory are not a complete sandbox. Production use needs operating-system or container isolation, outbound network controls, secret separation, action-level permissions, and retention rules for recordings.

  Once a helper has survived several independent runs, move it into a versioned library, assign an owner, document supported page states, and add it to a small benchmark of successful and negative browser tasks. The editable harness should remain an adaptation layer, not an unreviewed permanent codebase.

    14. Decision rule

    Call the experiment successful only when all of the following are true:

    • The agent controlled an isolated Chrome instance through CDP and stayed on the local fixture.

    • The existing high-level helpers were genuinely insufficient.

    • The agent added one bounded function only to agent_helpers.py.

    • The function produced a real click and the component emitted its normal event.

    • The fixture, harness core, and expectations remained unchanged.

    • After reset, a fresh process invoked the same helper for another valid option.

    • The helper’s hash did not change during reuse.

    • An unsupported label failed before interaction.

    • Page state, diff, hashes, and command output independently confirmed the result.

        If a developer had to repair the helper, the run still demonstrates that the harness can accept new browser primitives, but it does not demonstrate autonomous extension. If the helper uses fixed coordinates or bypasses the application event, classify the result as PARTIAL.

  The next useful experiment is not a more complicated prompt. Repeat the entire run in a clean profile, then present a second component with the same user-level operation but different layout. If the helper transfers unchanged, it is becoming reusable automation. If every page requires fresh code, the harness is still valuable as an exploration accelerator, but not yet as a self-growing tool library.

  The important capability is not that an agent can write another script. It is that the discovered interaction becomes visible, bounded, testable, and callable again.

  ← Browse all guides
  ·
  Open the glossary →

      Agent Lab Journal
      Practical experiments with verifiable outcomes.

    Original article: https://agentlabjournal.online/en/browser-harness-protiv-tradicionnoy-avtomatizacii.html?utm_source=devto&utm_medium=referral&utm_campaign=agentlabjournal-en-global-all&utm_content=article&utm_term=browser-harness-protiv-tradicionnoy-avtomatizacii

    Top comments (0)