DEV Community: CloakHQ

A Green Run Is Not a Green Light: Engineering Responsibility in Browser Automation

CloakHQ — Sat, 05 Sep 2026 22:47:29 +0000

The browser agent had one goal: check a set of public pages and return a structured result.

The first page responded with 429 Too Many Requests. The agent retried immediately. Then it received a 403, opened a fresh browser session, and repeated the step. Its planner treated every refusal as another obstacle between the current state and the goal.

By the time a human looked at the run, a temporary failure had become a burst of useless traffic.

The browser worked. The agent followed its objective. The missing component was an operating policy that knew when to wait, when to stop, and when to ask for review.

Browser automation is becoming more capable, especially when an AI agent controls the loop. That progress makes one distinction increasingly important: a workflow's technical ability to run says nothing by itself about whether the workflow should run in a particular context.

A successful request proves one thing

When a page loads, we learn that the current combination of browser, network, account, and target accepted one request. We do not learn that repeated requests are welcome, that the account may be automated, that the data may be collected for any purpose, or that the same action remains appropriate at production scale.

Technical accessibility is only one layer. A real automation decision can also involve platform rules, account permissions, contractual commitments, data rights, operational impact, and applicable law. The relevant mix changes from one workflow to another.

This matters because browser automation creates unusually persuasive feedback. A normal response looks like approval. A stable session feels safe. A green dashboard suggests that the problem has been solved.

None of those signals carries that meaning.

The same confusion appears around robots.txt. The Robots Exclusion Protocol communicates crawler preferences, and responsible crawlers should evaluate it. The standard also states that these rules are not access authorization. A missing Disallow rule is not a universal permission slip, while a Disallow rule is not the same mechanism as authentication or a legal ruling. It is one signal among several.

HTTP responses require context too. A 429 normally asks the client to reduce its request rate. A 503 can indicate temporary overload or maintenance. A 401 or 403 should usually move an automated job out of its normal retry path. A challenge page may even return 200, which means status codes alone cannot define the policy.

The engineering mistake is treating every response as either success or a technical problem to defeat.

Put restraint in the orchestration layer

Good intentions do not control production traffic. Code does.

The orchestration layer should have an explicit response policy before the first request is sent. That policy needs request budgets, bounded concurrency, capped retries, backoff, and stop conditions. An AI agent needs the same boundaries around its tools. A model that can choose the next action should not be allowed to invent its own retry policy under pressure to complete a goal.

The exact rules depend on the application. The following Playwright example shows a deliberately conservative policy: respect Retry-After for 429, use bounded backoff for common transient gateway failures, and send 401 or 403 to manual review instead of retrying automatically.

import type { Page, Response } from "playwright";

const sleep = (ms: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, ms));

function retryDelayMs(retryAfter: string | undefined, attempt: number) {
  if (retryAfter) {
    const seconds = Number(retryAfter);
    if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);

    const date = Date.parse(retryAfter);
    if (!Number.isNaN(date)) return Math.max(0, date - Date.now());
  }

  const base = Math.min(30_000, 1_000 * 2 ** (attempt - 1));
  return base + Math.random() * Math.min(1_000, base * 0.25);
}

export async function gotoWithPolicy(
  page: Page,
  url: string,
  maxAttempts = 3,
): Promise<Response> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const response = await page.goto(url, { waitUntil: "domcontentloaded" });
    if (!response) throw new Error("Navigation returned no HTTP response");

    const status = response.status();

    if (status === 401 || status === 403) {
      throw new Error(`Manual review required after HTTP ${status}`);
    }

    const retryable = status === 429 || [502, 503, 504].includes(status);
    if (!retryable) return response;

    if (attempt === maxAttempts) {
      throw new Error(`Retry budget exhausted after HTTP ${status}`);
    }

    await sleep(retryDelayMs(response.headers()["retry-after"], attempt));
  }

  throw new Error("Unreachable");
}

This is an example policy, not a universal HTTP client. Some systems can refresh an expired authorization once after 401. Some known endpoints have documented behavior for particular 403 responses. Long Retry-After values should usually be handed to a scheduler instead of keeping a worker asleep. Challenge detection also needs application-specific checks because the response may look successful at the HTTP layer.

The important part is that these decisions are explicit. A new browser context, proxy, or agent plan should not silently reset the retry budget. The budget belongs to the job, not to one browser session.

The same principle applies to bandwidth. Request interception can remove images, fonts, or media when they are genuinely irrelevant to a data workflow. It can also break an application, invalidate a visual test, or change the behavior being measured. Resource blocking belongs in a workload-specific design decision, not in a universal definition of responsible automation.

Permission is not a status code

Operational restraint reduces harm, but polite traffic does not automatically make a workflow authorized. A well-paced job can still use the wrong account, collect data outside its approved purpose, or violate rules that apply to the service.

Teams need a short decision record before automation reaches production. It should identify the purpose of the job, the systems and data in scope, the basis for access, the rules that were checked, the expected request volume, and the owner who can stop or approve changes to the run.

For routine work, this can be a lightweight template attached to the repository or deployment ticket. Testing your own application, monitoring an authorized account, or running a documented public-data workflow should not require a courtroom simulation before every commit. The goal is to replace assumptions with a visible decision.

Higher-impact or ambiguous cases need the appropriate owner. That may be the service owner, security lead, data-protection contact, customer, or legal counsel, depending on the question. Code review can verify implementation, but it cannot create missing authorization.

The method matters as well. An official API, export, test environment, or documented integration may offer clearer expectations and lower operational cost. Browser automation remains valuable when those options do not exist or do not reproduce the real user flow. The choice should follow the purpose of the job and the constraints of the environment.

Scale deserves its own review. A workflow tested against ten pages can behave very differently across ten thousand. Concurrency, caching, scheduling, duplicate detection, and global retry budgets determine whether a technically small action becomes a significant load on someone else's infrastructure.

Reliability and responsibility live in different layers

A mature browser automation stack separates three kinds of decisions.

The runtime layer is responsible for browser behavior: launching consistently, preserving the required session state, reproducing the intended environment, and exposing familiar automation interfaces. A stable runtime removes accidental failures and makes results easier to diagnose.

The orchestration layer controls the job: queues, pacing, concurrency, retries, request budgets, challenge handling, and stop conditions. This is where operational restraint becomes executable rather than aspirational.

The organizational layer defines purpose and accountability: which systems and data are in scope, who has authorized the workflow, how results may be used, and who owns exceptions.

CloakBrowser belongs primarily in the runtime layer. It provides a source-modified Chromium environment for Playwright, Puppeteer, Selenium, and CDP workflows where conventional headless setups or JavaScript-level patches can become fragile. A more consistent browser gives teams a cleaner foundation for QA, monitoring, browser agents, brand verification, and public-web research.

It does not choose the target, define the request rate, grant account access, or decide what happens after the data is returned. Those decisions belong to the layers above it.

That separation is useful for vendors and operators alike. The vendor can be accountable for reliable technology, accurate documentation, and clear boundaries. The operator can be accountable for purpose, authorization, configuration, scale, and data handling. The organization can make those responsibilities reviewable instead of leaving them implicit.

Powerful tools do not weaken the case for professional standards. They raise the value of having those standards encoded in the system.

Two green lights

Engineering culture rewards the first green light: Can we make it work reliably?

Production automation needs a second one: Should this workflow run here, in this way, and at this scale?

The second question should appear in architecture, code, deployment review, and monitoring. It should survive a browser restart and remain visible to an AI agent pursuing an objective. It should tell the system when to slow down, when to stop, and when a human decision is required.

A green run proves capability. A production run also needs permission, restraint, and an accountable owner.

This article presents a general engineering perspective and is not legal advice. Requirements vary by jurisdiction, platform, data, and use case.

We Were Scoring 0.1 on reCAPTCHA v3. So We Rewrote Chromium

CloakHQ — Thu, 07 May 2026 10:43:36 +0000

We were automating sites behind Cloudflare, reCAPTCHA, DataDome. Tried every stealth tool we could find. playwright-stealth, patchright, undetected-chromedriver. Each one worked until it didn't, and when it broke, it broke silently: sessions just stopped working and you'd spend days figuring out why.

reCAPTCHA v3 kept returning 0.1. If you haven't worked with v3 before: it doesn't show you a challenge. It watches the session silently and produces a score. Below 0.5 means bot. We were getting 0.1, which means "bot, confidently."

We needed something universal. One tool that held up across all of them, not just the one you were fighting that week.

Why JS patches don't work long-term

Most stealth library works the same way: inject JavaScript into the page before it loads, spoof the values detection scripts look for. navigator.webdriver set to false. Fake plugin lists. Canvas overrides.

They all failed the same check.

It took some time to understand why, but once we saw it, it was obvious. JavaScript patches run inside the browser. The browser itself (its C++ internals, the TLS fingerprint it sends, the CDP protocol behavior, the way input events are structured at the engine level) is still stock headless Chrome. Detection systems compare what JavaScript reports against everything else: what the network stack says, what the GPU reports, what the audio context returns. When those don't match, you get flagged.

A JS patch puts a mask on a robot. The seams are visible because they're there.

The fix had to happen at a different level.

What we built

We patched Chromium at the source. Actual C++ changes compiled into the binary before it runs. Canvas, WebGL, audio context, fonts, GPU strings, screen properties, network timing, CDP input behavior. All modified before compilation.

Websites can't tell it from a real Chrome session, because at the engine level, it is one.

The first test run: reCAPTCHA v3 returned 0.9. We ran it again. Same result. Ran it on a fresh session. Same result. Then we started running it against everything that had been blocking us (Cloudflare, DataDome, FingerprintJS) and it went through them one by one, cleanly. Honestly, we expected it to help. We didn't expect it to work that well, that consistently. Sites that had been blocking us for months just... loaded.

That internal fix became CloakBrowser. We open-sourced it in February.

Drop-in replacement

Same API as Playwright and Puppeteer. Swap the import, nothing else changes.

# before
from playwright.sync_api import sync_playwright
pw = sync_playwright().start()
browser = pw.chromium.launch()

# after
from cloakbrowser import launch
browser = launch()

// before
const browser = await chromium.launch();

// after
import { launch } from 'cloakbrowser';
const browser = await launch();

The binary ships with 57 source-level C++ patches and auto-generates a random fingerprint seed on every launch. Each session looks like a different device.

For behavioral detection on top of the fingerprint, humanize=True replaces mouse movement, keyboard input, and scroll with patterns that match real user behavior: Bezier curves, per-character typing delays, realistic scroll acceleration. One flag, no code changes.

browser = launch(humanize=True)
page = browser.new_page()
page.locator("#email").fill("user@example.com")  # types character by character
page.locator("button[type=submit]").click()       # Bezier curve to click target

Test results

Verified against live services, May 2026:

Service	Result
reCAPTCHA v3	0.9 (server-verified)
Cloudflare Turnstile (managed + non-interactive)	Pass
FingerprintJS	Pass
BrowserScan	Normal (4/4)
ShieldSquare	Pass
`navigator.webdriver`	false at source level
CDP automation detection	Not detected

reCAPTCHA scores it as a normal browser, because at the engine level, it is one.

What it doesn't solve

Proxy reputation is separate. Datacenter IPs get hard-blocked on aggressive sites regardless of how clean the fingerprint is. Residential proxies, ideally ISP/static rather than shared pools, are still part of the stack.

And some advanced configurations still catch us. The arms race is real, and we'd rather say that than pretend otherwise.

Try it

No install needed to test:

docker run --rm cloakhq/cloakbrowser cloaktest

Runs the full stealth test suite against live detection sites from your machine. Or install directly:

pip install cloakbrowser

npm install cloakbrowser playwright-core

Binary downloads automatically on first run, around 200MB, cached locally. Works on Linux, macOS, Windows.

The repo is github.com/CloakHQ/CloakBrowser. If you try it and something still gets blocked, open an issue with the site. That's exactly the feedback that drives the next build.

CloakBrowser is free and open source (MIT). The compiled binary has a separate license: free to use, no redistribution.