Robotex: a real browser for your bots, behind a plain HTTP API

Every bot that works with a website sooner or later hits one of three walls:

1. a captcha;
2. anti-bot protection (Cloudflare's "Just a moment…" page, Turnstile, reCAPTCHA);
3. heavy JavaScript that has to run before you can even submit a form.

A spider built on raw HTTP requests is helpless here. You need a real browser. But a browser is heavy: it eats half a gigabyte of RAM per window, it has to be hidden from detection, and it has to live next to your bot. Most of the time, though, the bot doesn't need the browser at all. It only needs it to get through the door.

That's the idea behind Robotex: a browser-as-a-service that you call only where there's no way around it.

The idea: the browser opens the door, the bot walks in

Robotex opens the page in real Chrome, passes the anti-bot checks, runs a declarative scenario (fill, click, wait, grab HTML or text) and returns:

• the cookies, including cf_clearance;
• the browser's own User-Agent;
• optionally, the rendered HTML and a screenshot.

From there your bot keeps working with a plain HTTP client, which is fast and cheap. When the site shows a challenge again, the bot calls Robotex once more in the same session.

It's a service, not a library, so the bot can be written in anything that speaks HTTP.

A first request

The synchronous endpoint is the simplest way in:

POST https://app.robotex.sokolab.xyz/v1/solve
X-API-Key: <your key>
Content-Type: application/json

{
  "url": "https://example.com/login",
  "proxy": "http://user:pass@host:port",
  "captcha_type": "cloudflare",
  "actions": [
    {"type": "fill", "selector": "#email", "value": "[e-mail]"},
    {"type": "fill", "selector": "#password", "value": "..."},
    {"type": "click", "selector": "#submit"},
    {"type": "wait_for", "selector": ".dashboard", "timeout": 30}
  ]
}

{
  "status": "ok",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) ... Chrome/155.0.0.0 Safari/537.36",
  "cookies": [{"name": "cf_clearance", "value": "...", "domain": ".example.com"}],
  "actions": [
    {"type": "fill", "ok": true, "duration_ms": 41},
    {"type": "fill", "ok": true, "duration_ms": 37},
    {"type": "click", "ok": true, "duration_ms": 120},
    {"type": "wait_for", "ok": true, "duration_ms": 1830}
  ],
  "duration_ms": 9120
}

The response also sets a session_id cookie. That's a persistent Chrome profile on the server: cookies, local storage and a browser fingerprint. Send it back next time, and to the site you're still the same visitor, so the anti-bot check usually doesn't come back.

Using the result in Python

The important part is what you do after the browser:

import httpx

ROBOTEX = "https://app.robotex.sokolab.xyz"
PROXY = "http://user:pass@host:port"

robotex = httpx.Client(base_url=ROBOTEX, timeout=300, headers={"X-API-Key": "rbx_..."})


def solve(url: str) -> dict:
    data = robotex.post("/v1/solve", json={
        "url": url,
        "proxy": PROXY,
        "captcha_type": "cloudflare",
    }).json()  # session_id stays in robotex's cookie jar for the next calls
    if data.get("status") != "ok":
        raise RuntimeError(f"{data.get('code')}: {data.get('error')}")
    return data


solution = solve("https://example.com/projects")

site = httpx.Client(
    proxy=PROXY,                                     # the same proxy
    headers={"User-Agent": solution["user_agent"]},  # the same UA
    cookies={c["name"]: c["value"] for c in solution["cookies"]},
)
page = site.get("https://example.com/projects")
if page.status_code == 403:
    solution = solve("https://example.com/projects")  # same session, fresh cf_clearance

Two rules make this work: the same User-Agent and the same proxy. Cloudflare ties cf_clearance to the UA and usually to the IP. Change either and the cookie is worthless.

Lessons learned the hard way

Don't spoof the User-Agent

Early versions let clients pass their own user_agent. It seemed harmless: it's just a string. But you can only change the string. navigator.userAgentData, client hints and the engine's behaviour still belong to the real Chrome, and Cloudflare sees the mismatch. I measured it on Chrome 155 / Linux against a site behind Cloudflare:

User-Agent in the browser	Cloudflare
native (Linux Chrome 155)	passed 3 of 3, 7–9 s
Linux Chrome 147	failed 3 of 3
Windows Chrome 155	failed

So the parameter is gone (passing it returns 422), and the rule is: the service defines the browser's identity, the client adapts to the response. The same went for the window size. Each profile instead gets a plausible fingerprint (screen size and devicePixelRatio drawn from a real-world distribution) generated once and applied natively by Chrome, with no JS patches, since those broke the Cloudflare check.

Use real Chrome, not "Chromium with flags"

Robotex drives real Google Chrome through Patchright, an undetected Playwright fork without the CDP leaks anti-bots look for. Chrome runs headful under Xvfb, because headless mode is one more signal to hide.

Make the proxy's geo and the browser agree

Locale and timezone are derived from the proxy's exit IP. A German IP with a browser in America/New_York is a red flag. And no fingerprint will save you on a dirty IP: proxy quality matters more than any trick on the page.

Architecture v2: a queue between the API and the browsers

The first version ran Chrome inside the API process. When all browser slots were busy, a request simply got a 503. The current version splits it in two:

Client → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb)
            ▲                                         │
            └── GET /v2/tasks/{id} ◀── Redis ◀────────┘

• api has no browser at all: it queues tasks and serves results;
• RabbitMQ holds tasks until a worker has a free slot, so they wait instead of failing;
• workers run Chrome, several windows each;
• Redis keeps task statuses and results for a day. Proxy credentials and captcha keys never go there.

/v1/solve still exists and runs on top of the same queue, so old clients didn't change a line. New clients can go async:

POST /v2/tasks
X-API-Key: <your key>

{"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10}

→ 202 {"task_id": "0b9c…", "status": "queued", "url": "/v2/tasks/0b9c…"}

Then poll GET /v2/tasks/{id} (with the same session_id cookie) until the status is done.

Keeping the browser window open

The feature I like most in v2 is browser_idle_timeout. With it, the session's Chrome window stays open for up to 120 idle minutes after the task. The next task with the same session_id is routed to the personal queue of the worker holding that window and runs right there: no Chrome start, no second anti-bot check. Leave out the url, and the actions run on the page that's already open.

That makes multi-step scenarios natural: log in, let the bot do its HTTP work, come back and continue filling the same form. When you're done, DELETE /v2/sessions/current closes the window and frees the slot.

What else it does

• Captchas inside pages: the captcha step solves Turnstile, reCAPTCHA and hCaptcha through your own 2captcha key before the form is submitted.
• Anti-bot pages: a detect → solve → verify loop for Cloudflare, before and during the scenario. Turnstile passes reliably.
• Secrets stay out of logs: proxy credentials, captcha keys, fill values and cookie values are masked.

Try it

Robotex is free. You only pay for your own proxies and, if you need it, 2captcha.

• Landing: robotex.sokolab.xyz
• API and Swagger: app.robotex.sokolab.xyz/docs
• API key: a test key for the demo instance is in the project boards
• Source code: git.sokolab.xyz/s0k0l/robotex
• Questions and bugs: Redmine forum and issues

Next on the roadmap: workers on several hosts, a second engine (Firefox via Camoufox) for sites where a Chrome fingerprint doesn't fit, and a bot that hands out personal API keys.

If you've fought Cloudflare from a scraper, I'd love to hear what worked for you, and what broke. And please use Robotex only on sites where you have the right to do so.