"What browser MCP to bypass Cloudflare?" is a live question on r/ClaudeAI, and the honest answer for most of 2026 was: none of them reliably, ours included.
In mid-September a bot-detection benchmark we ran against CrawlForge MCP v6.6.2 recommended routing through residential proxies. We configured it. It did nothing. Finding out why turned into six releases in eleven days, and this is the record of what they changed for anyone who needs to scrape a Cloudflare-protected site from an MCP server without pretending to be something they are not.
The plain fetch still runs first and still costs 2 credits. What changed is everything that happens after it is refused.
Table of contents
- What shipped, release by release
- Why a plain fetch fails
- One call, three tries
- A solved challenge stays solved
- The agent retries walled pages itself
- Bring your own proxies, and they route now
- Measured, not claimed
- What we will not do
- Running it for clients
- How to upgrade
What shipped, release by release
| Version | Date | The one-line version |
|---|---|---|
| v6.7.0 | 2026-09-16 |
proxyRotation routes traffic for real; Camoufox stops announcing itself as Chrome |
| v6.8.0 | 2026-09-22 | The stealth benchmark becomes npm run bench:stealth; fingerprint leaks closed; "auto" engine defaults to Camoufox |
| v6.9.0 | 2026-09-22 |
agent retries a walled page in the stealth browser, 8 credits + 5 per retry that gets the page |
| v6.10.0 | 2026-09-25 | Clearance jar: cf_clearance, __cf_bm and datadome cookies replayed to the next stealth context |
| v6.11.0 | 2026-09-26 | A Chrome TLS try (impit) before any browser launches; Turnstile checkbox click on Chromium; escalation audit rows |
| v6.12.0 | 2026-09-27 | App-error fallbacks caught as soft blocks; CRAWLFORGE_IMPIT=off per deployment |
No tool was added or renamed, and no price changed except the agent ceiling, which now depends on what got through. Every line above is drawn from the server's changelog.
Why a plain fetch fails
Cloudflare scores a request before the page is served, and a Node HTTP client fails that score in three places at once: the IP is a known datacenter range, the TLS handshake does not look like a browser's, and the interstitial needs JavaScript the client will never run. The result is a 403, or a 200 carrying nothing but a challenge shell.
Since v5.6.11, scrape has reported that wall as success: false with blocked.vendor naming Cloudflare, DataDome, PerimeterX, Akamai, Amazon or Vercel, whatever the HTTP status, and charged nothing for it. Since v5.9.0, escalate: true has let the same call render the page once in the stealth browser instead of returning the block. The six releases below are what that escalation stage turned into.
One call, three tries
With escalate: true and the default engine of "auto", one scrape call now climbs a ladder, and stops at the first rung that returns the page:
-
The plain fetch, with the
CrawlForge/<version>User-Agent. 2 credits. If the host walled a request in the last 24 hours, the MCP server skips this rung rather than repeat a doomed fetch. -
A Chrome TLS handshake through
impit, new in v6.11.0. It presents Chrome's TLS fingerprint and still carries the honest CrawlForge User-Agent. No browser starts. From a residential IP where Cloudflare blocks the plain fetch, indeed.com came back in about a second. -
The stealth browser, Camoufox first and Chromium with a warning when the Camoufox binary is absent (v6.8.0). The clearance jar is applied here, and on Chromium a wall that is still up after the usual wait, with a
challenges.cloudflare.comframe on the page, gets one click on the Turnstile checkbox (v6.11.0).
The price is 2 + 5 whichever of the last two rungs got the page, and a call the plain fetch served still pays 2. A page that meets the wall on every rung comes back as a block carrying escalated: true and is charged nothing.
One call with the TypeScript SDK
// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';
const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });
// One call. The plain fetch runs first; only a wall triggers the rest.
const result = await client.scrape({
url: 'https://www.indeed.com/cmp/Burger-King/reviews',
formats: ['markdown', 'metadata'],
escalate: true
});
const data = result.data as {
escalated?: boolean;
stealth?: { engine: string };
content: { markdown: string };
};
// "impit" means the TLS rung got it and no browser ever launched.
console.log(data.escalated, data.stealth?.engine, data.content.markdown.length);
Two guards sit inside the impit rung. Every redirect hop is SSRF-checked with its own DNS resolution, because impit resolves names itself. And a page with under 200 characters of visible text goes on to the browser rather than being returned: quora.com answers a Chrome handshake with its client app's "Something went wrong" fallback under a perfectly normal title, and v6.12.0 now treats that fallback as a soft block on every path.
One deliberate difference: the hosted REST API's scrape escalation stays browser-only. Measured from the hosted instance's datacenter IP, the impit step cleared none of the benchmark's walls, so the hosted service runs with CRAWLFORGE_IMPIT=off while an npm install keeps it on.
A solved challenge stays solved
Before v6.10.0, every stealth render started cold. If Cloudflare's interstitial let the browser through, that clearance died with the browser context, and the next call to the same host solved it again.
The clearance jar keeps exactly three cookies from a render that got past a wall: cf_clearance, __cf_bm and datadome. They are replayed to the next stealth context with the same engine, User-Agent and proxy, which is the identity the clearance was issued to. That covers the scrape escalation stage, stealth_mode, the agent's stealth retry, browser_session and scrape_with_actions.
Measured on stackoverflow.com from a residential IP: the first stealth call went through Cloudflare's interstitial in 4.1 seconds, with the orchestrate and fo challenge requests visible in the trace. The second went straight to 200 with neither, in 1.6 seconds. A second process reused a clearance from disk and loaded indeed.com with zero challenge-platform requests.
The boundaries matter as much as the feature:
- No other cookie is ever kept. A session cookie from one caller's login cannot reach another caller's context, because the jar does not store it.
- A render that meets the wall again drops that host's clearances, so a stale clearance cannot loop.
- Expiry is the cookie's own, capped at 24 hours. The jar holds at most 32 identities with 200 cookies each.
-
It persists at
~/.crawlforge/stealth-clearance.json, mode 0600, keys hashed.CRAWLFORGE_CLEARANCE_JAR=offturns it off.
A persistent Chromium profile pool was considered for the same phase and deliberately not built. A shared profile keeps logins and site storage, and on a hosted instance that would leak between customers.
The agent retries walled pages itself
Before v6.9.0, the agent tool's act stage ran only the plain fetch. A challenged seed page was dropped and the answer was assembled from search snippets. In our own review that produced the sentence "3.3 stars, based on 3.3 reviews" for a Burger King page on Indeed, which is what happens when a rating leaks into the count field of a snippet.
Now a page that hits a challenge, a 403 or 429, an empty shell or a timeout is retried through the same escalation stage scrape uses. URLs the caller named are retried first. A discovered URL is retried only when it has no relevant search snippet. Refusals, 404s and 5xx errors are never retried, a run makes at most two retries, and none starts with under 20 seconds of wall clock left.
With the Chromium engine, the same Indeed test read the seed page itself and answered 3.3 stars from 58,942 reviews, with the evidence marked via: "stealth".
| Run | Credits |
|---|---|
| No retry needed, or every retry met the wall | 8 |
| One retry that got its page | 13 |
| Two retries that got their pages (the ceiling) | 18 |
The result reports stealth_retries (attempts) and stealth_retries_charged (billed). A retry that meets the wall again, or throws, is free.
Bring your own proxies, and they route now
CrawlForge supplies no proxies. What v6.7.0 fixed is that the ones you supply are actually used. stealthConfig.proxyRotation had been a no-op in three separate ways: the proxy went onto Chromium's --proxy-server flag, which cannot carry the user:pass every residential proxy is issued with, so authenticating proxies answered 407; Camoufox's launch path returned before the argument list was built, so the Firefox engine ran unproxied whatever was asked; and the list was read once at launch, so rotationInterval could never elapse.
All three are fixed. Proxies are ordinary URLs (http, https, socks4, socks5, credentials percent-encoded), applied per browser context so both engines authenticate, and a malformed entry is an error rather than a silently unproxied request. Camoufox now runs with its own geoip, block_webrtc and humanize features on, so behind a proxy it derives locale, timezone and location from the exit IP.
v6.8.0 added the server-level form, CRAWLFORGE_STEALTH_PROXIES: a comma-separated list used by every stealth path when a call passes none. A proxy on the call always wins.
The same release fixed a Camoufox identity problem that predated all of this. About two thirds of Camoufox contexts had presented a Chrome User-Agent on a Gecko engine and sent sec-ch-ua client hints Firefox has never implemented. A detector could act on that from the request headers alone, before a line of script ran. Camoufox now presents its own Firefox identity, with nothing Chromium-shaped injected over it.
Measured, not claimed
Every number in this post comes from npm run bench:stealth, which v6.8.0 turned from a hand-run table into a command. It drives the stealth browser and the plain fetch against twelve bot walls and five detector pages, and heads its matrix with the host OS, exit-IP class, engine and browser versions, because a Cloudflare result from a residential connection and one from a datacenter are different measurements.
Measured against it, the fingerprint leaks were closed: navigator.webdriver reads false instead of being deleted, nothing is an own property of navigator, userAgentData drops its HeadlessChrome brand, the Chrome major comes from the installed binary, and a Web Worker answers what the document answers. Detector self-probes went to 19 pass, 0 fail, 1 skip on both engines. npm run bench:stealth:ci runs ten of those probes with no third-party site and fails only on a regression, with a negative control that forces navigator.webdriver to true and must be caught.
Two findings worth carrying with you:
-
Which engine passes depends on the exit IP. From a residential connection Camoufox cleared indeed.com where Chromium failed. From the hosted instance's datacenter address two runs found the exact reverse. That is why
CRAWLFORGE_STEALTH_ENGINEexists to pin a deployment, and why the npm default still prefers Camoufox. - A single DataDome cell is not a result. leboncoin on Chromium passed on one run and was blocked an hour later from the same IP with nothing changed in between.
What is not closed is stated rather than glossed. Chromium still leaks HeadlessChrome from a SharedWorker, a target Playwright attaches nothing to. And the Turnstile click was verified against Cloudflare's forced-interactive test sitekey, which proves the mechanism only; whether a real site accepts the click still depends on the IP and fingerprint it scores.
What we will not do
The searcher's word is "bypass". Ours is narrower, and these releases held the line on it:
- No CAPTCHA solving and no forged challenge tokens. The Turnstile click is a single click on a checkbox Cloudflare put on the page, on Chromium only. No token API is called.
-
The User-Agent is honest on every rung. The
impithandshake presents Chrome's TLS fingerprint and still identifies itself asCrawlForge/<version>. On indeed.com, the honest User-Agent passed 3 runs in 4 where a Chrome User-Agent passed 0, so the honest header was not the thing costing us the page. -
Clearance cookies are never moved between identities. Replaying a
cf_clearancecookie throughimpitwas declined, because the cookie is bound to the User-Agent it was earned with. -
robots.txt is checked before any rung runs, against the same
CrawlForgeproduct token the plain fetch uses. Arespect_robots: falseoverride is written to the compliance audit log. -
Every escalation is audited. Since v6.11.0, each escalation writes a
stealth_escalationrow tologs/compliance-audit.log, after the compliance gate and before the browser navigates, carrying the URL, the tool, the resolved engine and a truncated hash of the API key. A refused request writes none.
Running it for clients
If you run the server for several clients, the pieces above compose into one deployment:
The environment for a self-hosted, multi-client install
# Your residential or ISP proxies. CrawlForge supplies none.
CRAWLFORGE_STEALTH_PROXIES=http://user:pass@proxy-a:8000,socks5://user:pass@proxy-b:1080
# Pin the engine once you have measured which one passes from your exit IP.
CRAWLFORGE_STEALTH_ENGINE=chromium
# Leave the Chrome TLS rung on unless your exit IP is a datacenter range.
# CRAWLFORGE_IMPIT=off
# Clearances persist per engine, User-Agent and proxy; off if you would rather start cold.
# CRAWLFORGE_CLEARANCE_JAR=off
Then measure before you promise anything: npm run bench:stealth from the box that will do the work, so the matrix carries your exit-IP class and not ours. The audit log gives you a per-key record of every stealth escalation, and the clearance jar's three-cookie rule means one client's login never sits in a context another client's job reuses.
For the hosted API, the honest description is this: stealth traffic exits from a datacenter IP, the escalation is browser-only, and which walls that clears is a property of that IP. Where a client's targets sit behind IP-reputation checks, the self-hosted MCP server with your own proxy list is the setup that measures well.
How to upgrade
npm install -g crawlforge-mcp-server@latest
crawlforge --version # 6.12.0
An MCP client that launches the server with npx picks up v6.12.0 on its next restart. impit is an optional dependency with prebuilt binaries; when it is absent, escalation behaves exactly as it did in v6.10.0. Camoufox is optional too, needs Node 22 for its transitive dependencies, and "auto" falls back to Chromium with a warning that says so. No schema or output shape changed on any tool.
The package is crawlforge-mcp-server on npm, and the full release history is on the changelog. If you have a page that has been blocking you, the free plan's 1,000 one-time credits are enough to see which rung of the ladder clears it from your IP.
This article was written with the help of AI and reviewed by the author.
Top comments (1)