CrawlForge v5.0.0: Security, Correctness, MCP Spec

http://2130706433/ is a valid URL. Your browser will happily resolve it to 127.0.0.1, because the WHATWG URL parser normalizes decimal, hex (0x7f000001), and octal integer forms into dotted-quad IPv4.

Our SSRF guard did not know that. It resolved hostnames through DNS and range-checked the resulting addresses — but Node never routes an IP literal through lookup, so a URL whose host was already an IP sailed straight past the check. Loopback, link-local, cloud metadata: all reachable, in a server whose entire job is fetching URLs a model picked for you.

That is one bug out of the seven-phase internal audit that became CrawlForge MCP Server v5.0.0. The unit suite went from 480 tests to 914. npm audit went from 16 vulnerabilities to 0. Almost none of it is new features.

Table of contents

• What actually shipped
• The one breaking change: Node 20
• Phase 1: the SSRF bypass we shipped
• Phase 2: 52 ways tools were silently wrong
• Phase 3: safe to run for days
• Phase 4: HTTP mode only ever had one session
• Phase 5: zero npm audit vulnerabilities
• Phase 6: MCP spec adoption
• Pricing and how to upgrade

What actually shipped

Phase	Theme	Headline result
0	Dependency currency	npm audit 16 vulns → 4 moderate, zero code change
1	Critical security	SSRF IP-literal bypass, OAuth token minting, secret leakage, billing
2	Correctness	52 fixes — including a crawl_deep rewrite
3	Leaks and timeouts	24 fixes — browser contexts, unbounded caches, real deadlines
4	HTTP transport	19 fixes — multi-session streamable HTTP, working prompts, webhook HMAC
5	Dependency modernization	Node ≥ 20 floor, 0 npm audit vulnerabilities
6	MCP spec adoption	Structured output, async tasks, tool whitelisting, registry server.json

MCP protocol compliance held at 100.0% COMPLIANT, 0 errors at every phase gate.

The one breaking change: Node 20

engines.node moved from >=18.0.0 to >=20.16.0.

Node 18 hit end-of-life in April 2025, and 20.16 is the floor required by pdf-parse 2.4.5 — the maintained ESM rewrite we needed to clear the last audit findings. Our Dockerfile (node:20-alpine) and CI (Node 22) already satisfied it.

That is the entire breaking surface. No tool schema, output shape, or credit cost changed, and the tool count stays at 27.

node --version   # must be >= 20.16.0

Phase 1: the SSRF bypass we shipped

Read this phase if you run any MCP scraping server near a private network.

BEFORE: url -> parse -> DNS lookup -> ipBlocked(resolved)?  -> fetch
                          |
                          +--> IP literal? no lookup happens.
                               guard never runs. request goes out.

AFTER:  url -> parse -> ipBlocked(literal host)? --------+
                     -> DNS lookup -> ipBlocked(addrs)? -+-> fetch
                     -> per-connect check in the undici dispatcher
                        (catches every redirect hop too)

v5.0.0 runs ipBlocked() on IP-literal hostnames at pre-flight and wraps the undici dispatcher's buildConnector with a per-connect check, so a redirect hop straight to an internal address is blocked as well.

Three more guard fixes landed with it:

• IPv4-mapped IPv6. ::ffff:127.0.0.1 and ::ffff:169.254.169.254 are normalized to their embedded IPv4 before range checks, in both default and strict modes. Kills the DNS-controlled AAAA-record bypass.
• BLOCKED_DOMAINS was dead config. It was declared and read by nothing. It is now enforced at pre-flight.
• The allowlist is evaluated per hop. An allowlisted first hop used to unguard every redirect after it.

We also wired the guard into five paths that never had it: scrape_with_actions (with a post-navigation page.url() re-check that closes the page on a redirect into a blocked range — that was a Playwright internal-network read primitive), map_site, process_document PDF downloads, webhook delivery and health checks, and deep_research webhook notifications.

Beyond SSRF: OAuth, secret leakage, and billing

OAuth. /oauth/authorize now requires proof of the operator's API key before issuing a code, with constant-time digest comparison. The anonymous register → authorize → token flow that minted operator-billed bearer tokens is closed.

Secret leakage. Usage telemetry passes tool params through maskSecrets() before the payload leaves the process — third-party API keys, auth headers, and webhook signing secrets no longer travel in plaintext. deep_research stopped writing LLM API keys to Winston file logs.

Billing. A throw from the credit check itself now bills zero; the error-path half-charge only applies once the handler has actually started. checkCredits distinguishes 401/403 (invalid or revoked key) from 5xx (grace window) instead of reporting both as "insufficient credits."

If you want the general version of this problem rather than our specific one, we wrote it up separately: SSRF in MCP servers.

Phase 2: 52 ways tools were silently wrong

This is the "passes smoke tests, returns misleading output" class — the one that never shows up as an error in your logs.

crawl_deep is usable for real crawls again. BFS child pages were awaited from inside an occupied queue slot, so the per-task queue timeout bounded the entire recursive crawl rather than one page. Any crawl outliving the 30-second timeout threw away every page it had already fetched with a bare Promise timed out, and low concurrency settings (including concurrency: 1) deadlocked outright. Both fixed.

A representative sample of the other 51

• Cache keys that contradicted the request. crawl_deep's result-cache key now covers extract_content, content length, include/exclude patterns, follow_external, respect_robots, concurrency, domain filter, and session. map_site's covers search, domain filter, include_metadata, and group_by_path. Previously a cached call could contradict your parameters for a full hour-long TTL.
• Character encoding. Bodies decode with their declared charset (Content-Type header or <meta charset> sniff) instead of always UTF-8. No more U+FFFD soup from ISO-8859-1 or Shift_JIS sites.
• Silently stripped options. The options schemas for extract_content, summarize_content, and analyze_content now use .passthrough(). Every documented option key was being stripped before it reached the handler — which is also why summarize_content always returned the same 2-sentence fallback mislabeled extractive. The extractive summarizer now actually runs, and summaryLength changes the output.
• Link resolution. extract_links resolves relative hrefs against the final page URL rather than the origin, honors <base href>, and classifies protocol-relative links as external. The same fixes landed in scrape's extractor, so the two finally agree.
• track_changes similarity. Now token-Jaccard over the content. It was previously Hamming distance between sha256 hex digests — so every trivial edit scored roughly 0% similar and fired a "moderate change" alert.
• search_web scoring. Partial ranking_weights deep-merge over the defaults instead of replacing them wholesale, so no more NaN final scores or silently disabled duplicate checks. The zero-result expansion retry is capped at one fallback instead of up to five billed backend searches.

Phase 3: safe to run for days

24 findings in the class that only surfaces in long-running processes.

Browser lifecycle. Closing a Playwright page does not close its context — so every scrape_with_actions call and every browser-rendered extract_content leaked one context until shutdown. Contexts are now closed alongside their page, and a failed page.goto (DNS error, timeout, blocked URL) tears down both instead of orphaning them.

Bounded caches. crawl_deep destroys its per-crawl CacheManager in a finally. Previously N crawls permanently leaked N caches of up to 1,000 full HTML documents each — every one of them re-running a JSON.stringify memory scan every 60 seconds, forever. Dropped instances are now GC-verified with a WeakRef regression test.

Deadlines on every body read. The abort timer stays armed through the body stream, so timeout finally covers a server that returns headers and then stalls. Chunk reassembly is single-pass — it was O(n²), roughly 1.5 seconds of synchronous event-loop block on a 25 MB body. PDF downloads got a real AbortSignal.timeout (the old timeout: fetch-init option is silently ignored by undici).

One for Claude Desktop users: snapshot storage defaults to ~/.crawlforge/snapshots instead of process.cwd(). MCP clients launch the server with a working directory of /, where every snapshot write silently failed.

Phase 4: HTTP mode only ever had one session

If you deployed over npm run start:http, it was worse than you thought. A single shared transport meant exactly one session ever existed, and any clean disconnect bricked /mcp until you restarted the process.

Stateful mode now follows the SDK's documented per-session pattern — a Map<sessionId, {transport, server}> with a fresh transport and cloned McpServer per initialize, disposal on DELETE, and a JSON-RPC 404 for unknown session IDs. Second concurrent client, reconnect after a network drop, DELETE then fresh initialize: all work now.

Also in Phase 4:

• The getting-started prompt was unretrievable by any client — the config object hit the SDK's positional argsSchema overload, advertising a bogus required argument and failing every prompts/get. The compliance suite now covers discovery and retrieval for all 6 prompts.
• Webhook HMAC signatures cover the exact serialized body that is POSTed. Only the data sub-object was being signed, so standard receiver-side raw-body verification failed every single time.
• scrape no longer inlines multi-megabyte base64 screenshots into the JSON result — it keeps metadata plus a crawlforge://screenshot/{id} resource URI.
• Auto-setup banners moved from stdout to stderr, so a first launch no longer injects non-JSON lines into the stdio JSON-RPC channel.

Phase 5: zero npm audit vulnerabilities

With the Node 20 floor in place, we retired every abandoned dependency and took the security upgrades the old floor had blocked. 4 moderate → 0.

Removed outright: node-cron (unused after Phase 3 moved scheduling to setInterval; removal cleared its vulnerable uuid chain), @googleapis/customsearch (unused — the Google adapter calls the REST endpoint directly), and node-summarizer (abandoned since 2019; the extractive summarizer was rewritten as a compromise-based Luhn-style word-frequency scorer with identical result shapes).

The upgrade that mattered most: pdf-parse 1.1.1 → 2.4.5. PDFProcessor was ported to the v2 class API, so the password option now actually decrypts protected PDFs — v1 silently ignored it.

On supply chain: this phase ran during the ChainDrop npm worm. Every install ran with --ignore-scripts, every adopted version was publish-date-gated, and the full lockfile diff was cross-checked against public compromised-package lists with zero matches.

Phase 6: MCP spec adoption

Structured output (MCP 2025-06-18). scrape, map_site, serp_rank, search_web, extract_structured, and crawl_deep declare an outputSchema and return structuredContent alongside the legacy JSON text. The schemas are permissive by design, so a legitimate result can never fail SDK output validation.

Async tasks. crawl_deep, batch_scrape, deep_research, and agent are registered with taskSupport: 'optional' under the io.modelcontextprotocol/tasks extension. Task-aware clients get a handle immediately and poll tasks/get; clients without task support still get the synchronous result exactly as before. This is the fix for long crawls timing out inside a client's tool-call window.

Client-side tool selection. Two env vars let you expose a subset of the 27 tools and cut context bloat:

# By name
CRAWLFORGE_TOOLS=scrape,search_web,extract_content

# Or by group — 12 available: basic, search, crawl, extract, batch,
# research, tracking, llmstxt, stealth, templates, scrape, agent
CRAWLFORGE_TOOL_GROUPS=search,extract

Unset means all tools. Unknown names are ignored with a stderr warning, and batch_scrape auto-enables get_batch_results.

Protocol hygiene. Schemas advertised in JSON Schema 2020-12 instead of draft-07. tools/list sorted deterministically for client prompt-cache stability. Invalid tool arguments come back as isError: true tool results — which a calling model can self-correct from — rather than -32602 protocol errors. And server.json is complete against the 2025-12-11 registry schema.

Missed v4.9.0 and v4.10.0? Two things landed in between

v4.9.0 added serp_rank, the 27th tool — real Google organic rank positions via DataForSEO, 5 credits per configured lookup. v4.10.0 made it return the full top-10 organic listing alongside your target domain's positions. Docs.

v4.10.0 also added server-level MCP instructions: the server tells any connecting client to prefer CrawlForge tools over its own built-in web capabilities. It ships in the server binary, so every client picks it up on the next launch after upgrade — no re-init. It is guidance, not enforcement; an MCP server cannot disable a client's built-in tools.

Pricing and how to upgrade

Nothing changed. All 27 tools are metered and require an API key, at 1-10 credits per call.

Plan	Price	Credits
Free	$0 (no card)	1,000 one-time trial (does not reset)
Hobby	$19/mo	5,000
Professional	$99/mo	50,000
Business	$399/mo	250,000

Every plan gets every tool. LLM extraction defaults to local Ollama, so you do not need an OpenAI or Anthropic key unless you opt in.

# existing users
npm install -g crawlforge-mcp-server@latest   # or an /mcp reconnect

# new users
npm install -g crawlforge-mcp-server && npx crawlforge init

Because Phase 2 fixed tool behavior rather than tool contracts, your existing calls keep working — they just return correct results now.

Deferred rather than rushed: a hosted remote endpoint with OAuth, a keyless tier, scheduled monitoring as a service, persistent sessions, and PII redaction.

Writing up your own bugs is uncomfortable, but "advertised control, silently non-functional" is the single most common failure mode we found across all seven phases — and it is invisible from the outside. If you find a CrawlForge control that does not behave the way the docs claim, that is exactly the bug we want.

npm: crawlforge-mcp-server · full v5.0.0 writeup · v4.8.0 release post

Start free with 1,000 credits