If you have ever dropped a markdown "skill" file into ~/.claude/skills/ and watched Claude completely ignore it, you are not imagining things. The file sits there, but nothing tells the model when it is relevant — so it never fires.
We had exactly that problem. CrawlForge shipped reference-markdown files that were supposed to teach Claude how to use our 26 web-scraping tools. They sat on disk. Claude never actually loaded them. You still had to spell out "use the stealth_mode tool to scrape this" — which defeats the point.
CrawlForge MCP Server v4.8.0 (shipped June 28, 2026) fixes that. It is a fully additive minor release: the tool count stays 26, and no tool schema, output shape, or credit cost changes for existing callers. Here is what changed.
Table of contents
- Before / after
- 7 Claude Agent Skills that actually auto-activate
- Two new scrape formats: branding and screenshot
- Built-in scheduled change monitoring
- Security: controls that were advertised but silently broken
- Pricing and how to upgrade
Before / after
BEFORE: "Use the crawlforge stealth_mode tool to scrape this protected page."
+--> you name the tool, every time
AFTER: "Scrape this Cloudflare-protected page for me."
+--> Claude matches the prompt, loads the stealth-browsing skill,
and calls the right tool on its own
One behavior change to flag up front, Stripe-changelog style: clients that support MCP elicitation will now see cost and safety confirmations that previously failed to appear (more on that below). Everything else is purely additive.
Upgrade in one line, then run init:
npm install -g crawlforge-mcp-server@latest
npx crawlforge init
Added: 7 auto-activating Agent Skills, branding + screenshot scrape formats, scheduled change monitoring, list_scheduled_monitors.
Fixed: SSRF now enforced on the live path, MCP elicitation now fires, screenshots actually render.
Security: per-host rate limiting, executeJavaScript hardening, DNS-rebinding protection.
7 Claude Agent Skills that actually auto-activate
A skill is a folder containing a SKILL.md file with YAML frontmatter. At startup Claude pre-loads only the name and description of every installed skill, then reads the full body only when it judges the skill relevant to your prompt. Anthropic calls this progressive disclosure — skills are not always-loaded context, they are loaded on demand.
The catch most people miss: auto-activation is model-judged, not guaranteed. The quality of that description field is what decides whether the right skill fires. Here is the frontmatter shape we now ship:
---
name: crawlforge-stealth-browsing
description: ">-"
Scrape sites protected by Cloudflare, anti-bot, or JavaScript
rendering. Use when a normal fetch is blocked, returns a challenge
page, or the content is rendered client-side. Triggers on:
"bypass bot detection", "Cloudflare", "headless", "stealth".
---
Trigger-rich descriptions dramatically raise the probability the right skill fires. v4.8.0 ships seven skills covering all 26 tools:
crawlforge-web-scrapingcrawlforge-deep-researchcrawlforge-stealth-browsingcrawlforge-structured-extractioncrawlforge-change-trackingcrawlforge-batch-automationcrawlforge-getting-started
They install to the personal scope at ~/.claude/skills/<name>/SKILL.md, and because activation is a heuristic, you can always name the skill or tool explicitly in your prompt to force it.
Upgrades self-heal: the installer removes the legacy bare reference files and leaves unrelated skills untouched. Want to push activation even harder? There is an opt-in forced-eval hook — an idempotent It is OFF by default.How skills relate to MCP (they complement, not replace)
MCP and skills solve different halves of the same problem. MCP exposes the 26 tools to Claude. Skills teach Claude when and how to use them — think of MCP as giving a new hire access to the tools, and skills as the onboarding guide that explains which tool to reach for.
npm run skills:gen regenerates the root SKILL.md.UserPromptSubmit reminder:
npx crawlforge init --with-hook # or: install-skills --with-hook
npx crawlforge uninstall-skills --remove-hook
npx crawlforge init does the whole setup: configures your API key, installs the skills, and registers the MCP server with your AI clients.
Two new scrape formats: branding and screenshot
The scrape tool gained two formats. Both cost the same 2 credits as before.
branding extracts design tokens statically from HTML + CSS with no browser required — color palette, fonts and typography, logo and favicons, plus border-radius, shadow, and spacing tokens. It is SSRF-guarded, and linked-CSS fetches are count- and size-capped.
{
"url": "https://stripe.com",
"formats": ["branding"]
}
screenshot now actually works (it was a no-op before). It lazily renders through the shared browser pool and returns crawlforge://screenshot/{id} MCP resources. The browser launches only when you request a screenshot, and a render failure degrades to a warning so partial success is preserved.
Built-in scheduled change monitoring
track_changes (3 credits) gained real scheduling operations — create_scheduled_monitor, stop_scheduled_monitor, and a new list_scheduled_monitors. These were previously dead code that threw. They are now backed by a persisted scheduler (MonitorScheduler.js + MonitorStore.js), and baselines rehydrate from snapshots on restart.
You can attach an optional plain-English goal that is LLM-judged (Ollama-first) and degrades gracefully to threshold significance when no LLM is available — e.g. "alert me only when the pricing changes, not when the copyright year flips."
The honest firing model (read this before you rely on it)
A stdio MCP server is not a long-lived daemon — it only runs while your client has it open. So reliable scheduled firing uses the CLI plus system cron:
crawlforge monitor:create --url https://api.example.com/changelog
crawlforge monitor:list
crawlforge monitor:stop <id>
# system cron runs this; it guarantees due monitors fire:
*/30 * * * * crawlforge monitor:run-due
monitor:run-due is a one-shot designed for cron. We would rather tell you this than pretend a stdio process magically wakes itself up.
See the track_changes docs for the full schema.
Security: controls that were advertised but silently broken
MCP servers are now a recognized SSRF attack surface. A web-scraping server that fetches arbitrary URLs on your behalf is exactly the shape of thing an attacker points at the cloud-metadata endpoint (169.254.169.254) to read credentials. We audited our own posture and found two controls that were advertised but silently non-functional. Both are fixed.
SSRF is now enforced on the live path. A robust ssrfProtection.js existed in the codebase but was never wired into the tools — every scrape used raw fetch() with zero IP/host validation. A new ssrfGuard.js injects an undici dispatcher whose connect-time lookup validates every connection — the initial request and every redirect hop — and pins to the validated IP, closing the DNS-rebinding TOCTOU window.
# Stage 1 (default): blocks loopback, link-local / cloud-metadata
# (169.254.169.254), and 0.0.0.0
SSRF_STRICT=true # adds full RFC1918 / ULA private-range enforcement
ALLOWED_DOMAINS=internal.corp # trusted-host bypass
SSRF_PROTECTION_ENABLED=false # kill switch
It is routed through roughly 14 modules — the basic fetch path, batch scrape, map_site, crawl, extract, process_document, research, llms-txt, robots/sitemap, and the track_changes differ.
MCP elicitation now actually fires. The old code called server.elicit() — a method that does not exist — and never checked the client capability, so every cost/safety confirmation (for example, deep_research over 50 URLs, batch_scrape, and crawl_deep) silently fail-opened. It now uses elicitInput, gates on the client's elicitation capability, and parses the accept / decline / cancel action. It still fail-opens for clients that do not support elicitation — but capable clients will finally see the prompts.
Two more layers of defense-in-depth:
-
Per-host rate limiting (
hostRateLimiter.js): default 10 req/s per host viaRATE_LIMIT_PER_DOMAIN, with no global cap, so broad multi-host crawls are unaffected. This is politeness/defense, not an SSRF boundary itself. -
executeJavaScripthardening (still OFF by default):JS_MAX_SCRIPT_LENGTH,JS_EXECUTION_TIMEOUT_MS, and a structured stderr audit log (script sha256 + length + url).
Pricing and how to upgrade
No new tools — the new capabilities were added to existing ones, so the count stays 26. All 26 are metered and require an API key; costs range 1-10 credits per call. Heads up: list_ollama_models is now 1 credit (it used to be free), so nothing is free per-call.
| Plan | Price | Credits |
|---|---|---|
| Free | $0 (no card) | 1,000 one-time trial (does not reset) |
| Hobby | $19/mo | 5,000 |
| Professional | $99/mo | 50,000 |
| Business | $399/mo | 250,000 |
Every plan gets every tool. LLM extraction defaults to local Ollama, so you do not need an OpenAI or Anthropic key unless you opt in.
# new users
npm install -g crawlforge-mcp-server && npx crawlforge init
# existing users
npm install -g crawlforge-mcp-server@latest # or an /mcp reconnect
Re-running init (or install-skills) gets you the 7 skills and self-heals the legacy bare files. v4.8.0 is additive, so nothing breaks. This continues the 4.7.x correctness work: 4.7.2 live-audited all 26 tools (scrape_with_actions, extract_structured, resources/read), 4.7.1 fixed deep_research credibilityThreshold and generate_llms_txt emitting literal "undefined", and 4.7.0 moved to a fully metered model.
That is the whole release: skills that fire, controls that enforce, screenshots that render. Try it, break it, and tell us what does not work — that feedback is what drives this cadence.
npm: crawlforge-mcp-server · Best MCP servers for web scraping in 2026
Top comments (0)