DEV Community: Simon

We stopped trusting our test suite and pointed 28 tools at the real web

Simon — Wed, 26 Aug 2026 19:24:33 +0000

Every one of our amazon-product tests passed. The tool returned null for the price currency, null for the rating, null for every image, and the string "Brand: Amazon" where the brand should have been.

Nothing looked broken in CI, because the fixtures had been written to match the selectors instead of the site. The tests were checking that our code agreed with itself.

So over two days we ran all 28 tools of CrawlForge MCP against real sites — Wikipedia, Hacker News, Amazon, Shopify storefronts, npm, YouTube, GitHub, Reddit, an IRS PDF — and shipped six releases: v5.2.0 through v5.2.5, plus four releases of the shared crawlforge-extractors package.

This is the whole set, organised by what it does rather than by version number.

Why six passing tests meant nothing
Product data from the source
Reddit-wide search after PullPush went dark
Change tracking that actually fires
Content that was being deleted
Local LLMs finally work
Browser automation and the stealth wedge
Language detection and PDF tables
One copy of the extractors
How to upgrade

TL;DR — v5.2 is six releases shipped after a live sweep of all 28 tools. New: a shopify-product template reading a store's own JSON, templates that can fetch any machine-readable endpoint, responseTime on fetch_url, remote Ollama via OLLAMA_API_KEY. Fixed: an Amazon template returning nulls behind six green tests, price monitoring that never fired, scrape deleting visible content, Ollama never registered as an LLM provider, seven Playwright defects, a renderer leak, Reddit-wide search, and PDF tables returning [] for every document. 28 tools, no breaking changes, one price moved.

Why six passing tests meant nothing

The amazon-product failure is the clearest case. Six unit tests covered that template and all six passed, because every selector they exercised — a priceCurrency meta tag, #acrPopover .a-size-base, img.a-thumbnail-image — exists nowhere on Amazon today, and the fixture HTML had been written to contain them.

The pattern repeated:

scrape_with_actions had seven Playwright defects invisible to its suite, because its fake pages implemented whatever the executor called — including APIs Playwright does not have.
The youtube-video template read an attribute that appears nowhere on a watch page, while its fixture kept the test green.
process_document read a field the PDF library declares and never writes.

So the new suites were written to fail against the pre-fix code first: 15 of 24 for amazon-product, 8 of 10 for the crawl_deep cache scope, 7 for the scrape_with_actions Playwright APIs. The suite is now 1,140 tests with MCP protocol compliance at 100% across all 28 tools — but the number that matters is that every fix below was found by a real page.

Product data from the source

Four scrape_template templates changed, and three moved the same direction: stop parsing the rendered page, read the structured data the site already publishes.

Shopify's Dawn theme ships every price badge in the markup unconditionally and hides the inapplicable ones with component CSS. A scraper reading the DOM sees "Sold out" on a product with a hundred units in stock. Ask an LLM for a compare-at price on a product that has none and it invents one — in our testing, 27.99.

Shopify serves the same data as JSON at /products/<handle>.json on every storefront including custom domains, so the new template reads that:

{
  "tool": "scrape_template",
  "params": {
    "template": "shopify-product",
    "url": "https://shop.example.com/products/kelpie-bandana"
  }
}

Exact price, compare-at price, on_sale, currency, price range across variants, per-variant stock, options, images and tags — no HTML parsing, no LLM in the path.

Making that possible needed a change one level down: TemplateRegistry gained resolveUrl(url) to redirect the tool's single fetch and extractRaw(body, url) to parse a non-HTML response. The tool still owns the SSRF-guarded fetch.

npm-package moved the same way — npmjs.com answers plain fetches with 403, so it now reads the registry document. Weekly downloads are omitted rather than returned as null: they live on a separate endpoint, and the registry endpoint that carries them alongside package data is a search endpoint — ask it for left-pad and it answers pad-left. Returning another package's download count is worse than returning none.

And amazon-product was rebuilt from live captures of three pages. Currency comes from the hidden add-to-cart field, not a meta tag. Rating parses to a number from #acrPopover's title. Review count handles both "(198,594)" and "198,594 global ratings". Images drop Amazon's size token — the tokened URL is a 1 KB thumbnail, the same URL without it is the 16 KB original.

Reddit-wide search after PullPush went dark

If your Reddit pipeline started returning 429s this month, the fix is not a longer sleep, a rotating user agent or a proxy. We tested all three. Every request now returns:

{ "error": "Rate limit exceeded. This website does not provide free scraping resources for agents..." }

This is policy, not throttling. And it matters more than one dead endpoint, because Arctic Shift — the other public Reddit archive — cannot keyword-search across all of Reddit by design. Ask it for a query naming no subreddit and no author and it returns HTTP 400:

'query' query parameter requires one of: author, subreddit

So scoped searches still worked, and unscoped ones — where you know the phrase but not the community — had no backend at all.

reddit_search now serves that in two steps. Discovery: a site-restricted web search returns post URLs in relevance order, and post IDs come out of the /comments/<id>/ segment. Hydration: those IDs go to Arctic Shift in one batch lookup, returning the real post rows — score, comment count, subreddit, author, timestamp, full selftext.

These are archive rows, not search snippets. Discovery only supplies the addresses. Responses carry source: "web_discovery".

Two limits, both reported rather than hidden: after/before cannot apply on the discovery route, and the response says the filter was ignored instead of returning results that look filtered; and an unscoped comment search has no backend at all, so it asks for a scope rather than failing generically.

Change tracking that actually fires

track_changes had seven defects across this release, and the first meant price monitoring — the headline use case — did not work.

Price moves were scored by how much of the page they occupy. Significance was purely volumetric, so $19.99 → $29.99 and $19.99 → $99.99 both scored "minor". With notificationThreshold defaulting to "moderate", a monitor set up the obvious way never fired. Monetary amounts are now compared directly, and relative magnitude raises significance to at least "moderate", or "major" at 20%+. Only currency-tagged numbers count, so view counters do not fire.

customSelectors never scoped anything. It was read only inside section-level analysis, where it added hashes — so scoping made it worse. On an Amazon product page, scoping to the price block raised modified elements 456 → 3204 and payload 5.35 MB → 6.18 MB, on a page whose price had not moved.

structuralSimilarity lied in both directions — reporting 0 when it had not measured (a real score meaning "completely changed"), and never able to fall below 0.5, because the hierarchy half of the average compared an object that was initialised empty and never written.

Then 5.2.5 added three more, found while testing price tracking on Zillow and Newegg. Element-level analysis indexed a fixed tag allowlist — h1-h6, p, div, span, a — so a monitor scoped to address, td, li, tr or dd built a baseline of zero elements and could never report a change. A Zillow page scoped to ['address'] built a baseline of 0 elements from 9 matching nodes. A scoped compare also silently ran unscoped, and "Text content changed" appeared on compares that found no changes.

Content that was being deleted

Three scrape defects shared a root: the cleanup passes that strip invisible content were removing content that was visible.

Framework-streamed content was deleted. The hidden-content strip removed <div id="S:0" hidden> — where the Next.js App Router streams the rendered page. On a pricing page that wrapper is the whole visible page, so markdown came back empty and every price with it.

Script payload counted as page text. The bulk-removal guard sized elements with $('body').text(), which includes every inline <script>. On a Shopify storefront the denominator was 62,269 characters of which 4,295 was visible copy — so a wrapper holding the entire product section measured under the threshold and was deleted along with the price. The json path then had no price to extract, and the model invented one.

An LLM's schema echo was returned as data. The json format read only success from extract_with_llm and discarded the rest, so a schema document returned instead of page data reached callers looking like a clean extraction — well-formed JSON, and with no required fields it passed validation too.

And in 5.2.2, the markdown format stopped returning raw HTML. turndown-plugin-gfm converts a table only when its first row is entirely <th>; every other table hits the plugin's keep filter and is emitted verbatim. Real pages are full of layout tables — Hacker News is built out of them.

Local LLMs finally work

Ollama was never registered as an LLM provider. LLMManager registered only OpenAI and Anthropic, both gated on an API key. So on a machine running Ollama with no cloud keys, extract_structured skipped LLM extraction entirely and reported css_fallback — producing values like "$79.99$79.99" — and deep_research silently disabled query expansion, semantic ranking and synthesis. extract_with_llm has its own private client, which is why it worked and masked the gap.

Model routing now picks the best installed model instead of always llama3.2. Benchmarked against three live product pages with verified ground truth, gemma3:4b scored 18/18 at 1040 ms while llama3.2 scored 16/18 — and the failures are systematic: over five runs llama3.2 invented a compare-at price all five times. Parameter count did not predict accuracy; the 4B model beat both a 12B and a 20B.

Browser automation and the stealth wedge

Seven Playwright defects in scrape_with_actions, all invisible to the suite:

scroll toElement called scrollIntoView(), which does not exist on a handle or locator, so the branch threw every time it ran.
The wait action advertised enabled/disabled/stable but passed them to an API that rejects them.
The per-action Promise.race shared a deadline with the work it raced and won, replacing Playwright's real error with a bare "Action timeout" — and left a live timer per action.
Every recovery strategy sat behind retries > 0 while the schema defaulted retries to 0, so none could ever run.

Separately, stealth_mode create_page never closed its page, leaking one Chromium renderer per call until the instance ran out of memory. A wedged browser was then reused forever behind truthiness-only checks, and cleanup hung on protocol calls to the dead browser. There is now isConnected() corpse detection, cleanup racing closes against 5 s deadlines with a SIGKILL fallback, and an in-flight launch mutex.

Language detection and PDF tables

Five languages were undetectable. analyze_content filters franc's output against a map of language names, and five keys were ISO 639-2/B codes — chi, gre, ara, nor, msa — which franc, emitting ISO 639-3, never produces. A page written entirely in Chinese returned null. So did Greek, Arabic, Norwegian and Malay.

A second bug sat behind it: franc scores whichever script is most common, and a CJK page carries the usual run of English product names and code samples. Detection now short-circuits on a Han/kana/hangul share of at least 10% of letters — measured across live pages, Latin-script pages sit at 0% and genuine CJK pages at 24–51%.

PDF table extraction returned [] for every document ever passed to it. The route read TableResult.mergedTables, a field the PDF library declares but never writes; the tables live on result.pages[].tables. Against the arXiv "Attention Is All You Need" paper, the IRS W-9 and the IRS 1040, the field being read held 0 tables in every case while the per-page arrays held 7, 6 and 11.

One copy of the extractors

The MCP server and our REST API each carried their own copy of the template extractors, in two languages, with nothing detecting divergence — and it diverged twice in two days. amazon-product was repaired on the MCP side on 25 August while the REST copy kept returning a null rating until the 26th.

There is now one implementation, published as crawlforge-extractors, which both surfaces install. Two more behaviours moved there because one surface had them and the other did not: readBody (charset-aware decoding and a body-size cap — the REST API called response.text(), so Shift_JIS and GBK pages came back as mojibake) and structureSignature.

The alternative was a parity test that told us after the fact which copy was wrong. Deleting the second copy is cheaper than detecting drift in it.

How to upgrade

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 5.2.5

If your MCP client launches the server with npx, it picks up the release on the next restart. No tool was renamed and no output shape changed. The only price change is reddit_search, from 2 credits to 5 — it now spends the same upstream search call search_web does — so check your budget maths if you call it in a loop.

The package is crawlforge-mcp-server on npm, and the same fixes are already live on the hosted REST API.

Want to try any of this without setting up a server? The free plan's 1,000 one-time credits need no card, and the API reference covers all 28 tools.

Start free with 1,000 credits

Web Scraping API Credits That Don't Expire

Simon — Wed, 26 Aug 2026 19:17:02 +0000

Most web scraping APIs sell you credits, then quietly take back whatever you did not use. Burn 60% of your monthly allowance and the other 40% evaporates on the next billing date. If you are searching for web scraping API credits that don't expire, you have probably hit that wall already — so this post covers why expiry is the industry default, what it actually costs you, and how CrawlForge prices the alternative.

TL;DR: CrawlForge sells credits that don't expire: one-time packs from $3 per 1,000 up to $62.50 per 25,000, bought with no subscription, plus paid plans whose unused credits roll over month to month. Cancelling never wipes your balance, and the free 1,000 signup credits never refill — but never vanish either. Firecrawl's free 1,000 credits, by contrast, reset monthly and unused ones lapse.

Why Do Most Scraping API Credits Expire?

Because most scraping APIs are sold as subscriptions, and subscription accounting wants a clean monthly reset. A fixed allowance that expires makes revenue predictable, keeps no credit liability on the books, and turns every unused credit into pure margin. The industry term for that margin is breakage, and expiring credits are how vendors generate it.

The model is fine when your usage is flat. Scraping usage rarely is: it comes in bursts — a one-off dataset build, a competitor research sprint, a migration — followed by quiet weeks. On an expiring plan the burst months force an upgrade or overage fees, and the quiet months donate the allowance back to the vendor. Either way, you are paying for your peak month twelve times a year.

The CrawlForge Model

CrawlForge treats credits as something you bought, not something you rent. Four rules:

One-time credit packs never expire. Buy a pack in the dashboard — no subscription required — and the credits sit there until you spend them, whether that takes a week or a year.
Paid-plan credits roll over. On Hobby, Professional, and Business, all unused monthly credits carry forward and never expire while the subscription is active.
Cancelling never wipes your balance. Unused credits survive cancellation.
The free tier is one-time, not monthly. Sign up and you get 1,000 credits with no credit card required; they never refill, and they never vanish either.

The packs:

Pack	Price	Per 1,000 credits
1,000 credits	$3	$3.00
5,000 credits	$14	$2.80
10,000 credits	$27	$2.70
25,000 credits	$62.50	$2.50

For bursty workloads this is the whole pitch: buy exactly the burst, spend it on your own schedule, and pay nothing in the quiet weeks. Paid plans add optional auto-recharge — a top-up that fires only when the balance runs low — and the full plan grid is on the pricing page.

What 1,000 Credits Actually Buys

CrawlForge meters all 28 tools per call, at 1–10 credits depending on the work involved. So a 1,000-credit pack — or the free signup grant — translates to:

Tool	Credits per call	Calls per 1,000 credits
`fetch_url` (raw HTTP fetch)	1	1,000
`scrape` (markdown/HTML/text)	2	500
`extract_structured` (schema JSON)	3	333
`search_web` / `stealth_mode`	5	200
`deep_research` (multi-source)	10	100

You mix and match against one balance, over MCP (Claude Code, Claude Desktop, Cursor), REST, or the browser playground. And because none of it expires, "1,000 credits" means 1,000 page fetches whenever you get around to them — not 1,000 fetches this month or nothing.

How Firecrawl Handles the Same Question

Firecrawl — the best-known name in this space, YC-backed with an established community — is a solid product, and its pricing is the standard subscription shape. The free tier grants 1,000 credits per month, and unused credits do not roll over — Firecrawl's own pricing FAQ states that credits do not roll over on any self-serve plan, with rollover reserved for its Scale and Enterprise tiers. Paid plans (billed annually) run Hobby $16/mo for 5,000 credits, Standard $83/mo for 100,000, and Growth $333/mo for 500,000; there is no equivalent of CrawlForge's standalone one-time packs, and the same FAQ says outright that it does not offer a pay-per-use plan.

One caveat before you compare raw numbers: a Firecrawl credit buys one page, while CrawlForge bills 1–10 credits per tool call depending on the tool, so credit counts are not directly comparable between the two. What is directly comparable is the free-tier expiry policy: Firecrawl's free 1,000 credits reset monthly and unused ones lapse; CrawlForge's free 1,000 is a one-time grant that never expires. The feature-by-feature breakdown is in the CrawlForge vs Firecrawl comparison, with the wider field covered in CrawlForge vs Firecrawl, Tavily, and Exa.

When a Subscription Is the Better Deal

Honesty cuts both ways: at steady high volume, a subscription beats packs — including ours. CrawlForge Professional is $99/mo for 50,000 credits, about $1.98 per 1,000 — cheaper than even the largest one-time pack at $2.50. Business is $399/mo for 250,000, about $1.60 per 1,000, and plans also raise the rate limit: 2, 4, and 10 requests per second on Hobby, Professional, and Business versus 1 on free.

The rule of thumb: predictable volume in the tens of thousands of credits per month, subscribe; bursty or exploratory work, buy packs. And because CrawlForge plan credits roll over, subscribing does not reintroduce the use-it-or-lose-it problem the packs exist to solve — a slow month just banks credits for the next burst.

FAQ

Do CrawlForge credits expire?
No. One-time credit packs ($3 for 1,000 up to $62.50 for 25,000) never expire, and on paid plans all unused monthly credits roll over and never expire while the subscription is active. The free 1,000 signup credits are also permanent — they never refill, but they never disappear either. There is no use-it-or-lose-it mechanic anywhere in the pricing.

What happens to my credits if I cancel my subscription?
Nothing — cancelling a CrawlForge subscription never wipes unused credits, so whatever balance you have built up stays spendable, including plan credits rolled over from previous months. The one exception is choosing a full refund of an untouched billing period: the money goes back, and the credits that payment provided go with it. If you need more later, you can buy one-time credit packs without resubscribing.

Can you use a web scraping API without a subscription?
Yes. CrawlForge sells one-time credit packs — $3 for 1,000 up to $62.50 for 25,000 — in the dashboard with no subscription attached, and they never expire. Every account also starts with 1,000 free one-time credits, no credit card required. All 28 tools work on either footing, over REST, MCP, or the browser playground.

Do the free 1,000 credits expire or refill?
Neither. CrawlForge's free tier is a one-time grant of 1,000 credits at signup: it never refills monthly, and it never expires, with no credit card required and a rate limit of 1 request per second. Firecrawl's free tier works the other way around — 1,000 fresh credits each month, but unused ones do not roll over (noting a Firecrawl credit buys one page while CrawlForge tools cost 1–10 credits per call).

Start Without a Clock

The cheapest way to test any of this is the grant every account starts with: 1,000 one-time credits, no card, no expiry — enough for 1,000 raw fetches, 500 scrapes, or 100 deep-research runs. If they are still sitting there in six months, that is the feature working as designed.

Start Free — 1,000 Credits Included

Reddit 403'd our stealth browser, so we stopped scraping reddit.com entirely

Simon — Mon, 24 Aug 2026 23:29:33 +0000

Ask any scraper developer what the hardest mainstream site is in 2026 and Reddit is a safe answer. We would know — reddit.com blocks every direct access path CrawlForge has, from a plain fetch to a full stealth browser.

So CrawlForge MCP v5.1.0 stops fighting the wall and goes around it: the new reddit_search tool — our 28th — searches Reddit posts and comments and reads entire threads through community-run archives, with no Reddit API key, no OAuth app, and no credentials of any kind.

TL;DR: v5.1.0 adds reddit_search: search Reddit posts and comments, or read a full nested comment thread, without ever touching reddit.com. It queries two community archives — Arctic Shift (near-real-time) and PullPush (Pushshift-compatible full-text search) — so there is nothing to sign up for on Reddit's side. Three modes, 2 credits per call. Upgrade: npm install -g crawlforge-mcp-server@latest.

What shipped
Why reddit.com cannot be scraped
One tool, three modes
The archives: Arctic Shift and PullPush
How routing works
Honest caveats
Using it from MCP and REST
What about the Reddit thread template
Credit cost
How to upgrade

What shipped

v5.1.0 is a focused minor release with one headline:

reddit_search — search Reddit posts (title + selftext) and comments, or read a post plus its nested comment tree, via the Arctic Shift and PullPush community archives. Three modes, scoped filters, normalized output. 2 credits per call.
Tool count goes from 27 to 28, and the MCP server card, tool instructions, and getting-started prompt all know about the newcomer.
31 new unit tests (1,018 total in the suite) and 100% MCP protocol compliance across all 28 tools, live-verified over MCP stdio returning real r/ClaudeAI posts.

Why reddit.com cannot be scraped

We did not reach for archives out of preference — we measured the wall first. Before building the tool we ran every direct path against reddit.com, live:

fetch_url with a full browser User-Agent: 403
scrape against old.reddit.com: 403
scrape_template with the reddit-thread template: 403
stealth_mode at its most advanced level: 403

The block is layered — datacenter IP reputation first, TLS fingerprinting at the handshake, then a JavaScript challenge — which is why even a stealth browser on a server fails.

And the sanctioned door narrowed too: Reddit closed self-service API signup in November 2025 under its Responsible Builder Policy, so new applications for official API credentials go through an approval request rather than a signup form. Most Reddit MCP servers wrap that official API, which now makes "get credentials" the hardest step of their setup.

reddit_search sidesteps all of it. It never sends a single request to reddit.com.

One tool, three modes

posts (default) — keyword search across post titles and selftext, optionally scoped to a subreddit or author. Supports "quoted phrases", OR, and -exclusion.
comments — full-text search across comment bodies, something the official Reddit API cannot do at all.
thread — hand it a post ID and get the post plus its nested comment tree, with reddit-style collapse markers for branches too deep to expand.

Filters cover subreddit, author, after/before dates (ISO 8601, epoch seconds, or offsets like "7d"), limit (up to 100), and sort. Results come back normalized: full reddit.com permalinks, ISO dates, scores, comment counts, and text capped at 2,000 characters with truncation flags so a 100-result payload stays friendly to an LLM context window.

A typical call from an MCP client looks like this:

{
  "tool": "reddit_search",
  "arguments": {
    "query": "best mcp servers",
    "subreddit": "ClaudeAI",
    "mode": "posts",
    "limit": 10
  }
}

Take any post id from the results, pass it back as link_id with mode: "thread", and you have the whole discussion — ready for summarize_content or analyze_content to turn into sentiment, topics, and a briefing.

The archives: Arctic Shift and PullPush

If you remember Pushshift — the beloved Reddit research archive that lost its API access in 2023 — these two projects are its successors, and they are what reddit_search queries:

Arctic Shift ingests Reddit in near-real-time. During live testing it returned a post created the same day, and it serves proper nested comment trees. Its one documented constraint: keyword search must be scoped to a subreddit or author.
PullPush is Pushshift-compatible and does what Arctic Shift will not — full-text search across all of Reddit. The trade-offs: documented gaps in its post-2023 archive and aggressive rate limits.

Both are free, both are community-run, and neither needs credentials. Every response includes provenance notes naming the archive that answered, so downstream consumers always know where the data came from.

How routing works

You never pick a backend unless you want to. In the default auto mode:

scoped search (subreddit/author)  --+--> Arctic Shift (fresher)
thread read                       --+      |
                                           +--> error? --> PullPush
                                                           (fallback_used: true)

unscoped full-text search  ----------> PullPush
                                       (only archive that supports it)

The plumbing absorbs the archives' quirks: every request carries an identifying User-Agent (live testing showed Arctic Shift throttles anonymous clients into a shared bucket), transient throttle responses get one bounded retry, and PullPush's rate-limit messages pass through verbatim so you see the real reason instead of a generic error. A REDDIT_SEARCH_TIMEOUT_MS environment variable overrides the 30-second per-request cap if your pipeline needs a different budget.

Honest caveats

Archive data has edges, and we would rather document them than let you find them:

Very fresh content reads low. The archives capture posts the moment they appear, so scores and comment counts of content less than ~36 hours old often read 0 or 1. The content is there; the vote tallies have not caught up.
PullPush has post-2023 gaps. An empty unscoped search does not prove the content does not exist. Scope to a subreddit or author when you can — it routes you to the more complete archive.
Deleted content may persist in archives. That is inherent to how archives work, and it cuts both ways: it is also why researchers use them.

Using it from MCP and REST

In an MCP client — Claude Desktop, Claude Code, Cursor — just ask: "Search r/webdev for discussions about scraping infrastructure from the last month and summarize the top complaints." The agent picks reddit_search, scopes it, and chains the summary itself.

From the REST API, it is one authenticated POST:

const response = await fetch('https://crawlforge.dev/api/v1/tools/reddit_search', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    query: 'best mechanical keyboard',
    subreddit: 'MechanicalKeyboards',
    mode: 'posts',
    limit: 10,
  }),
});

const { data } = await response.json();
console.log(`${data.count} posts via ${data.source}`);

You can also try it with zero setup in the playground — reddit_search is live there alongside the other 27 tools.

What about the Reddit thread template

scrape_template still ships its reddit-thread template, but it points at reddit.com — the very wall this release routes around — so it no longer works reliably. The scrape_template docs now say exactly that and point Reddit work at reddit_search instead. The other nine templates (Amazon, GitHub, YouTube, Hacker News, and friends) are unaffected.

Also since v5.0.4
If you skipped the patch between releases, v5.0.5 fixed two numbers serp_rank had been asserting wrongly: the DataForSEO request timeout (now 120 seconds by default, tunable via DATAFORSEO_TIMEOUT_MS, because live Google scrapes at depth 100 routinely take 30-60+ seconds) and the documented upstream cost (the old figure was the depth-10 price, 10× lower than the depth-100 default actually bills). It also fixed the Smithery listing, which now derives its tool table from the live registry instead of a hand-written card. The full story of the v5.0.x hardening cycle is in the v5.0.4 live-testing post.

Credit cost

reddit_search costs 2 credits per call — search or full thread read, same price. For scale: the free plan's 1,000 starter credits cover 500 Reddit searches, and a search plus a thread read plus summarize_content on the result is 8 credits end to end. As always, failed requests are not charged.

How to upgrade

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 5.1.0

Or, if your MCP client launches the server with npx, it picks up 5.1.0 automatically on the next restart. No schema, output-shape, or credit-cost changes to any existing tool — this is a drop-in upgrade. The package is crawlforge-mcp-server on npm.

Ready to mine Reddit discussions without an API key? The free plan's 1,000 credits are enough for 500 searches, and the reddit_search API reference covers every parameter and mode.

Start free with 1,000 credits

Answer 3 Questions, Get 1,000 Free CrawlForge Credits

Simon — Mon, 24 Aug 2026 23:23:40 +0000

We just started paying for feedback. Complete the CrawlForge product survey — three questions, about twenty seconds — and we add 1,000 free credits to your account, instantly and automatically. That is the same amount a brand-new account gets on signup, so a free-tier member who claims it doubles their lifetime credits.

TL;DR: Sign in at crawlforge.dev, click the floating feedback button in the bottom-right corner, answer 3 questions. If your account has used 250+ credits on real API calls, 1,000 credits are granted on the spot. One reward per account, ever.

There is one honest condition, and we would rather state it up front than bury it in fine print: the reward only goes to accounts that have actually used the product — at least 250 credits of API usage. If your account has not reached that mark yet, the survey still works and we still read every answer — but no credits are granted.

The reasoning is simple. We are paying for feedback because feedback from real usage is worth paying for. An opinion about a product you have barely run tells us about our marketing, not our product, and a reward with no real-usage requirement would just be a faucet for throwaway accounts.

How to claim your 1,000 free credits

Sign in to your CrawlForge account. The grant is keyed to your session, so an anonymous submission cannot be rewarded — there is no account to credit. No account yet? Signing up is free and comes with its own 1,000 starter credits.
Make sure your account has used at least 250 credits. Every credit-consuming call made with your API key counts toward the total, whether it went through MCP, the REST API, or the CLI — 125 scrape calls (2 credits each) or 25 deep_research runs (10 credits each) gets you there. Just getting started? The quickstart guide gets you from zero to a first scrape in under five minutes.
Open the survey. It lives behind the floating message button in the bottom-right corner of every page on crawlforge.dev and the dashboard.
Answer the three questions and hit send. A rating, a one-tap question about what is blocking you, and an optional comment. If you qualify, the confirmation message tells you the credits were added on the spot — no code, no email, no waiting.

The whole thing takes less time than reading this section did.

Who qualifies

The rules, stated plainly:

You must be signed in when you submit the survey.
Your account must have used at least 250 credits on API calls before submitting. Grants and bonuses do not count toward the total — only actual tool calls do.
One reward per account, ever. Submitting the survey again is welcome (feedback changes as the product does), but it will not grant twice.
Already answered the survey before this announcement? Submit it again while signed in — earlier anonymous responses could not be matched to your account, so your claim is still available.

If you submit and do not see the credits message, it is almost always one of the first two rules: you were not signed in, or the account has not used 250 credits yet. Fix whichever applies and submit again.

What 1,000 credits actually buys

Every CrawlForge tool has a fixed credit price, so the reward converts directly into work. A few examples from the pricing page:

Tool	Credits per call	1,000 credits =
`scrape`	2	500 pages scraped
`extract_content`	2	500 clean article extractions
`search_web`	5	200 web searches
`serp_rank`	5	200 Google rank checks
`deep_research`	10	100 full research runs

Mix and match across all 28 tools — the full price list is in the API reference.

The survey stays anonymous if you want it to

Nothing about the reward changes how the survey handles your data. It is self-hosted — no third-party survey processor ever sees a response. Your IP address is only ever stored as a salted hash, the email field is optional, and you can submit without an account at all. Signing in is only required for the credits, because a reward needs an account to land in. Signed-out feedback is read with exactly the same attention.

The fine print

The reward is a one-time promotional grant: it does not renew, has no cash value, cannot be transferred, and — like all one-time credits — is not refundable.
Submissions are rate-limited, and automated or fraudulent submissions void the reward.
We may change or end this promotion at any time; credits already granted stay granted.

Why we are doing this

CrawlForge is built for people running real workloads — AI agents that need reliable web data, scrapers that hit anti-bot walls, research pipelines that fan out across hundreds of pages. Analytics tell us what those people do; they do not tell us what almost made them leave, or which missing tool they are quietly working around. Three questions from someone who has actually run the tools answer that better than any dashboard, and 1,000 credits is a fair trade for it.

The button is in the bottom-right corner. Sign in, tell us what you think, and put the credits to work.

Not a member yet? Start free with 1,000 credits — run a few tools, then come back and take the survey for 1,000 more.

Start free with 1,000 credits

CrawlForge v5.0.0: Security, Correctness, MCP Spec

Simon — Fri, 14 Aug 2026 13:58:26 +0000

http://2130706433/ is a valid URL. Your browser will happily resolve it to 127.0.0.1, because the WHATWG URL parser normalizes decimal, hex (0x7f000001), and octal integer forms into dotted-quad IPv4.

Our SSRF guard did not know that. It resolved hostnames through DNS and range-checked the resulting addresses — but Node never routes an IP literal through lookup, so a URL whose host was already an IP sailed straight past the check. Loopback, link-local, cloud metadata: all reachable, in a server whose entire job is fetching URLs a model picked for you.

That is one bug out of the seven-phase internal audit that became CrawlForge MCP Server v5.0.0. The unit suite went from 480 tests to 914. npm audit went from 16 vulnerabilities to 0. Almost none of it is new features.

What actually shipped
The one breaking change: Node 20
Phase 1: the SSRF bypass we shipped
Phase 2: 52 ways tools were silently wrong
Phase 3: safe to run for days
Phase 4: HTTP mode only ever had one session
Phase 5: zero npm audit vulnerabilities
Phase 6: MCP spec adoption
Pricing and how to upgrade

What actually shipped

Phase	Theme	Headline result
0	Dependency currency	`npm audit` 16 vulns → 4 moderate, zero code change
1	Critical security	SSRF IP-literal bypass, OAuth token minting, secret leakage, billing
2	Correctness	52 fixes — including a `crawl_deep` rewrite
3	Leaks and timeouts	24 fixes — browser contexts, unbounded caches, real deadlines
4	HTTP transport	19 fixes — multi-session streamable HTTP, working prompts, webhook HMAC
5	Dependency modernization	Node ≥ 20 floor, 0 npm audit vulnerabilities
6	MCP spec adoption	Structured output, async tasks, tool whitelisting, registry `server.json`

MCP protocol compliance held at 100.0% COMPLIANT, 0 errors at every phase gate.

The one breaking change: Node 20

engines.node moved from >=18.0.0 to >=20.16.0.

Node 18 hit end-of-life in April 2025, and 20.16 is the floor required by pdf-parse 2.4.5 — the maintained ESM rewrite we needed to clear the last audit findings. Our Dockerfile (node:20-alpine) and CI (Node 22) already satisfied it.

That is the entire breaking surface. No tool schema, output shape, or credit cost changed, and the tool count stays at 27.

node --version   # must be >= 20.16.0

Phase 1: the SSRF bypass we shipped

Read this phase if you run any MCP scraping server near a private network.

BEFORE: url -> parse -> DNS lookup -> ipBlocked(resolved)?  -> fetch
                          |
                          +--> IP literal? no lookup happens.
                               guard never runs. request goes out.

AFTER:  url -> parse -> ipBlocked(literal host)? --------+
                     -> DNS lookup -> ipBlocked(addrs)? -+-> fetch
                     -> per-connect check in the undici dispatcher
                        (catches every redirect hop too)

v5.0.0 runs ipBlocked() on IP-literal hostnames at pre-flight and wraps the undici dispatcher's buildConnector with a per-connect check, so a redirect hop straight to an internal address is blocked as well.

Three more guard fixes landed with it:

IPv4-mapped IPv6. ::ffff:127.0.0.1 and ::ffff:169.254.169.254 are normalized to their embedded IPv4 before range checks, in both default and strict modes. Kills the DNS-controlled AAAA-record bypass.
BLOCKED_DOMAINS was dead config. It was declared and read by nothing. It is now enforced at pre-flight.
The allowlist is evaluated per hop. An allowlisted first hop used to unguard every redirect after it.

We also wired the guard into five paths that never had it: scrape_with_actions (with a post-navigation page.url() re-check that closes the page on a redirect into a blocked range — that was a Playwright internal-network read primitive), map_site, process_document PDF downloads, webhook delivery and health checks, and deep_research webhook notifications.

Beyond SSRF: OAuth, secret leakage, and billing
OAuth. /oauth/authorize now requires proof of the operator's API key before issuing a code, with constant-time digest comparison. The anonymous register → authorize → token flow that minted operator-billed bearer tokens is closed.

Secret leakage. Usage telemetry passes tool params through maskSecrets() before the payload leaves the process — third-party API keys, auth headers, and webhook signing secrets no longer travel in plaintext. deep_research stopped writing LLM API keys to Winston file logs.

Billing. A throw from the credit check itself now bills zero; the error-path half-charge only applies once the handler has actually started. checkCredits distinguishes 401/403 (invalid or revoked key) from 5xx (grace window) instead of reporting both as "insufficient credits."

If you want the general version of this problem rather than our specific one, we wrote it up separately: SSRF in MCP servers.

Phase 2: 52 ways tools were silently wrong

This is the "passes smoke tests, returns misleading output" class — the one that never shows up as an error in your logs.

crawl_deep is usable for real crawls again. BFS child pages were awaited from inside an occupied queue slot, so the per-task queue timeout bounded the entire recursive crawl rather than one page. Any crawl outliving the 30-second timeout threw away every page it had already fetched with a bare Promise timed out, and low concurrency settings (including concurrency: 1) deadlocked outright. Both fixed.

A representative sample of the other 51

Cache keys that contradicted the request. crawl_deep's result-cache key now covers extract_content, content length, include/exclude patterns, follow_external, respect_robots, concurrency, domain filter, and session. map_site's covers search, domain filter, include_metadata, and group_by_path. Previously a cached call could contradict your parameters for a full hour-long TTL.
Character encoding. Bodies decode with their declared charset (Content-Type header or <meta charset> sniff) instead of always UTF-8. No more U+FFFD soup from ISO-8859-1 or Shift_JIS sites.
Silently stripped options. The options schemas for extract_content, summarize_content, and analyze_content now use .passthrough(). Every documented option key was being stripped before it reached the handler — which is also why summarize_content always returned the same 2-sentence fallback mislabeled extractive. The extractive summarizer now actually runs, and summaryLength changes the output.
Link resolution. extract_links resolves relative hrefs against the final page URL rather than the origin, honors <base href>, and classifies protocol-relative links as external. The same fixes landed in scrape's extractor, so the two finally agree.
track_changes similarity. Now token-Jaccard over the content. It was previously Hamming distance between sha256 hex digests — so every trivial edit scored roughly 0% similar and fired a "moderate change" alert.
search_web scoring. Partial ranking_weights deep-merge over the defaults instead of replacing them wholesale, so no more NaN final scores or silently disabled duplicate checks. The zero-result expansion retry is capped at one fallback instead of up to five billed backend searches.

Phase 3: safe to run for days

24 findings in the class that only surfaces in long-running processes.

Browser lifecycle. Closing a Playwright page does not close its context — so every scrape_with_actions call and every browser-rendered extract_content leaked one context until shutdown. Contexts are now closed alongside their page, and a failed page.goto (DNS error, timeout, blocked URL) tears down both instead of orphaning them.

Bounded caches. crawl_deep destroys its per-crawl CacheManager in a finally. Previously N crawls permanently leaked N caches of up to 1,000 full HTML documents each — every one of them re-running a JSON.stringify memory scan every 60 seconds, forever. Dropped instances are now GC-verified with a WeakRef regression test.

Deadlines on every body read. The abort timer stays armed through the body stream, so timeout finally covers a server that returns headers and then stalls. Chunk reassembly is single-pass — it was O(n²), roughly 1.5 seconds of synchronous event-loop block on a 25 MB body. PDF downloads got a real AbortSignal.timeout (the old timeout: fetch-init option is silently ignored by undici).

One for Claude Desktop users: snapshot storage defaults to ~/.crawlforge/snapshots instead of process.cwd(). MCP clients launch the server with a working directory of /, where every snapshot write silently failed.

Phase 4: HTTP mode only ever had one session

If you deployed over npm run start:http, it was worse than you thought. A single shared transport meant exactly one session ever existed, and any clean disconnect bricked /mcp until you restarted the process.

Stateful mode now follows the SDK's documented per-session pattern — a Map<sessionId, {transport, server}> with a fresh transport and cloned McpServer per initialize, disposal on DELETE, and a JSON-RPC 404 for unknown session IDs. Second concurrent client, reconnect after a network drop, DELETE then fresh initialize: all work now.

Also in Phase 4:

The getting-started prompt was unretrievable by any client — the config object hit the SDK's positional argsSchema overload, advertising a bogus required argument and failing every prompts/get. The compliance suite now covers discovery and retrieval for all 6 prompts.
Webhook HMAC signatures cover the exact serialized body that is POSTed. Only the data sub-object was being signed, so standard receiver-side raw-body verification failed every single time.
scrape no longer inlines multi-megabyte base64 screenshots into the JSON result — it keeps metadata plus a crawlforge://screenshot/{id} resource URI.
Auto-setup banners moved from stdout to stderr, so a first launch no longer injects non-JSON lines into the stdio JSON-RPC channel.

Phase 5: zero npm audit vulnerabilities

With the Node 20 floor in place, we retired every abandoned dependency and took the security upgrades the old floor had blocked. 4 moderate → 0.

Removed outright: node-cron (unused after Phase 3 moved scheduling to setInterval; removal cleared its vulnerable uuid chain), @googleapis/customsearch (unused — the Google adapter calls the REST endpoint directly), and node-summarizer (abandoned since 2019; the extractive summarizer was rewritten as a compromise-based Luhn-style word-frequency scorer with identical result shapes).

The upgrade that mattered most: pdf-parse 1.1.1 → 2.4.5. PDFProcessor was ported to the v2 class API, so the password option now actually decrypts protected PDFs — v1 silently ignored it.

On supply chain: this phase ran during the ChainDrop npm worm. Every install ran with --ignore-scripts, every adopted version was publish-date-gated, and the full lockfile diff was cross-checked against public compromised-package lists with zero matches.

Phase 6: MCP spec adoption

Structured output (MCP 2025-06-18). scrape, map_site, serp_rank, search_web, extract_structured, and crawl_deep declare an outputSchema and return structuredContent alongside the legacy JSON text. The schemas are permissive by design, so a legitimate result can never fail SDK output validation.

Async tasks. crawl_deep, batch_scrape, deep_research, and agent are registered with taskSupport: 'optional' under the io.modelcontextprotocol/tasks extension. Task-aware clients get a handle immediately and poll tasks/get; clients without task support still get the synchronous result exactly as before. This is the fix for long crawls timing out inside a client's tool-call window.

Client-side tool selection. Two env vars let you expose a subset of the 27 tools and cut context bloat:

# By name
CRAWLFORGE_TOOLS=scrape,search_web,extract_content

# Or by group — 12 available: basic, search, crawl, extract, batch,
# research, tracking, llmstxt, stealth, templates, scrape, agent
CRAWLFORGE_TOOL_GROUPS=search,extract

Unset means all tools. Unknown names are ignored with a stderr warning, and batch_scrape auto-enables get_batch_results.

Protocol hygiene. Schemas advertised in JSON Schema 2020-12 instead of draft-07. tools/list sorted deterministically for client prompt-cache stability. Invalid tool arguments come back as isError: true tool results — which a calling model can self-correct from — rather than -32602 protocol errors. And server.json is complete against the 2025-12-11 registry schema.

Missed v4.9.0 and v4.10.0? Two things landed in between
v4.9.0 added serp_rank, the 27th tool — real Google organic rank positions via DataForSEO, 5 credits per configured lookup. v4.10.0 made it return the full top-10 organic listing alongside your target domain's positions. Docs.

v4.10.0 also added server-level MCP instructions: the server tells any connecting client to prefer CrawlForge tools over its own built-in web capabilities. It ships in the server binary, so every client picks it up on the next launch after upgrade — no re-init. It is guidance, not enforcement; an MCP server cannot disable a client's built-in tools.

Pricing and how to upgrade

Nothing changed. All 27 tools are metered and require an API key, at 1-10 credits per call.

Plan	Price	Credits
Free	$0 (no card)	1,000 one-time trial (does not reset)
Hobby	$19/mo	5,000
Professional	$99/mo	50,000
Business	$399/mo	250,000

Every plan gets every tool. LLM extraction defaults to local Ollama, so you do not need an OpenAI or Anthropic key unless you opt in.

# existing users
npm install -g crawlforge-mcp-server@latest   # or an /mcp reconnect

# new users
npm install -g crawlforge-mcp-server && npx crawlforge init

Because Phase 2 fixed tool behavior rather than tool contracts, your existing calls keep working — they just return correct results now.

Deferred rather than rushed: a hosted remote endpoint with OAuth, a keyless tier, scheduled monitoring as a service, persistent sessions, and PII redaction.

Writing up your own bugs is uncomfortable, but "advertised control, silently non-functional" is the single most common failure mode we found across all seven phases — and it is invisible from the outside. If you find a CrawlForge control that does not behave the way the docs claim, that is exactly the bug we want.

npm: crawlforge-mcp-server · full v5.0.0 writeup · v4.8.0 release post

Start free with 1,000 credits

My Claude Code skills never auto-activated either — here's what we fixed

Simon — Mon, 29 Jun 2026 12:50:19 +0000

If you have ever dropped a markdown "skill" file into ~/.claude/skills/ and watched Claude completely ignore it, you are not imagining things. The file sits there, but nothing tells the model when it is relevant — so it never fires.

We had exactly that problem. CrawlForge shipped reference-markdown files that were supposed to teach Claude how to use our 26 web-scraping tools. They sat on disk. Claude never actually loaded them. You still had to spell out "use the stealth_mode tool to scrape this" — which defeats the point.

CrawlForge MCP Server v4.8.0 (shipped June 28, 2026) fixes that. It is a fully additive minor release: the tool count stays 26, and no tool schema, output shape, or credit cost changes for existing callers. Here is what changed.

Before / after
7 Claude Agent Skills that actually auto-activate
Two new scrape formats: branding and screenshot
Built-in scheduled change monitoring
Security: controls that were advertised but silently broken
Pricing and how to upgrade

Before / after

BEFORE: "Use the crawlforge stealth_mode tool to scrape this protected page."
        +--> you name the tool, every time

AFTER:  "Scrape this Cloudflare-protected page for me."
        +--> Claude matches the prompt, loads the stealth-browsing skill,
             and calls the right tool on its own

One behavior change to flag up front, Stripe-changelog style: clients that support MCP elicitation will now see cost and safety confirmations that previously failed to appear (more on that below). Everything else is purely additive.

Upgrade in one line, then run init:

npm install -g crawlforge-mcp-server@latest
npx crawlforge init

Added: 7 auto-activating Agent Skills, branding + screenshot scrape formats, scheduled change monitoring, list_scheduled_monitors.
Fixed: SSRF now enforced on the live path, MCP elicitation now fires, screenshots actually render.
Security: per-host rate limiting, executeJavaScript hardening, DNS-rebinding protection.

7 Claude Agent Skills that actually auto-activate

A skill is a folder containing a SKILL.md file with YAML frontmatter. At startup Claude pre-loads only the name and description of every installed skill, then reads the full body only when it judges the skill relevant to your prompt. Anthropic calls this progressive disclosure — skills are not always-loaded context, they are loaded on demand.

The catch most people miss: auto-activation is model-judged, not guaranteed. The quality of that description field is what decides whether the right skill fires. Here is the frontmatter shape we now ship:

---
name: crawlforge-stealth-browsing
description: ">-"
  Scrape sites protected by Cloudflare, anti-bot, or JavaScript
  rendering. Use when a normal fetch is blocked, returns a challenge
  page, or the content is rendered client-side. Triggers on:
  "bypass bot detection", "Cloudflare", "headless", "stealth".
---

Trigger-rich descriptions dramatically raise the probability the right skill fires. v4.8.0 ships seven skills covering all 26 tools:

crawlforge-web-scraping
crawlforge-deep-research
crawlforge-stealth-browsing
crawlforge-structured-extraction
crawlforge-change-tracking
crawlforge-batch-automation
crawlforge-getting-started

They install to the personal scope at ~/.claude/skills/<name>/SKILL.md, and because activation is a heuristic, you can always name the skill or tool explicitly in your prompt to force it.

How skills relate to MCP (they complement, not replace)
MCP and skills solve different halves of the same problem. MCP exposes the 26 tools to Claude. Skills teach Claude when and how to use them — think of MCP as giving a new hire access to the tools, and skills as the onboarding guide that explains which tool to reach for.

Upgrades self-heal: the installer removes the legacy bare reference files and leaves unrelated skills untouched. npm run skills:gen regenerates the root SKILL.md.

Want to push activation even harder? There is an opt-in forced-eval hook — an idempotent UserPromptSubmit reminder:

npx crawlforge init --with-hook          # or: install-skills --with-hook
npx crawlforge uninstall-skills --remove-hook

It is OFF by default.

npx crawlforge init does the whole setup: configures your API key, installs the skills, and registers the MCP server with your AI clients.

Two new scrape formats: branding and screenshot

The scrape tool gained two formats. Both cost the same 2 credits as before.

branding extracts design tokens statically from HTML + CSS with no browser required — color palette, fonts and typography, logo and favicons, plus border-radius, shadow, and spacing tokens. It is SSRF-guarded, and linked-CSS fetches are count- and size-capped.

{
  "url": "https://stripe.com",
  "formats": ["branding"]
}

screenshot now actually works (it was a no-op before). It lazily renders through the shared browser pool and returns crawlforge://screenshot/{id} MCP resources. The browser launches only when you request a screenshot, and a render failure degrades to a warning so partial success is preserved.

Built-in scheduled change monitoring

track_changes (3 credits) gained real scheduling operations — create_scheduled_monitor, stop_scheduled_monitor, and a new list_scheduled_monitors. These were previously dead code that threw. They are now backed by a persisted scheduler (MonitorScheduler.js + MonitorStore.js), and baselines rehydrate from snapshots on restart.

You can attach an optional plain-English goal that is LLM-judged (Ollama-first) and degrades gracefully to threshold significance when no LLM is available — e.g. "alert me only when the pricing changes, not when the copyright year flips."

The honest firing model (read this before you rely on it)
A stdio MCP server is not a long-lived daemon — it only runs while your client has it open. So reliable scheduled firing uses the CLI plus system cron:

crawlforge monitor:create --url https://api.example.com/changelog
crawlforge monitor:list
crawlforge monitor:stop <id>

# system cron runs this; it guarantees due monitors fire:
*/30 * * * * crawlforge monitor:run-due

monitor:run-due is a one-shot designed for cron. We would rather tell you this than pretend a stdio process magically wakes itself up.

See the track_changes docs for the full schema.

Security: controls that were advertised but silently broken

MCP servers are now a recognized SSRF attack surface. A web-scraping server that fetches arbitrary URLs on your behalf is exactly the shape of thing an attacker points at the cloud-metadata endpoint (169.254.169.254) to read credentials. We audited our own posture and found two controls that were advertised but silently non-functional. Both are fixed.

SSRF is now enforced on the live path. A robust ssrfProtection.js existed in the codebase but was never wired into the tools — every scrape used raw fetch() with zero IP/host validation. A new ssrfGuard.js injects an undici dispatcher whose connect-time lookup validates every connection — the initial request and every redirect hop — and pins to the validated IP, closing the DNS-rebinding TOCTOU window.

# Stage 1 (default): blocks loopback, link-local / cloud-metadata
#                     (169.254.169.254), and 0.0.0.0
SSRF_STRICT=true              # adds full RFC1918 / ULA private-range enforcement
ALLOWED_DOMAINS=internal.corp # trusted-host bypass
SSRF_PROTECTION_ENABLED=false # kill switch

It is routed through roughly 14 modules — the basic fetch path, batch scrape, map_site, crawl, extract, process_document, research, llms-txt, robots/sitemap, and the track_changes differ.

MCP elicitation now actually fires. The old code called server.elicit() — a method that does not exist — and never checked the client capability, so every cost/safety confirmation (for example, deep_research over 50 URLs, batch_scrape, and crawl_deep) silently fail-opened. It now uses elicitInput, gates on the client's elicitation capability, and parses the accept / decline / cancel action. It still fail-opens for clients that do not support elicitation — but capable clients will finally see the prompts.

Two more layers of defense-in-depth:

Per-host rate limiting (hostRateLimiter.js): default 10 req/s per host via RATE_LIMIT_PER_DOMAIN, with no global cap, so broad multi-host crawls are unaffected. This is politeness/defense, not an SSRF boundary itself.
executeJavaScript hardening (still OFF by default): JS_MAX_SCRIPT_LENGTH, JS_EXECUTION_TIMEOUT_MS, and a structured stderr audit log (script sha256 + length + url).

Pricing and how to upgrade

No new tools — the new capabilities were added to existing ones, so the count stays 26. All 26 are metered and require an API key; costs range 1-10 credits per call. Heads up: list_ollama_models is now 1 credit (it used to be free), so nothing is free per-call.

Plan	Price	Credits
Free	$0 (no card)	1,000 one-time trial (does not reset)
Hobby	$19/mo	5,000
Professional	$99/mo	50,000
Business	$399/mo	250,000

Every plan gets every tool. LLM extraction defaults to local Ollama, so you do not need an OpenAI or Anthropic key unless you opt in.

# new users
npm install -g crawlforge-mcp-server && npx crawlforge init

# existing users
npm install -g crawlforge-mcp-server@latest   # or an /mcp reconnect

Re-running init (or install-skills) gets you the 7 skills and self-heals the legacy bare files. v4.8.0 is additive, so nothing breaks. This continues the 4.7.x correctness work: 4.7.2 live-audited all 26 tools (scrape_with_actions, extract_structured, resources/read), 4.7.1 fixed deep_research credibilityThreshold and generate_llms_txt emitting literal "undefined", and 4.7.0 moved to a fully metered model.

That is the whole release: skills that fire, controls that enforce, screenshots that render. Try it, break it, and tell us what does not work — that feedback is what drives this cadence.

npm: crawlforge-mcp-server · Best MCP servers for web scraping in 2026

Start free with 1,000 credits

How to Give ChatGPT Web Scraping with MCP Connectors (2026)

Simon — Tue, 16 Jun 2026 20:20:51 +0000

ChatGPT can now call your own tools through custom MCP connectors — including web scraping. But there is a catch the marketing pages skip: connectors must be remote servers, so a local tool like CrawlForge cannot be pasted in directly. This is the honest version: what is actually possible, why a wrapper is needed, and the exact bridge to build.

TL;DR: ChatGPT custom MCP connectors (renamed "apps" in Dec 2025) work on Plus, Pro, Business, Enterprise, and Edu via Developer mode — not Free/Go. Connectors must be remote HTTPS servers, so a local stdio server like CrawlForge can't be added directly. The fix: a ~30-line remote MCP wrapper that proxies CrawlForge's REST API.

What ChatGPT Connectors Are
Which Plans Can Use Them
The Transport Catch: Remote Only
Why CrawlForge Needs a Wrapper
Build the Bridge
Add the Connector in ChatGPT
Auth and Safety
A Simpler Alternative

What ChatGPT Connectors Are

ChatGPT supports custom MCP connectors — renamed "apps" in December 2025, so the UI now says "Apps & Connectors." Through Developer mode, you connect an external MCP server and ChatGPT calls its tools mid-conversation, asking you to confirm before any write action. Same Model Context Protocol that powers web scraping in Claude — different client. Developer mode is explicitly a beta.

Which Plans Can Use Them

Per OpenAI's plan table, adding a custom MCP connector is available on Plus, Pro, Business, Enterprise, and Edu — not Free or Go. Full write-action support is rolling out most broadly to Business, Enterprise, and Edu. If you only need ChatGPT to read scraped data, the read-only path below is enough.

The Transport Catch: Remote Only

This trips people up. A ChatGPT connector must be a remote MCP server reachable over HTTPS (SSE or Streamable HTTP transport). You paste a URL; you do not point it at a command on your machine. That rules out local stdio servers — the kind you install with npx. To use one, host it publicly or tunnel a local server via ngrok or Cloudflare Tunnel.

There is also a naming rule: ChatGPT's deep research / company-knowledge paths require two read-only tools named search and fetch with a specific schema. Full Developer mode allows arbitrary tools, so that constraint applies only to the deep-research path.

Why CrawlForge Needs a Wrapper

CrawlForge ships as a local stdio MCP server (via npx) plus a REST API at https://www.crawlforge.dev/api/v1/tools/. Neither is a remote MCP URL, and its tools are named search_web, fetch_url, and extract_content — not the search/fetch pair deep research expects. So you cannot paste CrawlForge straight into ChatGPT today. The practical path is a thin remote MCP wrapper — about 30 lines.

Build the Bridge

FastMCP (Python) is the quickest way to stand up a remote MCP server exposing the search + fetch tools ChatGPT wants. Each calls CrawlForge's REST API with your cf_live_ key in the X-API-Key header:

server.py — the full bridge

import os
import httpx
from fastmcp import FastMCP

mcp = FastMCP("CrawlForge Bridge")
BASE = "https://www.crawlforge.dev/api/v1/tools"
HEADERS = {"X-API-Key": os.environ["CRAWLFORGE_API_KEY"]}

@mcp.tool()
async def search(query: str) -> list[dict]:
    """Search the web. Returns id/title/url results for ChatGPT."""
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.post(f"{BASE}/search_web", headers=HEADERS,
                              json={"query": query, "limit": 10})
    results = r.json().get("results", [])
    return [{"id": x["link"], "title": x["title"], "url": x["link"]} for x in results]

@mcp.tool()
async def fetch(id: str) -> dict:
    """Fetch full page content by id (the URL) for ChatGPT."""
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.post(f"{BASE}/extract_content", headers=HEADERS,
                              json={"url": id})
    data = r.json()
    return {"id": id, "title": data.get("title", id),
            "text": data.get("content", ""), "url": id}

if __name__ == "__main__":
    mcp.run(transport="http", host="0.0.0.0", port=8000)

Run it and expose it over HTTPS. For a quick test, tunnel your local port:

pip install fastmcp httpx
export CRAWLFORGE_API_KEY="cf_live_your_key_here"
python server.py
# in another terminal:
ngrok http 8000

For Developer mode you can skip the search/fetch naming and map tools one-to-one to CrawlForge — expose scrape_structured, stealth_mode, or deep_research directly. Same pattern.

Add the Connector in ChatGPT

Settings → Apps & Connectors → Advanced → enable Developer mode.
Apps & Connectors → Create.
Paste your public HTTPS MCP URL (e.g. your ngrok URL plus /mcp), name it, choose an auth method.
Confirm the "I trust this application" checkbox.

Your search and fetch tools appear. In a chat, select the connector and ask ChatGPT to research a topic — it calls search, then fetches the best results through CrawlForge.

Auth and Safety

Connectors authenticate with none (public) or OAuth — there is no API-key-header option in the UI, which is why the wrapper holds your CrawlForge key server-side. ChatGPT confirms before write actions, and you can inspect each call before approving.

Take OpenAI's warnings seriously: only connect servers you trust. Custom connectors increase risk, including prompt injection, and a model mistake on a write action could destroy or leak data. A read-only scraping bridge is low-risk; lock it down with OAuth before sharing.

A Simpler Alternative

If you would rather not host anything, use CrawlForge from code with the OpenAI Agents SDK or Responses API — no remote server required. See CrawlForge with the OpenAI Agents SDK.

Get a Free CrawlForge Key — 1,000 Credits

How to Use Web Scraping Templates the Right Way (2026)

Simon — Sat, 06 Jun 2026 15:23:29 +0000

Most web scraping projects are not unique snowflakes. Track competitor prices. Enrich a list of leads. Audit a site for SEO. Pull training data for a model. It is the same handful of recipes, over and over.

A web scraping template is one of those recipes, pre-wired: a ready-to-use JSON config that chains the right tools in the right order, so you copy it, point it at your targets, and run. CrawlForge ships 24 of them in the templates gallery. This guide is about using them well — not just copy-paste, but read, adapt, and cost them out before you scale.

TL;DR: A CrawlForge template is a copy-paste JSON config that chains multiple MCP tools into one workflow (price monitoring, lead enrichment, SEO audits, market research, AI training data). There are 24 across 9 categories, each costing 3–19 credits per run. Run them from Claude/Cursor, the crawlforge CLI, or the REST API. Free tier = 1,000 credits, no credit card.

What Is a Web Scraping Template?
Templates Gallery vs the scrape_template Tool
How to Use a Template the Right Way
8 Templates Worth Copying First
The Other 16 Templates
Customizing or Building Your Own
FAQ

What Is a Web Scraping Template?

A template is a saved configuration that orchestrates two or three CrawlForge tools into one workflow with a business outcome attached. Instead of wiring search_web then scrape_structured then analyze_content yourself — and guessing every parameter — you copy a config that already does it.

Each template in the gallery carries:

A category — E-commerce, Research, Data Collection, Monitoring, AI & LLM, Sales, SEO, Content, or Advanced Scraping (nine in total).
A difficulty — beginner, intermediate, or advanced.
The tool chain it runs and a fixed credit cost per run (3–19 credits).
A copy-paste JSON config with sensible default parameters.

You run that config from any MCP client (Claude, Cursor, Windsurf), the crawlforge CLI, or the REST API. Same config, same shape of result.

Templates Gallery vs the scrape_template Tool

This trips people up, so let's be precise. CrawlForge has two different things with "template" in the name:

	Templates gallery	`scrape_template` tool
What it is	A library of multi-tool config chains	A single tool with 10 site schemas
Scope	Any workflow (pricing, SEO, research…)	10 specific sites (Amazon, LinkedIn, GitHub…)
Output	Whatever the chained tools return	Structured JSON for that one site
Cost	3–19 credits/run (sum of its tools)	1 credit/call
Use when	You want a whole workflow, ready-made	You want data from one popular site

If your target is one of the ten supported sites, reach for the tool — covered in depth in Scrape Amazon, LinkedIn & 8 More Sites With One Tool. For everything else — a full pricing-monitoring or lead-enrichment pipeline — you want a gallery template. This guide is about the gallery.

How to Use a Template the Right Way

Copy-paste is step one. Using a template well is six.

1. Pick by outcome, not by tool. Start from the job ("monitor competitor prices") and filter the gallery by category and difficulty. New to this? Start with the cheap, two-tool, beginner templates before reaching for a 19-credit research pipeline.

2. Read the config before you run it. Look at the tool order, the parameters, and whether it carries a schedule (hourly, daily, or weekly). Order matters: search_web finds the URLs, then scrape_structured extracts from them.

3. Swap the placeholders. Every config ships with example values — https://competitor-a.com/pricing, {company_name}, "product name", a default schema. Replace those with your real targets and the exact fields you want back. The schema is your output contract; trim it to what you'll actually use.

4. Do the credit math before you scale. A template's cost is just the sum of its tools' costs. Multiply by frequency: a 7-credit template run hourly is 168 credits/day. Here is the per-tool table the configs draw from:

Credits	Tools
1	`fetch_url`, `extract_text`, `extract_links`, `extract_metadata`, `scrape_template`
2	`scrape_structured`, `extract_content`, `map_site`, `process_document`, `localization`
3	`analyze_content`, `track_changes`, `extract_structured`, `extract_with_llm`
4	`summarize_content`, `crawl_deep`
5	`stealth_mode`, `scrape_with_actions`, `batch_scrape`, `search_web`, `generate_llms_txt`
10	`deep_research`

5. Run it from your stack of choice. In an MCP client, paste the goal and let the agent call the tools. From a terminal or cron job, use the crawlforge CLI. In an app, hit the REST API. All three share one API key and one credit balance.

6. Schedule and monitor. Templates built for monitoring carry a schedule. Pair them with track_changes so you act on diffs, not on every identical run.

8 Templates Worth Copying First

The gallery has 24. These eight cover the highest-demand jobs and span beginner to advanced. Expand each for the copy-paste config.

1. Competitor Pricing Monitor

E-commerce · intermediate · 7 credits/run · batch_scrape + scrape_structured

Scrape a set of competitor pricing pages on a schedule and normalize them into a clean plan / price / features structure.

Config + how to adapt

{
  "tools": [
    {
      "name": "batch_scrape",
      "params": {
        "urls": [
          "https://competitor-a.com/pricing",
          "https://competitor-b.com/pricing"
        ],
        "selectors": { "price": ".price", "name": "h1" }
      }
    },
    {
      "name": "scrape_structured",
      "params": {
        "schema": {
          "plans": [{ "name": "string", "price": "string", "features": ["string"] }]
        }
      }
    }
  ],
  "schedule": "daily"
}

Replace urls with your competitors' pricing pages, then tune selectors and schema to the fields you track. Keep schedule at daily for most pricing work. Full walkthrough: build an AI price-monitoring system.

2. Contact Enrichment Pipeline

Sales · intermediate · 7 credits/run · search_web + extract_metadata + extract_links

Turn a bare company name into an enriched record — official site, social handles, and key links.

Config + how to adapt

{
  "tools": [
    {
      "name": "search_web",
      "params": { "query": "{company_name} official website" }
    },
    {
      "name": "extract_metadata",
      "params": { "include": ["og:title", "og:description", "twitter:site"] }
    },
    {
      "name": "extract_links",
      "params": { "filter": ["linkedin.com", "twitter.com", "github.com"] }
    }
  ]
}

Drive {company_name} from your CRM export, and widen the extract_links filter to the domains you care about. Run it per row to enrich a whole list. Full walkthrough: build a lead-enrichment engine.

3. SEO Site Audit

SEO · beginner · 6 credits/run · map_site + extract_metadata + analyze_content

Crawl a site, pull every page's metadata, and score content quality — a fast, repeatable audit.

Config + how to adapt

{
  "tools": [
    {
      "name": "map_site",
      "params": { "url": "https://your-site.com", "max_depth": 3 }
    },
    {
      "name": "extract_metadata",
      "params": {}
    },
    {
      "name": "analyze_content",
      "params": { "metrics": ["readability", "topics", "sentiment"] }
    }
  ]
}

Point url at your domain and raise or lower max_depth to control crawl breadth (and cost). One of the cheapest templates to run regularly. Full walkthrough: automating SEO audits with CrawlForge.

4. AI Training Data Collector

AI & LLM · intermediate · 7 credits/run · batch_scrape + extract_content

Collect and clean web pages at scale into model-ready text — no navigation, no boilerplate.

Config + how to adapt

{
  "tools": [
    {
      "name": "batch_scrape",
      "params": {
        "urls": ["https://docs.example.com/page-1", "https://docs.example.com/page-2"],
        "format": "markdown"
      }
    },
    {
      "name": "extract_content",
      "params": { "format": "text", "remove_navigation": true }
    }
  ]
}

Feed urls from a sitemap or CSV, and keep remove_navigation on so menus and footers don't pollute your dataset. Full walkthrough: web scraping for AI training data pipelines.

5. Market Intelligence Dashboard

Research · advanced · 19 credits/run · deep_research + batch_scrape + summarize_content

The flagship. Run multi-source research, scrape the key industry sources, and summarize it all into a daily briefing.

Config + how to adapt

{
  "tools": [
    {
      "name": "deep_research",
      "params": {
        "query": "SaaS market trends and funding rounds",
        "sources": 10,
        "conflict_detection": true
      }
    },
    {
      "name": "batch_scrape",
      "params": {
        "urls": ["https://techcrunch.com", "https://www.saastr.com"],
        "format": "markdown"
      }
    },
    {
      "name": "summarize_content",
      "params": { "max_length": 300, "format": "bullet_points" }
    }
  ],
  "schedule": "daily"
}

Change the query to your market and swap urls for your trusted sources. At 19 credits/run it is the most expensive template here — run it daily, not hourly. Related: competitive intelligence with AI agents.

6. Review Sentiment Analyzer

E-commerce · intermediate · 10 credits/run · search_web + scrape_structured + analyze_content

Find reviews across platforms, structure them, and score sentiment and topics.

Config + how to adapt

{
  "tools": [
    {
      "name": "search_web",
      "params": { "query": "\"product name\" reviews", "max_results": 10 }
    },
    {
      "name": "scrape_structured",
      "params": {
        "schema": {
          "reviewer": "string",
          "rating": "number",
          "text": "string",
          "date": "string"
        }
      }
    },
    {
      "name": "analyze_content",
      "params": { "metrics": ["sentiment", "topics"] }
    }
  ]
}

Put your product in the query, raise max_results for more coverage, and keep the schema tight so sentiment scoring stays clean. Related: e-commerce product data extraction at scale.

7. Job Listings Scraper

Data Collection · intermediate · 7 credits/run · search_web + scrape_structured

Search job boards and pull listings into a structured feed — title, company, location, salary, date.

Config + how to adapt

{
  "tools": [
    {
      "name": "search_web",
      "params": { "query": "software engineer remote jobs 2026", "max_results": 20 }
    },
    {
      "name": "scrape_structured",
      "params": {
        "schema": {
          "title": "string",
          "company": "string",
          "location": "string",
          "salary": "string",
          "posted_date": "string"
        }
      }
    }
  ]
}

Change the query to your role and region, and add schema fields (remote flag, seniority) as needed. See it on the Job Listings Scraper template page.

8. Website Change Detector

Monitoring · beginner · 6 credits/run · fetch_url + extract_content + analyze_content

Watch a single page and surface when its content shifts — pricing, terms, or announcements.

Config + how to adapt

{
  "tools": [
    {
      "name": "fetch_url",
      "params": { "url": "https://example.com/page-to-monitor" }
    },
    {
      "name": "extract_content",
      "params": { "format": "text" }
    },
    {
      "name": "analyze_content",
      "params": { "metrics": ["topics"] }
    }
  ],
  "schedule": "hourly"
}

Set url to the page you care about and dial schedule to your tolerance for staleness — hourly for fast-moving pages, daily for the rest. Related: build a competitive-intelligence agent.

The Other 16 Templates

The remaining gallery entries, grouped by category — each is a copy-paste config on the templates page:

Research: News Aggregation Pipeline (11cr), Multi-Source Research Agent (12cr), Academic Paper Research (14cr).
Data Collection: Real Estate Listings Tracker (7cr), PDF Document Processor (6cr), Government Data Extractor (5cr).
Monitoring: Compliance Monitoring (9cr), Social Media Monitoring (12cr).
E-commerce: E-commerce Product Extraction (3cr).
AI & LLM: Documentation Knowledge Base (10cr).
Sales: Tech Stack Detector (3cr).
SEO: Link Building Prospector (7cr).
Content: Content Migration Tool (7cr), Localization Content Audit (7cr).
Advanced Scraping: Dynamic SPA Scraper (7cr), Stealth Data Extraction (7cr).

Customizing or Building Your Own

No template is a perfect fit out of the box — that is the point of step three. When a config gets you 80% there, swap the parameters and schema and you are done. When nothing fits:

Start from the closest template and rewrite its schema and parameters.
Compose tools yourself. Use scrape_structured when you know stable CSS selectors, or extract_with_llm when the layout shifts and you want schema-driven, layout-resilient extraction.
Request a template. If you want a recipe we don't ship yet, ask on Discord — popular requests get added to the gallery.

FAQ

What is a web scraping template?

A ready-to-use JSON config that chains multiple CrawlForge tools into one workflow with a specific outcome — price monitoring, lead enrichment, SEO auditing, and so on. Copy the config, swap in your URLs and schema, and run it from an MCP client, the crawlforge CLI, or the REST API. CrawlForge ships 24 templates across 9 categories.

Templates gallery vs the scrape_template tool?

The gallery is a library of multi-tool config chains for complete workflows (3–19 credits/run). The scrape_template tool is a single tool with pre-built schemas for 10 popular sites (Amazon, LinkedIn, GitHub…) at 1 credit/call. Use a gallery template for a whole workflow; use scrape_template for data from one of the ten supported sites.

How many credits does a template cost?

A template costs the sum of its tools per run, from 3 credits (E-commerce Product Extraction, Tech Stack Detector) to 19 (Market Intelligence Dashboard). A Competitor Pricing Monitor is batch_scrape (5) + scrape_structured (2) = 7. Multiply by frequency to budget: 7 credits hourly is 168/day.

Can I customize a template or change its schema?

Yes — that is the intended workflow. Every template ships with placeholders (example URLs, a default schema, sample queries) you replace with your real targets. The schema is your output contract, so trim or extend it. If nothing fits, start from the closest template or compose tools with scrape_structured / extract_with_llm.

How do I run a template?

Three ways, sharing one API key and credit balance: paste the goal into an MCP client like Claude, Cursor, or Windsurf; run it from a terminal or cron job with the crawlforge CLI; or call the REST API from an app. The same config produces the same result across all three.

Ready to run your first template? Free tier includes 1,000 credits and no credit card.

Start Free — 1,000 Credits Included

Or keep reading:

CrawlForge v4.2.2: New CLI + 3 Tools for Local AI Scraping

Simon — Mon, 18 May 2026 23:21:46 +0000

Today we are shipping CrawlForge v4.2.2, our biggest release since launch. It brings three new tools, a standalone command-line interface, and a quiet shift in how we think about web scraping for AI: most of it should run locally, on your own machine, without API keys.

This post is the umbrella for everything in 4.2.2. Three deep-dive guides follow in the next nine days.

TL;DR: v4.2.2 adds the crawlforge CLI (a standalone CLI that ships with crawlforge-mcp-server -- no MCP client needed), extract_with_llm (local LLM extraction via Ollama, no OpenAI/Anthropic key required), scrape_template (one-line scrapers for 10 popular sites), and list_ollama_models (free model discovery). Tool count goes from 20 to 23. Free tier still includes 1,000 credits. Install: npm install -g crawlforge-mcp-server.

What Shipped
The New CrawlForge CLI
Extract With LLM: Local AI Extraction
Scrape Template: Ten Sites, One Call
list_ollama_models: Free Model Discovery
Old Workflow vs v4.2.2 Workflow
Credit Costs
How to Upgrade
What Is Next

What Shipped

v4.2.2 adds four things:

The crawlforge CLI -- a standalone command-line tool, shipped in the crawlforge-mcp-server package, that exposes all 23 CrawlForge tools to your shell. No MCP client required.
extract_with_llm -- LLM-powered structured extraction that defaults to local Ollama. No external API key needed.
scrape_template -- pre-built scrapers for Amazon, LinkedIn, GitHub, YouTube, Reddit, Hacker News, Stack Overflow, npm, Product Hunt, and Twitter/X.
list_ollama_models -- a free discovery tool that lists models on your local Ollama instance.

Tool count goes from 20 to 23. The CLI is brand new -- it is not a tool, it is a delivery channel.

+----------------+       +-------------------+       +----------------+
|   Your Shell   | <-->  |  crawlforge CLI   | <-->  |  CrawlForge    |
|   (cron, CI)   |       |  (JSON in/out)    |       |   API + Tools  |
+----------------+       +-------------------+       +----------------+
                                  ^
                          No MCP handshake.
                          Just HTTPS + stdout.

The New CrawlForge CLI

The CLI is the shortest path from intent to scraped data. You install it once, set an environment variable, and every CrawlForge tool becomes a command:

npm install -g crawlforge-mcp-server
export CRAWLFORGE_API_KEY="cf_live_your_key_here"

crawlforge scrape https://example.com
crawlforge search "best MCP servers 2026"
crawlforge research "AI agent frameworks" --depth deep

Why does this matter? Because MCP is great for AI agents, but a lot of scraping work is not an AI agent task. It is a cron job. A CI step. A one-off pull from your terminal. For that, you want JSON on stdout that pipes into jq, not a JSON-RPC handshake.

Why have a CLI when MCP already exists?

MCP is optimized for AI agents picking tools dynamically. The CLI is optimized for humans typing commands and scripts piping JSON. Different shapes for different jobs:

Workflow	Best fit
Claude/Cursor agent	MCP
Cron job	CLI
GitHub Actions step	CLI
One-off terminal	CLI
Server in a loop	Raw API

All three paths hit the same backend, share the same credit balance, and use the same API key.

Read the complete CrawlForge CLI guide for the full command reference and real-world workflows.

Extract With LLM: Local AI Extraction

extract_with_llm is structured extraction powered by a language model. You hand it a URL and a schema, it gives you back JSON. The new part is that it defaults to local Ollama rather than calling OpenAI or Anthropic.

{
  "url": "https://news.ycombinator.com/item?id=123456",
  "schema": {
    "type": "object",
    "properties": {
      "title":    { "type": "string" },
      "points":   { "type": "number" },
      "comments": { "type": "number" }
    }
  },
  "provider": "ollama",
  "model": "llama3.1:8b"
}

Three things follow from the local-first default:

No third-party API costs. The LLM is free. You only pay 3 CrawlForge credits per extraction.
No data leaving your machine. Scraped content stays on localhost.
No new API key to manage. If Ollama is installed, you are done.

When to still use OpenAI or Anthropic

Local models are great for predictable schemas (titles, prices, counts, ratings). For long-form reasoning -- summarizing a 10,000-word article, classifying nuanced sentiment, extracting fields that require world knowledge -- a frontier model still wins.

Switch providers with one parameter:

crawlforge extract https://example.com \
  --provider anthropic \
  --model claude-sonnet-4-6

You pay the provider's per-token cost plus 3 CrawlForge credits. Same schema, same output shape.

Detailed guide: extract data with local LLMs.

Scrape Template: Ten Sites, One Call

scrape_template is for the long tail of scraping requests that all look the same: "get me product data from Amazon", "get me a GitHub repo's metadata", "get me the top posts on Hacker News today". You should not need to write CSS selectors for these. We did it once, we maintain it, you call it.

crawlforge template amazon-product "https://www.amazon.com/dp/B0CHX1W1XY"
crawlforge template github-repo "https://github.com/anthropics/anthropic-sdk-python"
crawlforge template hacker-news-front-page "https://news.ycombinator.com"

Ten templates ship in this release:

Template	What it returns	Credits
`amazon-product`	Product title, price, rating, reviews, images	1
`linkedin-profile`	Profile name, headline, location, about	1
`github-repo`	Repo metadata, stars, language, topics	1
`youtube-video`	Video title, views, channel, description	1
`reddit-thread`	Post title, score, author, body	1
`hacker-news-front-page`	Front-page story titles, points, URLs	1
`stackoverflow-question`	Question, answers, accepted, vote counts	1
`npm-package`	Package metadata, weekly downloads, version	1
`producthunt-launch`	Product name, tagline, upvotes, topics	1
`tweet`	Tweet text, author, URL, image	1

Full walkthrough with code: scrape Amazon, LinkedIn, and GitHub with one tool.

list_ollama_models: Free Model Discovery

Most useful as a sanity-check before running extract_with_llm. Lists every model on your local Ollama instance with name, size, and modified date.

crawlforge extract --list-ollama-models

Costs zero credits. It does no scraping, no LLM call -- it just queries Ollama's local API on 127.0.0.1:11434 and returns the result. If you have ever wondered which model you actually have installed, this is the answer.

Old Workflow vs v4.2.2 Workflow

Task	Pre-4.2.2	v4.2.2
Scrape from your terminal	curl + custom parser, or boot a Node REPL	`crawlforge scrape <url>`
Extract structured data with LLM	`extract_structured` (CSS selectors) or roll your own with Puppeteer + OpenAI	`extract_with_llm` (Ollama default)
Scrape Amazon, LinkedIn, GitHub	`scrape_structured` with hand-maintained selectors	`scrape_template` (we maintain selectors)
Run scraping in CI/cron	curl with API key in headers	`crawlforge <cmd>` with env var

Credit Costs

The three new tools follow our existing credit-cost model. No surprises:

Tool	Credits	Why
`list_ollama_models`	0	Free discovery helper
`scrape_template`	1	Single page, pre-built schema
`extract_with_llm`	3	LLM inference (provider-agnostic)

The CLI itself is free. It uses your existing API key and bills against your normal credit balance.

How to Upgrade

Existing users do not need to do anything. The new tools are live on all plans -- Free, Hobby, Professional, and Business -- and show up automatically in your MCP client.

Install the CLI

npm install -g crawlforge-mcp-server
export CRAWLFORGE_API_KEY="cf_live_..."
crawlforge --help

Add the export line to your shell profile (~/.zshrc, ~/.bashrc) so it persists. For CI, set CRAWLFORGE_API_KEY as a repository secret.

Try Ollama-powered extraction

# 1. Install Ollama (one-time)
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull a model (llama3.1:8b is a good start)
ollama pull llama3.1:8b

# 3. Run extraction through CrawlForge
crawlforge extract https://example.com \
  --provider ollama \
  --model llama3.1:8b

The first run pulls about 5 GB. After that, every extraction is local, free, and offline-capable.

What Is Next

We are working on three things for 4.3:

More templates -- Etsy, eBay, TikTok, Instagram, Google Maps. Send us requests on Discord.
Webhook delivery for batch_scrape -- get results pushed to your endpoint when long-running jobs complete.
CLI watch mode -- crawlforge track --watch for live diffs on monitored pages.

Ready to try the new tools? Free tier still includes 1,000 credits and no credit card.

Get 1,000 Free Credits

Or jump straight into the deep dives:

How to Scrape Websites with Claude Code (2026 Guide)

Simon — Fri, 17 Apr 2026 22:00:26 +0000

Claude Code can edit files, run shell commands, and write tests — but it cannot fetch live web pages on its own. Connect it to CrawlForge MCP and it gains 20 scraping tools that run straight from your terminal.

npm install -g crawlforge-mcp-server
npx crawlforge-setup  # paste your API key
# Now in Claude Code:
# > Fetch https://news.ycombinator.com and list the top 5 stories

This guide shows you how to scrape websites with Claude Code using CrawlForge MCP, from installation to stealth-mode bypass. Every code block below is runnable.

TL;DR: Claude Code has no network access by default. Install CrawlForge MCP and Claude Code gets 20 scraping tools — fetch_url, extract_content, scrape_with_actions, stealth_mode, and more. Free tier = 1,000 credits, no credit card.

The Problem: Claude Code Cannot Fetch URLs
Prerequisites
Step 1: Install CrawlForge MCP
Step 2: Get Your API Key
Step 3: Register the MCP Server with Claude Code
Step 4: Verify the Connection
Step 5: Your First Scrape
Full Working Example: Scrape a Pricing Page
Advanced: Scrape JavaScript-Rendered Sites
Tool Quick Reference
Troubleshooting
FAQ

The Problem: Claude Code Cannot Fetch URLs

By default, Claude Code has no network access. Ask it to "read this blog post" and it will tell you it cannot open URLs. The built-in WebFetch helper exists in Claude Desktop but is limited, rate-capped, and frequently blocked by Cloudflare, Akamai, and other edge protections.

CrawlForge MCP solves this by exposing 20 scraping tools — fetch_url, extract_content, scrape_structured, stealth_mode, deep_research, and more — as Model Context Protocol tools that Claude Code can call like any other function. For more background on the protocol itself, see the complete guide to MCP web scraping.

+------------+     MCP       +-------------+     HTTPS     +-----------+
| Claude Code| <-----------> | CrawlForge  | <-----------> |  Target   |
| (terminal) |  JSON-RPC     |   Server    |               |   site    |
+------------+   over stdio  +-------------+               +-----------+
                                    |
                                Credits +
                                Stealth +
                                Actions

Prerequisites

Node.js 18+ — check with node --version
Claude Code — install with npm install -g @anthropic-ai/claude-code
A CrawlForge account — free at crawlforge.dev/signup (1,000 credits included, no credit card)

Step 1: Install CrawlForge MCP

npm install -g crawlforge-mcp-server

Verify the install:

crawlforge-mcp-server --version
# crawlforge-mcp-server 3.0.16

Step 2: Get Your API Key

Go to crawlforge.dev/signup and create an account.
Open the dashboard at crawlforge.dev/dashboard/api-keys.
Copy the key — it starts with cf_live_.

Step 3: Register the MCP Server with Claude Code

The fastest path is the setup wizard:

npx crawlforge-setup

It writes the correct entry to ~/.config/claude-code/mcp.json (Linux/macOS) or %APPDATA%\claude-code\mcp.json (Windows).

Prefer manual configuration? Here's the exact JSON

Add this to your Claude Code MCP config file:

{
  "mcpServers": {
    "crawlforge": {
      "command": "crawlforge-mcp-server",
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

Restart Claude Code so it picks up the new server.

Step 4: Verify the Connection

Open Claude Code and run:

/mcp

You should see crawlforge listed as connected with 20 tools available. If not, jump to Troubleshooting.

Step 5: Your First Scrape

Paste this prompt into Claude Code:

Fetch https://news.ycombinator.com using CrawlForge and give me the
top 5 story titles with their URLs as a JSON array.

Claude Code will call fetch_url (1 credit), parse the HTML, and return something like:

[
  { "title": "Show HN: My side project", "url": "https://example.com/post/1" },
  { "title": "Why X is changing Y", "url": "https://example.com/post/2" }
]

That is it. You are scraping.

Full Working Example: Scrape a Pricing Page

Here is a realistic task: extract pricing tiers from a SaaS site. Paste this prompt:

Use scrape_structured to extract pricing from https://crawlforge.dev/pricing.
Return an array of { plan, price, credits, features[] }.

See what Claude Code builds behind the scenes

// What Claude Code sends to CrawlForge via MCP
const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_structured', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CRAWLFORGE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://crawlforge.dev/pricing',
    selectors: {
      plan: '.pricing-card h3',
      price: '.pricing-card .price',
      credits: '.pricing-card .credits',
      features: '.pricing-card ul li',
    },
  }),
});

const data = await response.json();
console.log(data.plan); // ['Free', 'Hobby', 'Professional', 'Business']

Cost: 2 credits. Compare that to running a headless browser locally: zero infrastructure, no Puppeteer debugging, no Cloudflare roulette.

Advanced: Scrape JavaScript-Rendered Sites

Some sites render pricing or product data through client-side React. fetch_url returns the pre-hydration HTML shell and misses the data. Switch to scrape_with_actions (5 credits):

scrape_with_actions — clicks, waits, and scrolls for SPAs

// Prompt Claude Code with the goal, and it generates this call
const payload = {
  url: 'https://app.example.com/dashboard',
  actions: [
    { type: 'wait', selector: '.data-grid', timeout: 5000 },
    { type: 'click', selector: 'button.load-more' },
    { type: 'wait', timeout: 1500 },
  ],
  formats: ['json', 'markdown'],
};

const result = await fetch('https://crawlforge.dev/api/v1/tools/scrape_with_actions', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CRAWLFORGE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(payload),
}).then(r => r.json());

For Cloudflare and Akamai-protected sites, use stealth_mode (also 5 credits). The fingerprint-rotation tradeoffs are covered in the stealth mode deep dive.

Tool Quick Reference

Tool	Credits	When to use
`fetch_url`	1	Static HTML, you will parse yourself
`extract_text`	1	Clean readable text from article pages
`extract_content`	2	Readability-style main content extraction
`scrape_structured`	2	CSS selectors into typed fields
`search_web`	5	You do not know the URL yet
`scrape_with_actions`	5	SPA requires clicks, waits, scrolls
`stealth_mode`	5	Anti-bot systems (Cloudflare, DataDome)
`deep_research`	10	Multi-source research with citations

Full list in the 20-tools overview.

Troubleshooting

MCP server failed to start

Confirm crawlforge-mcp-server is on your PATH:

which crawlforge-mcp-server

If empty, reinstall globally:

npm install -g crawlforge-mcp-server

Unauthorized or 401 errors

Your API key is missing or malformed. It must start with cf_live_. Re-export in your shell and restart Claude Code:

export CRAWLFORGE_API_KEY="cf_live_..."

Insufficient credits

Check usage at crawlforge.dev/dashboard/usage. Free tier = 1,000 credits/month. Upgrade to Hobby ($19/mo) for 25,000.

Tools list is empty when running /mcp

MCP config is not being read. Check the right path for your OS:

macOS: ~/Library/Application Support/claude-code/mcp.json
Linux: ~/.config/claude-code/mcp.json
Windows: %APPDATA%\claude-code\mcp.json

Cloudflare 403 on every fetch

Swap fetch_url for stealth_mode. If you still get blocked, the target uses server-side JA3/JA4 fingerprinting — open a GitHub issue with the URL.

FAQ

Can Claude Code scrape websites without CrawlForge?

Not reliably. Claude Code has no built-in network access, and the WebFetch helper in Claude Desktop is rate-limited and blocked by most anti-bot systems. CrawlForge MCP adds 20 dedicated scraping tools that handle static HTML, JavaScript-rendered pages, and Cloudflare-protected sites.

How much does it cost to scrape with Claude Code?

CrawlForge uses a credit model: basic fetches cost 1 credit, structured extraction 2 credits, search 5, stealth 5, deep research 10. Free accounts get 1,000 credits per month with no credit card. The Hobby plan ($19/mo) includes 25,000 credits.

Why do I get 403 errors when Claude Code fetches certain URLs?

Sites protected by Cloudflare, DataDome, or Akamai block generic HTTP clients via TLS fingerprinting and JavaScript challenges. Switch from fetch_url (1 credit) to stealth_mode (5 credits), which rotates browser fingerprints and solves challenges automatically.

Does CrawlForge MCP work with Claude Code on Windows?

Yes. Install via npm, run npx crawlforge-setup, and the config lands at %APPDATA%\claude-code\mcp.json. Node.js 18+ is the only system requirement. Windows users should run the setup command from PowerShell or Windows Terminal for the cleanest experience.

What is the difference between fetch_url and scrape_with_actions?

fetch_url returns raw HTML via a fast HTTP request (1 credit). scrape_with_actions spins up a headless browser, executes clicks/waits/scrolls, then captures the hydrated DOM (5 credits). Use fetch_url for static pages and scrape_with_actions only when JavaScript rendering is required.

Next Steps

Read the CrawlForge quick start for five copy-paste examples
Browse the getting started docs for full API reference
Compare MCP clients in Claude Desktop vs Claude Code
Evaluate alternatives at Firecrawl alternative

Start Free — 1,000 Credits Included

im about to launch my new saas project next tuesday. kind of excited this will be my first. here is the link in case anyone wants to try it out for free. hope everyone has a great weekend. https://www.crawlforge.dev/

Simon — Fri, 27 Mar 2026 18:18:29 +0000

CrawlForge - Enterprise Web Scraping for AI Applications

18 specialized MCP tools for seamless web scraping and AI-ready data extraction

crawlforge.dev

The Complete Guide to MCP Web Scraping: Everything Developers Need to Know

Simon — Fri, 27 Mar 2026 17:45:08 +0000

The Model Context Protocol (MCP) has fundamentally changed how AI assistants interact with the web. If you've been building with Claude, Cursor, or any MCP-compatible client, you've probably wondered: what's the best way to give your AI agent access to live web data?

This guide covers everything -- from foundational MCP concepts to all 18 CrawlForge tools and how to pick the right one for each job.

TL;DR: MCP is the open standard for connecting AI to tools. CrawlForge is the most comprehensive MCP web scraping server with 18 tools -- 4x more than alternatives. Free tier includes 1,000 credits, no credit card required.

Part 1: Understanding MCP
Part 2: The MCP Web Scraping Ecosystem
Part 3: All 18 Tools Explained
Part 4: Integration Guide
Part 5: Best Practices
Part 6: Getting Started

Part 1: Understanding MCP

What is the Model Context Protocol?

MCP is an open standard developed by Anthropic that lets AI assistants connect to external tools and data sources. Think of it as a universal adapter between AI models and specialized tools.

+--------------+      +----------------+      +---------------+
|   Claude     | <--> |  MCP Server    | <--> |  External     |
|  (AI Model)  |      |  (CrawlForge)  |      |  Resources    |
+--------------+      +----------------+      +---------------+
                             ^
                      MCP Protocol
                   (JSON-RPC over stdio)

Why MCP Matters for Web Scraping

Before MCP, getting real-time web data into AI was painful:

Approach	Problems
Training data	Outdated, knowledge cutoff
RAG (Retrieval)	Limited to indexed documents
Function calling	Requires custom implementation
Browser plugins	Inconsistent, security concerns

MCP solves these with:

Standardized interface -- One protocol for all tools
Real-time data -- Fresh information from any source
Tool composability -- Combine multiple tools seamlessly
Security model -- Controlled access to external resources

How MCP Works

MCP uses a client-server architecture with JSON-RPC:

Step 1: Server Discovery

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["crawlforge-mcp-server"]
    }
  }
}

Step 2: Tool Registration -- the server advertises its capabilities

{
  "tools": [
    {
      "name": "fetch_url",
      "description": "Fetch content from a URL",
      "inputSchema": {
        "type": "object",
        "properties": {
          "url": { "type": "string", "format": "uri" }
        },
        "required": ["url"]
      }
    }
  ]
}

Step 3: Tool Invocation -- the AI calls a tool

{
  "method": "tools/call",
  "params": {
    "name": "fetch_url",
    "arguments": { "url": "https://example.com" }
  }
}

Step 4: Response -- structured data comes back

{
  "result": {
    "content": "<html>...",
    "status": 200,
    "headers": {}
  }
}

Part 2: The MCP Web Scraping Ecosystem

Several MCP servers provide web scraping capabilities:

Server	Tools	Focus
CrawlForge	18	Comprehensive scraping, research, stealth
Firecrawl	~5	Basic scraping and crawling
Browser MCP	~3	Browser automation
Fetch MCP	1	Simple HTTP requests

CrawlForge: ████████████████████ 18 tools
Firecrawl:  █████                 5 tools
Browser:    ███                   3 tools
Fetch:      █                     1 tool

Part 3: CrawlForge's 18 Tools Explained

Basic Scraping (1-2 credits)

1. fetch_url (1 credit) -- Fetch raw HTML from any URL

The foundation of web scraping. Always try this first.

// Usage: "Fetch https://example.com"
// Returns:
{
  html: "<html>...",
  status: 200,
  headers: { /* ... */ },
  timing: { total: 523 }
}

When to use: Starting point for any scraping task.

2. extract_text (1 credit) -- Clean text extraction

Removes HTML tags, scripts, and styles. Returns readable text with word count and reading time.

When to use: Blog posts, articles, documentation where you need readable text.

3. extract_links (1 credit) -- Discover all links on a page

{
  url: "https://example.com",
  filter_external: true  // Internal links only
}
// Returns: { links: [{ href: "/about", text: "About Us" }, ...], total: 45 }

When to use: Site exploration, finding pages to scrape, building sitemaps.

4. extract_metadata (1 credit) -- SEO metadata extraction

Pulls title, description, Open Graph, JSON-LD structured data.

When to use: SEO analysis, content previews, structured data extraction.

Structured Extraction (2-3 credits)

5. scrape_structured (2 credits) -- CSS selector-based extraction

{
  url: "https://example.com/products",
  selectors: {
    title: "h1.product-title",
    price: "span.price",
    description: ".product-description"
  }
}
// Returns: { data: { title: "Product Name", price: "$99.99", description: "..." } }

When to use: E-commerce scraping, structured data, known page layouts.

6. extract_content (2 credits) -- Intelligent article extraction

Like Readability -- returns title, author, published date, clean content, images, reading time.

When to use: News articles, blog posts, editorial content.

7. map_site (2 credits) -- Site structure discovery

{
  url: "https://example.com",
  max_urls: 1000,
  include_sitemap: true
}
// Returns: pages[], structure tree

When to use: Site audits, crawl planning, content discovery.

8. analyze_content (3 credits) -- NLP analysis

Language detection, sentiment scoring, topic extraction, named entity recognition, readability analysis.

When to use: Content analysis, sentiment tracking, topic extraction.

Advanced Scraping (4-5 credits)

9. process_document (2 credits) -- PDF and document handling

{
  source: "https://example.com/report.pdf",
  sourceType: "pdf_url"
}
// Returns: { text: "...", pages: 15, metadata: { author, created } }

When to use: Research papers, reports, documentation PDFs.

10. summarize_content (4 credits) -- AI-powered summarization

Returns concise summary, key points, and word reduction percentage.

When to use: Long documents, research synthesis, content digests.

11. crawl_deep (4 credits) -- Multi-page crawling

{
  url: "https://example.com",
  max_depth: 3,
  max_pages: 100,
  include_patterns: ["/blog/*"],
  exclude_patterns: ["/admin/*"]
}

When to use: Full site scraping, content aggregation, archiving.

12. batch_scrape (5 credits) -- Parallel scraping of multiple URLs

Scrape up to 50 URLs in parallel with concurrency control.

When to use: Multiple known URLs, competitor monitoring, price tracking.

13. scrape_with_actions (5 credits) -- Browser automation

{
  url: "https://example.com/app",
  actions: [
    { type: "wait", selector: ".content" },
    { type: "click", selector: "#load-more" },
    { type: "wait", timeout: 2000 },
    { type: "scroll", selector: "body" },
    { type: "screenshot" }
  ]
}

When to use: SPAs, infinite scroll, dynamic content, login-required pages.

14. search_web (5 credits) -- Google search integration

{
  query: "web scraping best practices 2026",
  limit: 10,
  site: "github.com"  // Optional site filter
}

When to use: Discovery, finding sources, research starting point.

Specialized Tools (2-10 credits)

15. stealth_mode (5 credits) -- Anti-detection bypass

Hides WebDriver, randomizes fingerprints, simulates human behavior. Configurable stealth levels.

When to use: Protected sites, Cloudflare bypass, anti-bot evasion.

16. track_changes (3 credits) -- Content monitoring

Set a baseline, then detect changes with before/after diffs and significance scoring.

When to use: Price monitoring, competitor tracking, content updates.

17. localization (2 credits) -- Geo-targeted scraping

Set country code and language to get region-specific content and pricing.

When to use: Regional pricing, localized content, geo-restricted data.

18. deep_research (10 credits) -- Multi-source research

{
  topic: "quantum computing commercialization",
  maxUrls: 50,
  enableSourceVerification: true,
  enableConflictDetection: true
}
// Returns: { synthesis, sources[], conflicts[], citations[] }

When to use: Research projects, due diligence, market analysis.

Part 4: Integration Guide

Claude Code Setup

# 1. Install CrawlForge MCP server
npm install -g crawlforge-mcp-server

# 2. Run setup wizard
npx crawlforge-setup

# 3. Add to Claude Code
claude
> /mcp add crawlforge npx crawlforge-mcp-server

# 4. Verify
> /mcp list
# Should show: crawlforge (18 tools)

Claude Desktop Setup

Edit your config file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["crawlforge-mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

Custom Application (MCP SDK)

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

const transport = new StdioClientTransport({
  command: "npx",
  args: ["crawlforge-mcp-server"],
  env: { CRAWLFORGE_API_KEY: process.env.CRAWLFORGE_API_KEY }
});

const client = new Client({ name: "my-app", version: "1.0.0" });
await client.connect(transport);

// List available tools
const tools = await client.listTools();
console.log(`Available tools: ${tools.tools.length}`);

// Call a tool
const result = await client.callTool({
  name: "fetch_url",
  arguments: { url: "https://example.com" }
});

Part 5: Best Practices

Credit Optimization

Pick the cheapest tool that gets the job done:

Goal	Expensive	Efficient
Check if page exists	`deep_research` (10)	`fetch_url` (1)
Get article text	`scrape_with_actions` (5)	`extract_content` (2)
Find competitor URLs	`search_web` x 10 (50)	`extract_links` (1)
Scrape 20 product pages	`fetch_url` x 20 (20)	`batch_scrape` (5)

Error Handling

try {
  const result = await fetchUrl(url);
  if (result.status >= 400) {
    // Try with stealth mode for blocked requests
    return await stealthMode(url);
  }
  return result;
} catch (error) {
  await sleep(retryDelay * attempt);
  return retry(url, attempt + 1);
}

Rate Limiting

Respect target sites:

// Good: Reasonable delays
for (const url of urls) {
  await scrape(url);
  await sleep(1000 + Math.random() * 2000); // 1-3s delay
}

// Better: Use batch_scrape with built-in rate limiting
await batchScrape(urls, { delayBetweenRequests: 1500 });

Caching

Don't scrape the same URL twice:

const cache = new Map<string, ScrapedContent>();

async function smartScrape(url: string) {
  if (cache.has(url)) return cache.get(url);

  const result = await fetchUrl(url);
  cache.set(url, result);
  return result;
}

Part 6: Getting Started

Free Tier -- 1,000 one-time trial credits. All 18 tools available. No credit card required.

npm install -g crawlforge-mcp-server
npx crawlforge-setup

What 1,000 Credits Gets You

Use Case	Tool	Credits	Capacity
Basic scraping	`fetch_url`	1	1,000 pages
Article extraction	`extract_content`	2	500 articles
Site mapping	`map_site`	2	500 sites
Batch jobs	`batch_scrape`	5	200 batches
Research projects	`deep_research`	10	100 topics

Key Takeaways

MCP is the standard -- All major AI assistants support it
CrawlForge leads with 18 tools -- 4x more than alternatives
Start simple -- Use fetch_url (1 credit) before reaching for advanced tools
Combine tools -- Chain operations for powerful workflows
Be ethical -- Respect robots.txt and rate limits

More from CrawlForge:

Start Free -- 1,000 Credits Included

DEV Community: Simon

We stopped trusting our test suite and pointed 28 tools at the real web

Table of contents

Why six passing tests meant nothing

Product data from the source

Reddit-wide search after PullPush went dark

Change tracking that actually fires

Content that was being deleted

Local LLMs finally work

Browser automation and the stealth wedge

Language detection and PDF tables

One copy of the extractors

How to upgrade

Web Scraping API Credits That Don't Expire

Why Do Most Scraping API Credits Expire?

The CrawlForge Model

What 1,000 Credits Actually Buys

How Firecrawl Handles the Same Question

When a Subscription Is the Better Deal

FAQ

Start Without a Clock

Reddit 403'd our stealth browser, so we stopped scraping reddit.com entirely

Table of contents

What shipped

Why reddit.com cannot be scraped

One tool, three modes

The archives: Arctic Shift and PullPush

How routing works

Honest caveats

Using it from MCP and REST

What about the Reddit thread template

Credit cost

How to upgrade

Answer 3 Questions, Get 1,000 Free CrawlForge Credits

How to claim your 1,000 free credits

Who qualifies

What 1,000 credits actually buys

The survey stays anonymous if you want it to

Why we are doing this

CrawlForge v5.0.0: Security, Correctness, MCP Spec

Table of contents

What actually shipped

The one breaking change: Node 20

Phase 1: the SSRF bypass we shipped

Phase 2: 52 ways tools were silently wrong

Phase 3: safe to run for days

Phase 4: HTTP mode only ever had one session

Phase 5: zero npm audit vulnerabilities

Phase 6: MCP spec adoption

Pricing and how to upgrade

My Claude Code skills never auto-activated either — here's what we fixed

Table of contents

Before / after

7 Claude Agent Skills that actually auto-activate

Two new scrape formats: branding and screenshot

Built-in scheduled change monitoring

Security: controls that were advertised but silently broken

Pricing and how to upgrade

How to Give ChatGPT Web Scraping with MCP Connectors (2026)

Table of Contents

What ChatGPT Connectors Are

Which Plans Can Use Them

The Transport Catch: Remote Only

Why CrawlForge Needs a Wrapper

Build the Bridge

Add the Connector in ChatGPT

Auth and Safety

A Simpler Alternative

How to Use Web Scraping Templates the Right Way (2026)

Table of Contents

What Is a Web Scraping Template?

Templates Gallery vs the scrape_template Tool

How to Use a Template the Right Way

8 Templates Worth Copying First

1. Competitor Pricing Monitor

2. Contact Enrichment Pipeline

3. SEO Site Audit

4. AI Training Data Collector

5. Market Intelligence Dashboard

6. Review Sentiment Analyzer