DEV Community

AgentStack Notes
AgentStack Notes

Posted on Originally published at dapper-shortbread-aad01d.netlify.app AI-assisted

Firecrawl's Crawl Limit, Credit Pre-Check and Webhooks for AI Agents: What the Docs Say

Originally published at AgentStack Notes. Last fact-checked: November 17, 2026.

Disclosure: This article was drafted with AI assistance and checked against Firecrawl's official documentation. We may receive a commission if you sign up through the links marked † in this article (Firecrawl †), at no extra cost to you. We have not run production workloads on Firecrawl. See our Advertising Disclosure.

Note on experience: we have not run Firecrawl crawls, tested its webhooks or measured its credit use. Pages were read through a web-fetch tool that summarises them, on November 17, 2026. Check the live docs before relying on any statement. "Not stated" means the page we read did not say.

Give an agent the instruction "crawl this documentation site" and a single request may turn into thousands of billable pages. Firecrawl † documents a handful of controls for this: a page limit, a credit check at crawl start, async job status, a cancel endpoint and webhooks. Below is what the docs say, where two doc pages disagree, and what is left unstated.

1. The one cap that is documented: limit

limit is the "maximum number of pages to crawl", and the API reference lists its default as 10000. Leave it out and the crawl may run to 10,000 pages. The crawl page says each crawled page uses 1 credit, JSON mode adds 4 credits per page, and PDF parsing costs 1 credit per PDF page. By those numbers, a default-limit crawl of a big site can be a five-figure credit event if every page is returned.

Other parameters narrow what is crawled rather than what it may cost: includePaths and excludePaths (regex arrays, up to 1,000 entries each), maxDiscoveryDepth, allowExternalLinks (default false), allowSubdomains (default false) and crawlEntireDomain (default false). maxDiscoveryDepth counts hops from the root URL, not path segments; its default is not stated.

No parameter that caps total credits is documented on the crawl pages we read.

2. The credit pre-check

The crawl page says the endpoint verifies that your remaining credits cover limit before starting, and returns 402 (Payment Required) if not; it suggests lowering limit. Our reading, not vendor guidance: a small site can still get a 402 if limit is high and the balance is low. Whether credits are reserved, and whether add-ons are counted in the check, is not stated.

3. Status polling

POST /v2/crawl returns a job ID; poll GET /v2/crawl/{id}. The crawl page lists scraping, completed, failed and cancelled; the status endpoint's field list shows three values, so the pages are not fully consistent. The response includes total, completed, creditsUsed and expiresAt.

  • completed == total is not proof that every discovered page succeeded; use the Get Crawl Errors endpoint for pages attempted but not returned.
  • A next URL appears for responses over 10 MB and can also appear on failed or cancelled crawls, so it is not an exit condition.
  • Results stay available through the API for 24 hours after completion, then remain in the activity logs.

4. Cancelling

The crawl page lists cancelled but describes no way to cancel. The API reference has DELETE /crawl/{id} (200 with "status": "cancelled", or 404). Whether already-used credits are refunded is not stated.

5. Webhooks

  • Events: crawl.started, crawl.page, crawl.completed (the events option lists completed, page, failed, started); default is all events.
  • Config: HTTPS url (required), optional headers, metadata and events.
  • Delivery: answer 2xx within 10 seconds. On timeout, non-2xx or network error, retries come 1, 5 and 15 minutes after the failure; after three failed retries the webhook is marked failed.
  • Signature: the crawl page describes an X-Firecrawl-Signature header (HMAC-SHA256, sha256=...) to verify with a timing-safe comparison; the webhooks overview we read says signature verification is not stated.
  • Payload: success, type, id, webhookId, data, metadata, and error when success is false.

Where the pages differ: the crawl page says an unscrapable page arrives with success: false; the events page we read says how failed pages appear is not stated. Do not read a missing crawl.page as "skipped" without checking the live docs and Get Crawl Errors.

6. A cautious checklist (our reading)

  • Set limit explicitly; constrain with includePaths, excludePaths and maxDiscoveryDepth first.
  • Treat a 402 at crawl start as "lower the limit or top up", not a transient error.
  • Read creditsUsed from status; fetch results within 24 hours.
  • Make webhook handlers fast and idempotent (webhookId identifies each delivery).

7. What we could not confirm

  • Any parameter that caps total credits for a crawl.
  • Whether the pre-check reserves credits, and how it counts add-ons.
  • Whether credits are refunded on cancel.
  • Whether failed pages in a crawl are charged.
  • The default for maxDiscoveryDepth.
  • A maximum number of URLs per batch scrape.
  • Whether the signature header is documented consistently across pages.

Sources: docs.firecrawl.dev (Crawl feature page; API reference for crawl start, status and cancel; Webhooks overview and event types; Batch scrape page). Checked November 17, 2026.

More: the original article, and the earlier piece on Firecrawl credits.

Top comments (0)