Feed an LLM anything current and you run into the same problem within a week. The URL you want returns a consent modal, a skeleton loader, or 200KB of JavaScript where the article should be. From there it is a build-or-rent decision: stand up your own headless browser fleet, or pay somebody per API call to keep one running.
I priced the five options I keep seeing in production against a single workload, and published the long version of this comparison on DevToolLab: Best Web Scraping APIs for AI in 2026. This is the compressed take.
Two data points frame the market. Firecrawl's repository crossed 176,000 GitHub stars, more than the open source projects of Bright Data, Zyte and Apify put together. And Apify's State of Web Scraping Report 2026, out January 29, 2026, found 62.5 percent of scraping teams spending more on infrastructure than a year earlier. Demand is not the constraint. Price is.
Who Actually Runs This
The Apify survey is worth reading for the buyer profile alone. Proxy spend rose year over year for 58.3 percent of respondents, total infrastructure spend for 62.5 percent, and nearly a quarter saw that bill climb by more than 30 percent, mostly because target sites keep hardening their anti-bot layers.
AI adoption inside these teams is still ahead of its own hype: 54.2 percent have not put AI anywhere in the workflow yet, though 66.2 percent intend to try, and of the minority already using an AI-assisted tool, 72.7 percent said it measurably helped. The population is also smaller than the enterprise sales pitch implies. 35.8 percent freelance, 49.1 percent sit inside a startup or SMB.
One Workload, Five Bills
No vendor publishes what the others charge, so pick a workload and price it yourself. Mine: 100,000 pages a month into a RAG index, priced from each vendor's own rate card as of September 3, 2026.
// Per-page cost of a 100,000-page/month RAG ingestion job.
const PAGES = 100_000
const options = [
// Firecrawl Standard covers exactly 100k credits; 1 credit = 1 page.
{ vendor: 'Firecrawl Standard (annual prepay)', monthly: 83 },
{ vendor: 'Firecrawl Standard (month to month)', monthly: 99 },
// Bright Data bills per successfully delivered record, not per attempt.
{ vendor: 'Bright Data (standard $1.00/1k)', monthly: (PAGES / 1000) * 1.0 },
{ vendor: 'Bright Data (promo $0.75/1k)', monthly: (PAGES / 1000) * 0.75 },
// Zyte prices by site-difficulty tier; these are HTTP responses.
{ vendor: 'Zyte pay-as-you-go, simple sites', monthly: (PAGES / 1000) * 0.13 },
{ vendor: 'Zyte pay-as-you-go, hard sites', monthly: (PAGES / 1000) * 1.27 },
{ vendor: 'Zyte $500 commitment, simple sites', monthly: (PAGES / 1000) * 0.06 },
{ vendor: 'Zyte $500 commitment, hard sites', monthly: (PAGES / 1000) * 0.61 },
// Crawl4AI is free; you rent the box that runs headless Chromium.
{ vendor: 'Crawl4AI self-hosted (4GB VPS)', monthly: 24 },
]
const rows = options
.map(o => ({ ...o, perPage: o.monthly / PAGES }))
.sort((a, b) => a.perPage - b.perPage)
const width = Math.max(...rows.map(r => r.vendor.length))
for (const r of rows) {
console.log(
`${r.vendor.padEnd(width)} $${r.monthly.toFixed(2).padStart(7)}/mo $${r.perPage.toFixed(5)}/page`
)
}
const spread = rows.at(-1).perPage / rows[0].perPage
console.log(`\nCheapest to most expensive: ${spread.toFixed(1)}x, same 100,000 pages.`)
Zyte $500 commitment, simple sites $ 6.00/mo $0.00006/page
Zyte pay-as-you-go, simple sites $ 13.00/mo $0.00013/page
Crawl4AI self-hosted (4GB VPS) $ 24.00/mo $0.00024/page
Zyte $500 commitment, hard sites $ 61.00/mo $0.00061/page
Bright Data (promo $0.75/1k) $ 75.00/mo $0.00075/page
Firecrawl Standard (annual prepay) $ 83.00/mo $0.00083/page
Firecrawl Standard (month to month) $ 99.00/mo $0.00099/page
Bright Data (standard $1.00/1k) $ 100.00/mo $0.00100/page
Zyte pay-as-you-go, hard sites $ 127.00/mo $0.00127/page
A 21x spread on identical volume, and the ordering is doing something sneaky. Two caveats before you read anything into it. Zyte's commitment rows assume you are already spending $500 a month, so the $6 line is a marginal rate, not a bill. And Firecrawl's $83 only exists if you prepay twelve months; month to month it is $99, which puts it beside Bright Data's standard rate rather than under it.
Apify is missing from the list because it cannot be on it. It meters compute units, not pages, so the number depends on your actor's code rather than your page count.
Firecrawl
Firecrawl takes a URL and hands back clean Markdown or structured JSON in one call. Scrape, Crawl and Map each cost 1 credit per page with JavaScript rendering already included, which is the part that usually carries a surcharge elsewhere.
Version 2.11.0, shipped June 19, 2026, added a research index over 3 million arXiv papers, automatic PII redaction, and a deterministicJson format that caches a per-site extractor so repeat crawls stop paying an LLM to re-learn the same page shape.
The catches: Stealth Mode costs 5 credits a page, Search and Interact bill above the base rate, and credits expire monthly on everything except Scale. The self-hosted repo is also behind the hosted product, since the managed anti-bot layer is not in the open source code.
Priced yearly: free tier at 1,000 credits, Hobby $16/month, Standard $83/month for 100,000, Scale $599/month for a million. Month to month those are $19, $99 and $749. AGPL-3.0 core, MIT SDKs.
Apify
Apify is a marketplace rather than a single endpoint. You run Actors, small scraping programs you write yourself or pull from the 67,000 already published, and you are billed for compute: one CU equals 1GB of RAM for an hour, at $0.20 down to $0.13 depending on plan.
That model rewards good code and punishes lazy code. A Cheerio actor parsing static HTML is genuinely cheap. Swap in Playwright to render the pages Cheerio skipped and the same job burns CU several times faster. There is no per-page figure to budget from until you have written the actor and watched it run.
Two dates matter. Apify went live on x402 in June 2026, so an agent can pay per run instead of holding a prepaid balance. And monthly actor rental retires October 1, 2026, with anything unmigrated moved automatically to pay-per-usage.
Plans: free with $5 usage, Starter $19/month, Scale $199/month, Business $999/month. The platform is proprietary; its Crawlee library is Apache-2.0.
Crawl4AI
Crawl4AI is what the managed vendors sell, unbundled and unmetered. Point it at a URL, get LLM-ready Markdown, no key, no invoice.
It is past 81,000 stars, runs headless Chromium, handles CSS-selector and LLM-guided extraction, and ships a Docker image, so nothing has to leave your infrastructure. Version 0.9.3, released August 31, 2026, was a security release closing five coordinated-disclosure bugs including an arbitrary file write and an SSRF in the PDF path. Worth noting if you are pinned to something older.
What it does not solve is the reason the paid tiers exist. No managed proxy pool, no SLA. The first time a target fingerprints your IP you are shopping for proxies, and that bill is now yours to negotiate.
Free, Apache-2.0, your own compute.
Bright Data
Bright Data is the oldest name here, built on a residential proxy network it advertises at 400 million monthly IPs across 195 countries, with a Web Scraper API sitting on top.
Its best property is billing per successfully delivered record, so blocked attempts cost nothing. It ships more than 1,500 site-specific scrapers that handle parsing for you, and bulk jobs take up to 5,000 URLs per request. Standard pricing is $1.00 per 1,000 records, or $0.75 with promo code APIS25 for six months.
No self-hosted path exists, and the catalog is sprawling: proxies, SERP API, Unlocker and datasets are all separate line items.
Zyte
Zyte was Scrapinghub until February 2021 and still employs the team maintaining Scrapy, which means a lot of this industry's existing spiders were written against their work.
Zyte prices by difficulty, not volume: five tiers from Simple to Advanced, different rates for plain HTTP versus browser-rendered responses, and a lower rate at every tier once you commit to a monthly spend. A $5 trial credit runs for 30 days.
The tradeoff is that you cannot budget in advance, because Zyte classifies your targets, not you. The same 1,000 requests land anywhere between $0.06 and $16.08 depending on tier, commitment and render mode. That is a 268x range inside one rate card. Existing Scrapy users also get Scrapy Cloud, which is usually the cheaper migration than rewriting spiders.
Side by Side
| Tool | Billing model | Entry price | Self-host | License |
|---|---|---|---|---|
| Firecrawl | Credits per page | $16/month yearly, $19 monthly | Core only | AGPL-3.0 |
| Apify | Compute units | $19/month plus usage | Via Crawlee | Proprietary |
| Crawl4AI | None | $0 | Yes | Apache-2.0 |
| Bright Data | Per successful record | $0.75 to $1.00 per 1,000 | No | Proprietary |
| Zyte | Per response, by tier | $0.06 to $16.08 per 1,000 | Scrapy only | Proprietary |
Choosing Once Instead of Twice
Sort your real domain list before you look at any pricing page. Split it into static HTML, JavaScript-rendered, and CAPTCHA or login-walled. That distribution moves your bill far more than raw page count does, and it is the input every vendor's rate card quietly depends on.
Then decide who owns the anti-bot problem. Renting it from Firecrawl, Bright Data or Zyte buys you a bill that scales with target difficulty. Owning it with Crawl4AI buys a flat server cost plus a proxy budget that arrives unannounced the first time you get blocked.
Check the license before you modify anything you plan to redistribute: AGPL-3.0 on Firecrawl's core obliges you to publish modifications offered as a network service, while Apache-2.0 and BSD do not. Then test your single hardest target, not the vendor demo. Advertised entry pricing describes the easiest tier, and easy sites are not why you are shopping. The full DevToolLab guide works through each of these steps with the per-tier numbers.
The Short Answer
Building a RAG pipeline or agent and want Markdown without plumbing: Firecrawl. Predictable credits, JS rendering at base rate, and the deterministic JSON caching pays for itself on recrawls.
Thousands of sites at unpredictable volume: Apify if you will tune actor code, Bright Data if you would rather pay flat per record and use its prebuilt scrapers.
Data cannot leave your infrastructure, or there is no budget: Crawl4AI, accepting that you are now the proxy team.
No commitment and tolerance for variance: Zyte, whose $5 trial is the cheapest way to learn what your specific sites actually cost.
Already running Scrapy: Scrapy Cloud, same maintainers, no rewrite.
Whichever you pick, the raw HTML arriving at your ingestion step is rarely what you want in a context window. HTML Cleaner strips inline styles and markup noise before it burns tokens, and JSON to CSV Converter turns a scrape result into a spreadsheet when you just need to eyeball what came back.
Conclusion
Firecrawl's star count and Apify's survey point the same way: this category grew because AI pipelines need it, not because scraping became easier. Apify killing rental pricing in October 2026 and Firecrawl shipping an arXiv index rather than more output formats both read like a market still deciding what it sells. Before you renew anything, answer one question. Are you billed per page or per unit of compute, and does that match the axis your workload grows along?
References
- Best Web Scraping APIs for AI in 2026: Firecrawl vs Apify vs Bright Data vs Zyte - the original, with the full pricing script and per-vendor detail
- Apify State of Web Scraping Report 2026 - January 29, 2026
- Firecrawl pricing
- Bright Data Web Scraper API
- Zyte pricing
- Crawl4AI on GitHub
- Best RAG Platforms and Tools - where these pages end up indexed
- Best AI Browser Automation Tools - the layer above extraction
Top comments (0)