Originally published at AgentStack Notes. Last fact-checked: November 10, 2026.
Disclosure: This article was drafted with AI assistance and checked against each vendor's official documentation. We may receive a commission if you sign up through the Bright Data †, ScraperAPI † or Firecrawl † links marked † in this article, at no extra cost to you. We have not run production workloads on any of the three. See our Advertising Disclosure.
Your agent decides to check forty URLs and issues forty tool calls at the same moment. One service answers with an error, another quietly queues the extra work, a third says it throttles on its side. Which behaviour you get depends on the vendor, and the docs describe it unevenly. This is a side-by-side of what three vendors publish about requests in flight, the 429 response and timeouts.
Note on experience: we have not load-tested any of these services. Pages were read through a web-fetch tool that summarises them, on November 10, 2026. Check the live docs before relying on any number here. † marks a referral link; "not stated" means the page we read did not say.
Side by side
| Question | ScraperAPI † | Firecrawl † | Bright Data † |
|---|---|---|---|
| What limits you | Concurrent threads, set by plan | Requests per minute per endpoint, plus concurrent browsers, both by plan | Not stated as numbers on the pages we read |
| Published numbers | Free 5; Hobby 20; Startup 50; Business 100; "Scaling" 200 concurrent threads (pricing page) | /scrape per minute: Free 10, Hobby 100, Standard 500, Growth 5,000, Scale 10,000. Concurrent browsers: Free 2, Hobby 5, Standard 25, Growth 50, Scale 100+ | None found |
| When you exceed it | HTTP 429: "You are sending too many simultaneous requests." | Rate limit: HTTP 429. Concurrency: extra jobs wait in a queue. Queue full: 429 | HTTP 429 means auto-throttling, described as rare |
| Retry guidance | Match concurrency to your plan's limit and try again | Retry-After documented for some endpoints only; general retry timing not stated on this page | Contact support on a 429 |
| Timeout advice | Set client timeout to 70 seconds | Queue time counts against the request's timeout | Not stated for Web Unlocker on the pages we read |
ScraperAPI: the limit is requests in flight
The limit is on simultaneous requests, not on requests per minute. The pricing page lists concurrent threads per plan, from 5 on Free to 200 on the plan it calls "Scaling". The status-code page says a 429 means "You are sending too many simultaneous requests" and tells you to keep your concurrency within your plan's limit. Retries happen on ScraperAPI's side for up to 70 seconds, so the docs say to set your own timeout to 70 seconds; requests that fail after 70 seconds of retrying are not charged.
Who it suits, per the docs: a client that can cap its own parallelism to a plan's thread count. Where the docs stop: we found no requests-per-second figure and did not confirm whether a 429 itself costs credits. ScraperAPI link †
Firecrawl: two controls, and a queue
Requests per minute are capped per endpoint and per plan, and rate limits are applied per team, so all API keys on a team share the same counters. Separately, concurrent browsers are capped per plan. Exceeding a rate limit returns HTTP 429. Exceeding the concurrency limit does not error: jobs beyond it wait in a queue until a browser frees up. Queue time counts against the request's timeout, and queued jobs time out after a maximum of 48 hours. Max queued jobs are 50,000 on Free, Hobby and Standard, 100,000 on Growth and 200,000+ on Scale and Enterprise; when the queue is full, new jobs are rejected with a 429.
Who it suits, per the docs: bursts where waiting in a queue is acceptable. Where it needs care: because wait time counts toward the timeout, a long queue can turn into a timeout error. A Retry-After header is documented only for the government and research indexes; general retry timing is not stated on the page we read. Firecrawl link †
Bright Data: no numbers published
The Web Unlocker introduction says it retries failed attempts with alternative configurations until the request succeeds, and states no rate limit, concurrency cap or request timeout. The features page describes a 429 as "a rate limit (rare)" caused by auto-throttling, and says to contact support. We did not find limits for the Web MCP free tier on the pages we read.
Who it suits, per the docs: nobody can plan a concurrency budget from published figures here, so it fits only if you will measure it yourself. Bright Data link †
What to do with this
- Put your own cap below the plan's. An agent that spawns parallel tool calls can exceed 5 ScraperAPI threads or 2 Firecrawl browsers instantly. Use a semaphore in your client sized to the plan.
- Treat the three 429s differently. ScraperAPI's means too many in flight, so reduce parallelism. Firecrawl's can mean a minute-bucket limit or a full queue. Bright Data's means throttling on its side, where more parallelism will not help.
- Mind timeouts under queueing. On Firecrawl, waiting counts toward your timeout. On ScraperAPI, a client timeout under 70 seconds can cut off retries.
- Check billing on throttled calls before you scale. The pages we read do not say whether a 429 consumes credits. Send a small burst and compare your dashboard to your logs.
What we could not confirm
- ScraperAPI's requests-per-second behaviour, and whether 429s are billed.
- Any numeric rate or concurrency limit for Bright Data's Web Unlocker or Web MCP.
- Firecrawl's general retry-timing guidance for ordinary 429s.
- Real-world behaviour under load, success rates and speed of any of the three.
Sources (checked November 10, 2026): ScraperAPI pricing, ScraperAPI API Status Codes, Firecrawl rate limits, Bright Data Web Unlocker introduction, Bright Data Web Unlocker features.
Top comments (1)
Good comparison. Hitting 429 rate limits under high concurrency is always a headache managing proper backoff logic and request distribution saves a ton of failing requests.