DEV Community

AgentStack Notes
AgentStack Notes

Posted on Originally published at dapper-shortbread-aad01d.netlify.app AI-assisted

Retry Budgets for AI Agents: What ScraperAPI, Firecrawl and Bright Data Document About 429s, Retries and Charges

Originally published at AgentStack Notes. Last fact-checked: November 15, 2026.

Disclosure: This article was drafted with AI assistance and checked against each vendor's official documentation. We may receive a commission if you sign up through the Bright Data †, ScraperAPI † or Firecrawl † links marked † in this article, at no extra cost to you. We have not run production workloads on any of the three. See our Advertising Disclosure.

An agent gets a 429 from a scraping API, waits a moment and tries again. Then again. Nothing in the loop says when to stop, and two of the three vendors below already retry on their own side before you ever see an error. Here is what each vendor's docs say about 429s, retries and charges, and a simple cap you can put in your own code.

Note on experience: we have not load-tested these services. Pages were read through a web-fetch tool that summarises them, on November 15, 2026. Check the live docs before relying on any statement. † marks a referral link; "not stated" means the page we read did not say.

Why an agent needs a retry budget

A retry budget is a cap you set in your own code: at most N retries per task, and at most M per run, after which the agent stops and reports. Without one, an agent that keeps retrying a rate-limited call can spend credits or time without finishing. The vendors do not set this cap for you, and two of them already retry on their side, which changes how you should count.

What each vendor documents

Question ScraperAPI † Firecrawl † Bright Data †
What a 429 means "You are sending too many simultaneous requests." Rate limit exceeded, or the concurrency queue is full "Rate limit (rare)"; auto-throttling
Vendor-side retries Retries failed requests for up to 70 seconds, then returns 500 Not stated on the pages we read Not stated on the Web Unlocker features page
Retry guidance for you Keep concurrency within your plan; set your timeout to 70 seconds Retry-After documented only for the government/research indexes; the Rate Limits page points to an Errors page for a backoff snippet Open a ticket or email support
Is a 429 charged? Not stated. Charges apply to 200 and 404 responses Not stated Not stated; billing is described for successful requests

ScraperAPI: the vendor already retries for 70 seconds

The status-code page says ScraperAPI retries failed requests for up to 70 seconds before it responds with a 500, and tells you to set your timeout to 70 seconds. A 500 is documented as "Request failed. You will not be charged for this request." The page says you are charged only for 200 and 404 responses. A 429 is described as too many simultaneous requests, with advice to keep concurrency within your plan's limit; whether a 429 itself is charged is not stated.

What that means for a budget: one call from your agent can occupy up to 70 seconds before you see a failure. If your client retries immediately on a 500, you are stacking your retries on top of the vendor's. Count wall-clock time as part of the budget, not only attempts. For a 429, the documented fix is to lower concurrency, not to retry harder. ScraperAPI link †

Firecrawl: a lower timeout is the documented way to fail fast

Exceeding a rate limit returns a 429. Exceeding the concurrent-browser limit does not: jobs wait in a queue, and queue time counts against the request's timeout, so the docs say you can set a lower timeout to fail fast instead of waiting. Queued jobs expire after at most 48 hours, and when the team's queue is full, new jobs are rejected with a 429. The Retry-After header is documented for the government and research indexes; for ordinary endpoints, general retry timing is not stated on the page we read, which points to an Errors page for a retry-with-backoff snippet that we did not read. Whether a 429 or failed request consumes credits is not stated on the pages we read. A SCRAPE_TIMEOUT (408) error advises increasing the timeout parameter, up to 60000 ms.

What that means for a budget: a long timeout plus a retry loop can mean long waits in a queue. Set the timeout deliberately, and honor Retry-After where the docs give it. Firecrawl link †

Bright Data: contact support, billing depends on the feature

The Web Unlocker features page describes a 429 as a rate limit ("rare") from auto-throttling, for example "the request was auto-throttled due to low success rate", and says to contact support. It gives no numeric limit and no retry advice. On billing, the page says only successful Web Unlocker requests count toward billing, but for some custom features (the page names headers and cookies) it says you are billed for 100% of requests, successful and failed. Check which features your agent switches on before assuming failed calls are free. Bright Data link †

A budget template you can adapt (our suggestion, not vendor guidance)

These numbers are placeholders we made up for illustration; none come from a vendor or a test.

  • Per task: allow 2 retries, then stop and return the error to the agent.
  • Per run: cap total retries (for example 20) so one bad site cannot consume the run.
  • On a 429: reduce concurrency first, then retry; do not retry in a tight loop.
  • On 5xx from ScraperAPI: remember the vendor already retried for up to 70 seconds.
  • Log every retry with the vendor, status code and elapsed time, so you can see from your own data which calls burn budget.

What we could not confirm

  • Whether a 429 consumes credits at any of the three vendors.
  • Firecrawl's general backoff guidance for ordinary endpoints (the Errors page).
  • Any Bright Data retry guidance beyond "contact support".

Sources: ScraperAPI API status codes, Firecrawl docs, Bright Data Web Unlocker features. Checked November 15, 2026.

More: the original article, and the earlier pieces on burst limits and 429s and failed scrapes and your bill.

Top comments (0)