A production trace for one user request might look like this:
RUN 7F21
Goal: Prepare a renewal quote
Model calls: 4
Database queries: 7
CRM calls: 3
Pricing API calls: 2
CRM response: 429
SDK retry: 1
Agent retry: 1
Final status: Unknown
The customer clicked one button.
The system performed eighteen operations, retried the same dependency at two different layers and no longer knew whether the CRM update had succeeded.
That is how rate limiting changes when an AI agent enters a SaaS product.
Traditional SaaS rate limiting controls how many requests a user or tenant can send. Agent rate limiting must also control how much work a single goal is allowed to create.
One Request Is No Longer One Unit of Work
A normal API route has a reasonably predictable cost.
POST /reports
It validates input, queries data, creates a report and returns a response.
An agent starts with an objective rather than a fixed execution path. It may search documents, ask the model to choose a tool, call an API, inspect the response and repeat the process.
Two users can submit the same request and create very different workloads.
One run may finish after three tool calls. Another may search five times, load more context after each result and repeatedly call a throttled service.
The user-level limit still sees one request.
The infrastructure sees a small distributed system.
A 429 Can Leave the Workflow Half Finished
In an ordinary request-response flow, 429 Too Many Requests usually means the requested operation was rejected.
That assumption becomes dangerous after an agent has already completed earlier steps.
Imagine an onboarding agent that:
- Creates a customer account
- Adds the customer to the CRM
- Generates an invoice
- Sends a welcome email
If the invoice API returns 429, the first two changes may already exist. Restarting the entire agent can create duplicate accounts or CRM entries.
The failure is no longer simply “rate limited.” It is partially completed.
This is why agent workflows need persistent step state:
CREATE_ACCOUNT completed
UPDATE_CRM completed
CREATE_INVOICE waiting
SEND_EMAIL not_started
Resume from the failed step. Do not ask the model to reconstruct progress from conversation history.
Rate-Limit the Run, Not Only the Endpoint
Each agent run should receive an execution budget before it starts.
from dataclasses import dataclass
from time import monotonic
@dataclass
class RunBudget:
model_calls: int = 6
tool_calls: int = 12
external_writes: int = 2
retries: int = 3
deadline: float = monotonic() + 45
def consume(self, resource: str, amount: int = 1):
if monotonic() > self.deadline:
raise RuntimeError("RUN_DEADLINE_EXCEEDED")
remaining = getattr(self, resource)
if remaining < amount:
raise RuntimeError(f"{resource.upper()}_BUDGET_EXCEEDED")
setattr(self, resource, remaining - amount)
This budget answers questions an endpoint limiter cannot:
- How many times may the model reconsider its plan?
- How many tools may one run call?
- How many external writes may it perform?
- How many retries are allowed across every layer?
- How long can the run remain active?
The agent can choose its next action. The surrounding application decides whether enough budget remains to execute it.
Retries Need One Owner
Agent systems often contain several automatic retry layers.
The model retries a tool. The orchestration framework retries the model. The API client retries a 429. The queue worker retries the entire job.
Two retries at each of four layers can turn one failed call into many attempts.
OpenAI’s rate-limit guidance specifically warns that nested retry loops can multiply requests. It recommends respecting Retry-After, applying exponential backoff with jitter and limiting both attempts and total retry time. Unsuccessful requests can also count toward the rate limit.
Anthropic separately measures requests, input tokens and output tokens. Its documentation notes that short bursts can trigger throttling even when the apparent minute-level rate looks safe.
Choose one layer to own retries. Every other layer should report the failure upward.
Backoff Does Not Prevent Duplicate Actions
Backoff controls when another attempt happens. It does not make that attempt safe.
Before retrying a write, assign it a stable idempotency key derived from the agent run and intended business action:
renewal-run-7F21:create-invoice:customer-1842
The same logical action must reuse the same key, even when the model generates slightly different wording.
Stripe’s documentation explains that repeating a request with the same idempotency key can return the original result instead of creating another object.
AWS also recommends combining capped retries with exponential backoff, jitter and idempotent operations.
Measure Work Amplification
Requests per minute are no longer enough.
Track an amplification ratio for every agent workflow:
(model calls + tool calls + retries) / user requests
Also record the ratio by tenant, workflow version, model and failure type.
A sudden increase can reveal a broken stopping condition, poor retrieval, a looping planner or a dependency repeatedly returning incomplete data.
This metric connects reliability, cost and abuse detection. A tenant sending only ten requests could still consume more capacity than another tenant sending one thousand predictable API calls.
The Limit Is Part of the Product
Agent limits should not exist only as invisible infrastructure settings.
When a run exhausts its budget, the product needs a defined outcome: pause, request approval, schedule continuation, return partial work or escalate to a human.
For teams working with a SaaS application development company, this is an important architecture question. Ask how the system limits complete agent runs, prevents duplicate side effects and allocates usage fairly between tenants—not only how many requests its API gateway allows.
A rate limit should not merely stop traffic.
It should stop one uncertain goal from becoming an unlimited amount of work.
What does your SaaS product limit today: users, endpoints, tenants or complete agent runs?
Top comments (1)
The run-level budget idea is a good one, and the persisted step state is what makes retries safe. One bug to watch in the RunBudget sketch: a dataclass default such as deadline: float = monotonic() + 45 is evaluated once, when the class is defined, so every run would share the same deadline and later runs would fail almost immediately. Use field(default_factory=lambda: monotonic() + 45) so each run gets its own clock. Also, Stripe's docs say idempotency keys can be pruned once they are at least 24 hours old, so a run that is paused for approval and resumed the next day should check the stored step state before it replays a write with the old key.