Every "we hit a limit" story in an AI API account has the same shape: a ceiling that is generous for chat and hostile for agents. The difference is not traffic volume. It is the shape of the workload.
Chat is short and bursty, agents are long and recursive
A chat turn sends a prompt and gets an answer. The context is whatever the user typed. Token consumption is bounded by the length of the conversation, and the conversation is usually short.
An agent loop is different in a way that matters for budgeting. Every step re-sends the accumulated context. A fifty-step task re-reads the same files, the same tool outputs, the same instructions fifty times. Token consumption grows roughly with the square of task length in the worst case, because each step carries everything before it.
That is the mechanism behind the ceiling surprise. The account looked fine all month on chat traffic, then one long-running coding agent drained the quota in an afternoon.
What a ceiling actually does to an agent
The failure is not "request rejected." The failure is "session degrades."
When a token-metered API starts refusing calls mid-task, the agent does not roll back cleanly. It loses the step it was on. Depending on the framework, it either retries the same oversized context (making the problem worse), truncates history and loses coherence, or dies and leaves a half-applied refactor on disk. All three are worse than the request never having started.
A rate limit that returns a clean 429 with Retry-After is survivable. A mid-stream drop is not.
Budgeting as a design decision, not a post-mortem
Under token metering, the cost of a task is only knowable after it finishes. You find out what the run cost by reading the bill. That makes budgeting a retrospective exercise, and it makes per-client attribution a spreadsheet reconstruction problem.
Under a per-request allowance, the cost is bounded by the number of steps you permit. That is a number you choose in advance. It converts an accounting problem into a configuration decision, which is where it belongs.
A checklist for evaluating a gateway against agent workloads
- Does it speak the [OI] schema, including streaming and tool calls, so your framework needs only a base-URL change?
- Are request limits published per plan, or only token limits? Token-only limits are the ones that ambush you.
- What happens exactly at the ceiling: clean 429 with
Retry-After, or a dropped stream? - Can you issue separate keys per client or per environment, so attribution is a query rather than a reconciliation?
- Is the limit enforced per key or per account? Per-account limits let one runaway agent starve every other job.
Practical mitigation, whichever pricing model you use
Cap the step count per task. Summarise context rather than re-sending it verbatim. Queue long jobs instead of firing them concurrently. Treat the ceiling as a design input rather than an operational surprise.
The setup notes for the gateway I use are in the APIClaw documentation — it is [OI]-compatible and priced per request rather than per token, which is what made the budgeting above a configuration decision instead of a monthly surprise.
Top comments (0)