DEV Community

nRouter
nRouter

Posted on

Budgets vs rate limits: dollars over a window, velocity right now

Every gateway gives you two ceilings. Most teams set one and stop — and the incident that eventually finds them belongs to the other one.

A budget caps how much you spend over a window. A rate limit caps how fast you go right now. From the client they look identical (both are rejections, both can wear a 429), but they answer different questions, and neither degrades gracefully into the other.

The two blind spots

  • A monthly budget does not notice a burst. An agent loop that runs for twenty minutes spends less than a month's cap. The budget watches it pass in silence.
  • An RPM ceiling does not notice expense. Velocity is not priced. The same request rate aimed at a small model and a flagship model is the same graph and a wildly different invoice.

Classify before you configure

One question: if this goes wrong, does it hurt because of how much it costs, or because of how fast it happens?

Risk Control
Agent loop, retry storm Rate limit
Leaked key Both — RPM bounds damage per minute, budget bounds it in total
Expensive-but-steady feature Budget
Per-team allowance Budget
One tenant starving the others Rate limit
Batch job with a fixed total Budget, duration = Total (never resets)

Configuration details that are not details

Rate limits live on the key — RPM and TPM, sliding window, continuous replenishment. That makes key structure the real policy: one key per environment-and-service pair, or throttling the batch job throttles checkout. The same key can carry allowed models, allowed endpoints and an IP allowlist, evaluated before the key consumes budget, so denied requests cost zero credits.

Budgets carry scope and mode. Scope (organization / team / user / API key) decides which rejection your client sees: scoped budgets return 402 budget_exceeded; a single key's budget returns 429 key_budget_exceeded. Mode decides whether the budget is a control at all: Block rejects at the cap; Warn alerts and keeps serving (observability, not a ceiling); Throttle alerts and flags for rate reduction.

And a gateway budget is evaluated on the request, before the provider call — the only place a dollar ceiling can actually stop a dollar. Cost tooling that refreshes a few times a day cannot catch a loop.

The client rule

Three rejections, one retryable:

  • rate_limit_exceeded → transient. Back off, honour Retry-After, retry.
  • budget_exceeded / key_budget_exceeded → ceiling. Fails until a human raises the cap. Never retry.

Branch on the error code, not the status. The most expensive mistake in this area is not picking the wrong control — it is retrying a rejection that will never succeed.

Measure first (median day and worst day, per key), set both controls on purpose, and make each one reject something deliberately once, so the first time you see the rejection is not during the incident.

Full guide: https://nrouter.ai/blog/guides/budgets-vs-rate-limits

Top comments (0)