Rate Limiting AI Agent Endpoints in Node When One Request Costs €2

#ai #typescript #agents #node

Book: AI That Ships
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Standard rate limiting counts requests. That works when requests are roughly
interchangeable — 100 per minute is 100 units of load.

An agent endpoint is not like that. One request might be a two-turn answer
costing a fraction of a cent. Another might be a twenty-turn run reading six
documents and costing two euros. Same route, same method, two orders of
magnitude apart.

Limit on requests and you either throttle the cheap ones pointlessly or let a
handful of expensive ones spend your monthly budget before lunch.

Limit on the three things that actually run out

Money. Your provider spend. This is the one that ends careers.

Concurrency. Your provider's parallel-request limit and your own worker
pool. Exceeding it produces 429s from upstream, which arrive as latency.

Requests. Still worth having as a crude abuse guard, just not as the
primary control.

export type Limits = {
  costUsdPerHour: number;
  costUsdPerDay: number;
  concurrent: number;
  requestsPerMinute: number;
};

export const LIMITS: Record<Plan, Limits> = {
  free:  { costUsdPerHour: 0.50, costUsdPerDay: 2,  concurrent: 1, requestsPerMinute: 5 },
  pro:   { costUsdPerHour: 5,    costUsdPerDay: 40, concurrent: 3, requestsPerMinute: 60 },
  team:  { costUsdPerHour: 40,   costUsdPerDay: 300, concurrent: 10, requestsPerMinute: 300 },
};

Hourly and daily. Hourly stops a runaway loop inside an hour; daily stops a
sustained drip that never trips the hourly limit.

The problem: you do not know the cost until afterwards

Every other rate limiter decrements before the work. Here the cost is unknown
until the run finishes.

Reserve an estimate, then reconcile:

export async function reserve(userId: string, estUsd: number) {
  const key = `cost:${userId}:${hourBucket()}`;
  const used = await redis.incrByFloat(key, estUsd);
  await redis.expire(key, 7200);

  if (used > LIMITS[planOf(userId)].costUsdPerHour) {
    await redis.incrByFloat(key, -estUsd);          // give it back
    throw new RateLimited("cost", await ttl(key));
  }
  return { key, estUsd };
}

export async function settle(r: Reservation, actualUsd: number) {
  await redis.incrByFloat(r.key, actualUsd - r.estUsd);   // correct the difference
}

incrByFloat is atomic, so concurrent requests cannot both pass a check that
only one should. The estimate should be pessimistic — the median run cost
times about two, because under-estimating lets a burst through before any of
them settle.

Settle in a finally, including on failure. A run that burned a euro and then
timed out still burned a euro:

const r = await reserve(user.id, estimate(intent));
try {
  const out = await runAgent(input, ctx);
  await settle(r, out.costUsd);
  return out;
} catch (err) {
  await settle(r, ctx.spentSoFar());          // partial spend still counts
  throw err;
}

Concurrency needs a slot, not a counter

const SLOT_TTL = 300;                          // longer than any run

export async function acquire(userId: string): Promise<Slot> {
  const limit = LIMITS[planOf(userId)].concurrent;
  const id = crypto.randomUUID();
  const key = `slots:${userId}`;

  const n = await redis.zcard(key);
  if (n >= limit) throw new RateLimited("concurrency");

  await redis.zadd(key, Date.now() + SLOT_TTL * 1000, id);
  await redis.expire(key, SLOT_TTL * 2);
  return { key, id };
}

export async function release(s: Slot) {
  await redis.zrem(s.key, s.id);
}

A sorted set scored by expiry rather than a plain counter, because a process
killed mid-run never decrements a counter, and after a few crashes the user is
permanently at their limit. Sweep expired members before counting:

await redis.zremrangebyscore(key, 0, Date.now());

Self-healing, which a counter is not.

Queue instead of rejecting

A 429 on an endpoint that takes 30 seconds anyway is a bad trade. If the
client is going to wait, let it wait in a queue where you control the order.

const job = await queue.add("agent-run", { userId, input }, {
  priority: user.plan === "free" ? 10 : 1,
  jobId: `${userId}:${hash(input)}`,          // dedupe identical retries
});

return res.status(202).json({
  runId: job.id,
  statusUrl: `/runs/${job.id}`,
});

202 with a poll URL turns rate limiting from a failure into a wait. And
jobId deduping identical payloads absorbs the double-submit that would
otherwise cost twice.

Workers enforce global concurrency, which is what protects your provider limit
regardless of how many users are active:

new Worker("agent-run", handler, { concurrency: 8 });

Tell the client what happened

res.set({
  "RateLimit-Policy": `${limits.costUsdPerHour};w=3600;unit=usd`,
  "RateLimit-Remaining": remaining.toFixed(3),
  "RateLimit-Reset": String(resetSeconds),
  "Retry-After": String(resetSeconds),
});
return res.status(429).json({
  error: "cost_limit_exceeded",
  message: `Hourly budget of $${limits.costUsdPerHour} reached. Resets in ${mins} minutes.`,
  upgradeUrl: "/billing",
});

A machine-readable error code and a human-readable message. Clients branch
on the code; the message goes in the UI. Retry-After is what stops a client
hammering a limit it has already hit.

Two limits people forget

Per-run cost. The limits above are per user over time. A single run also
needs a ceiling, or one pathological loop consumes an entire hourly budget in
ninety seconds:

if (ctx.budget.spent > flag.maxCostUsdPerRun) throw new BudgetExceeded();

Global. Sum across all users, as the last line of defence against a bug in
your own limiter:

const global = await redis.incrByFloat(`cost:global:${hourBucket()}`, estUsd);
if (global > GLOBAL_HOURLY_USD) {
  metrics.increment("agent.global_limit_hit");
  throw new ServiceBusy();
}

Set it around three times your normal peak. It should never fire, and the day
it does, it will be the cheapest alert you ever wrote.

Watch what the limiter is doing

metrics.increment("ratelimit.block", 1, { reason, plan });
metrics.histogram("ratelimit.est_error", actualUsd - estUsd, { intent });
metrics.gauge("ratelimit.util", used / limits.costUsdPerHour, { plan });

est_error is the one to check first after shipping. A consistently negative
skew means you are over-reserving and blocking users who had budget left; a
positive skew means bursts slip through before settlement.

And blocks by reason: if concurrency dominates, add workers. If cost
dominates on the paid plan, your limits are a pricing conversation rather than
an engineering one.

If this was useful

AI That Ships covers the operational
side of AI features — cost-based limits, queueing, per-run budgets, and the
metrics that tell you whether your ceilings are protecting you or just
annoying your users.