Meter Your AI Feature at the Route Boundary Before the Invoice Does It for You

#ai #typescript #tutorial #webdev

A user clicks Regenerate on your AI summary feature for the ninth time in a minute. Each click is a fresh provider call, a fresh prompt stuffed with the full document, and a fresh line item on your invoice. The feature works. The failure isn't in the model, the prompt, or the UI — it's at the route boundary, the first layer where you could have said no and didn't.

This post walks through the metering slice I now add before any AI feature gets a real budget: a per-request token envelope, a per-user daily cap, and a deterministic downgrade path when the cap is hit. It is deliberately boring infrastructure, and it has saved me from the exact class of surprise that trends on this site every few months.

Why the route boundary, and not the provider client

When I first wired metering into the provider client (the SDK wrapper), it failed in two ways:

Retries inside the SDK were invisible to my counter. A timeout-and-retry billed twice but recorded once.
Streaming responses finished after my bookkeeping ran, so long generations blew past the envelope before anyone noticed.

Moving the decision to the route boundary fixed both, because at that layer you control the whole request lifecycle: you can reject before any provider call happens, and you can finalize accounting after the stream closes.

The working slice (Express + TypeScript)

Three pieces: a budget service with one seam, middleware that enforces it, and a downgrade contract your provider layer already understands.

// budget.ts — the only place limits live
export interface BudgetDecision {
  allow: boolean;
  downgradeTo?: string;        // provider/model alias, resolved elsewhere
  reason?: 'OVER_DAILY_CAP' | 'OVER_REQUEST_ENVELOPE';
  remainingToday: number;
}

export interface BudgetStore {
  tokensUsedToday(userId: string): Promise<number>;
  recordUsage(userId: string, tokens: number): Promise<void>;
}

const DAILY_CAP = 50_000;          // tokens per user per day
const REQUEST_ENVELOPE = 8_000;    // max estimated tokens per request

export async function decide(
  store: BudgetStore,
  userId: string,
  estimatedPromptTokens: number
): Promise<BudgetDecision> {
  const used = await store.tokensUsedToday(userId);
  const remainingToday = Math.max(0, DAILY_CAP - used);

  if (estimatedPromptTokens > REQUEST_ENVELOPE) {
    // Don't reject outright — offer the cheap path.
    return { allow: true, downgradeTo: 'small', reason: 'OVER_REQUEST_ENVELOPE', remainingToday };
  }
  if (remainingToday <= 0) {
    return { allow: false, reason: 'OVER_DAILY_CAP', remainingToday };
  }
  return { allow: true, remainingToday };
}

The key design choice: decide returns a decision object, not an exception. Callers can render 429s, downgrade, or queue — policy stays in one file, behavior stays with the route.

// middleware.ts
export function budgetGate(store: BudgetStore, estimate: (req: Request) => number) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const decision = await decide(store, req.user.id, estimate(req));
    if (!decision.allow) {
      return res.status(429).json({
        error: 'budget_exceeded',
        retryAfter: secondsUntilMidnightUtc(),
        remainingToday: 0,
      });
    }
    res.locals.modelAlias = decision.downgradeTo ?? 'standard';
    res.locals.budget = decision;
    next();
  };
}

// finalize AFTER the stream closes — this is the part SDK-level metering misses
export function finalizeUsage(store: BudgetStore) {
  return (req: Request, res: Response) => {
    res.on('finish', () => {
      const used = res.locals.actualTokens; // set by your streaming handler
      if (typeof used === 'number') store.recordUsage(req.user.id, used);
    });
  };
}

Your provider layer consumes res.locals.modelAlias through the same contract it already uses for model selection — no new coupling, just a new source for the alias. (If your provider selection is scattered across call sites, fix that first; I wrote about putting providers behind one route contract previously.)

Prototyping this without spending anything

Budget logic is exactly the kind of code you want to exercise hard before pointing it at a paid model. You need a model endpoint you can hammer with concurrency tests and a server to run the gate on.

I prototyped this slice on MonkeyCode, which offers free model access and a free server option — enough to run the gate, a stub provider client, and the concurrency tests below without an invoice of any size. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier was a fine fit here precisely because the artifact under test is the middleware, not the model; any OpenAI-compatible endpoint works. If you already have a cheap local model or a provider sandbox key, use that instead — the code doesn't care.

The test plan that matters

Unit tests for decide are trivial. The failures live in the seams, so test the seams:

#	Scenario	Expected behavior
1	20 concurrent requests from one user near the cap	At most the cap is consumed; losers get deterministic 429 with `retryAfter`
2	Request estimated over envelope	200, but `modelAlias === 'small'`; verify via provider stub's received model
3	Stream aborted by client mid-generation	Usage recorded for tokens actually generated, not the estimate
4	Provider timeout + SDK retry	One logical request; usage recorded once, at finalize
5	`BudgetStore` down (kill Redis)	Fail closed (429) or fail open with logging — pick one, test that it happens

Scenario 1 is the one that bites. A naive read → decide → write sequence races under concurrency; ten parallel requests all read "49,900 used" and all pass. Fixes: make the cap check a single atomic operation (INCRBY against the daily key in Redis, then compare), or accept slight overshoot and treat the cap as soft with an alerting threshold below it. I use atomic increment and treat Postgres as the async audit log, not the gatekeeper.

Scenario 5 is a policy decision disguised as a bug. Failing closed protects your wallet and angers users; failing open does the reverse. For a free tier product I fail closed; for a paid feature I'd fail open with a page to on-call. Write it down either way.

Limitations, and who should skip this

Estimates are estimates. Prompt token counts before generation are approximations; the envelope needs headroom (I use 20%) or scenario 3 becomes a user-facing lie.
This doesn't cap cost, it caps tokens. If your small and standard models have 10× price differences, meter in dollars, not tokens — same shape, different unit.
Single-tenant internal tools don't need this. If three coworkers use the feature, a spending alert on the provider dashboard is the right amount of engineering.
If your AI calls happen off-request (queues, agents with tool loops), the route boundary is the wrong layer — meter at the job dispatcher instead, with the same decision-object pattern.

Checklist

[ ] Budget policy lives in exactly one module; routes receive decisions, not exceptions
[ ] Rejection response includes retryAfter and remaining quota — clients can be well-behaved
[ ] Usage finalized after stream close, not estimated before it
[ ] Cap check is atomic under concurrency (test scenario 1, not just the happy path)
[ ] Store-outage behavior (open vs. closed) is chosen, documented, and tested
[ ] Downgrade path exercised end-to-end through the real provider contract

Metering is a layer handoff like any other: HTTP boundary → budget → provider → accounting, and the interesting bugs live in the joints. Which handoff is least stable in your stack right now? I'm especially curious about concrete failure states — if you've got a status code or an error shape that only appears under concurrency, I want to hear about it in the comments.