<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jack Ma</title>
    <description>The latest articles on DEV Community by Jack Ma (@jackmally).</description>
    <link>https://dev.to/jackmally</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4252457%2F281a7b0a-8421-4524-bc5c-9d070611645e.png</url>
      <title>DEV Community: Jack Ma</title>
      <link>https://dev.to/jackmally</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jackmally"/>
    <language>en</language>
    <item>
      <title>Per-customer API limits in six stacks: where the check has to run</title>
      <dc:creator>Jack Ma</dc:creator>
      <pubDate>Wed, 18 Nov 2026 19:04:26 +0000</pubDate>
      <link>https://dev.to/jackmally/per-customer-api-limits-in-six-stacks-where-the-check-has-to-run-537i</link>
      <guid>https://dev.to/jackmally/per-customer-api-limits-in-six-stacks-where-the-check-has-to-run-537i</guid>
      <description>&lt;p&gt;Every per-customer limit I've wired up comes down to the same five steps: work out which customer is calling, check their rate limit, reserve part of their quota, run the handler, then count what the request actually used. Those steps don't change between frameworks. What changes is where the framework lets you run code before and after a handler, what it tells you about the route that matched, and whether anything still runs once the response has gone out.&lt;/p&gt;

&lt;p&gt;This post skips the rate limiting algorithm itself and looks at where the check ends up in each framework and what caught us out. The examples use the UsageRail SDKs, which I work on. Most of the constraints below come from the frameworks and would hit your own middleware too; a few, such as the synchronous &lt;code&gt;consumer&lt;/code&gt; in Next.js, come from our SDK.&lt;/p&gt;

&lt;p&gt;Three questions decide the placement:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Does the check run before or after your own authentication?&lt;/li&gt;
&lt;li&gt;Which route can you record: &lt;code&gt;/v1/items/:id&lt;/code&gt;, or &lt;code&gt;/v1/items/42&lt;/code&gt;?&lt;/li&gt;
&lt;li&gt;Where does the work after the response run: a timer, a shutdown hook, or something like &lt;code&gt;waitUntil&lt;/code&gt;?&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Express: order is coverage
&lt;/h2&gt;

&lt;p&gt;Express middleware applies to routes registered after it, and that is most of the placement rule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;express&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"express"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;usageRail&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"@usagerail/node/express"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// import before your routers&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;express&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/health"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// registered first, never checked&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;use&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/v1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;usageRail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;appKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;USAGERAIL_APP_KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;auth&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"api-key"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// Authorization: Bearer &amp;lt;key&amp;gt;, or x-api-key&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Import order matters for a different reason. Express doesn't keep a router's mount path as you wrote it, so the SDK records mount paths as routers are mounted, which only works for routers mounted after the import. A router mounted before the import is still checked as long as its requests pass through the UsageRail middleware, but its usage is grouped by the concrete path, so &lt;code&gt;/tenants/42/reports&lt;/code&gt; becomes its own row for every tenant. If you identify callers yourself, put your auth middleware first, set &lt;code&gt;auth: "custom"&lt;/code&gt; and pass a &lt;code&gt;consumer&lt;/code&gt; function that reads &lt;code&gt;req.user&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;On the client side, both refusals are a 429. On a rate-limit 429, &lt;code&gt;Retry-After&lt;/code&gt; is the seconds until the customer's bucket has a token again. On a quota 429, it's the seconds left in the billing period, which can be days. A retry helper that sleeps on that header can park a job for 12 days, so clients should retry short waits and surface long ones.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hono: reserve before the handler runs
&lt;/h2&gt;

&lt;p&gt;Middleware registered with &lt;code&gt;app.use("/v1/*", ...)&lt;/code&gt; runs before it knows which handler will answer. That matters when the quota is counted in units, because the reservation has to happen before the handler runs. The SDK matches the raw path against the route templates you've configured and takes the largest reservation among the templates that match. After the handler returns, it reads the route that actually ran with &lt;code&gt;routePath(c)&lt;/code&gt; and records the usage there.&lt;/p&gt;

&lt;p&gt;The other Hono-specific thing: the middleware doesn't touch the process, because Hono also runs where there isn't one. On Node.js you add two things yourself: &lt;code&gt;flushOnExit(rail.engine)&lt;/code&gt;, which sends pending counts at shutdown (without it they're lost), and &lt;code&gt;fileSnapshotStore&lt;/code&gt;, which saves the key list so a restarted process can check keys before its first sync.&lt;/p&gt;

&lt;p&gt;Streaming takes more care. In &lt;a href="https://github.com/orgs/honojs/discussions/5600" rel="noopener noreferrer"&gt;this Hono example that meters streamed tokens per customer&lt;/a&gt;, an SSE response counts once when the stream opens, and the units from &lt;code&gt;track()&lt;/code&gt; are added when it ends or the client disconnects. Pass &lt;code&gt;track()&lt;/code&gt; the running total after each chunk: it replaces the previous value, so calling &lt;code&gt;track(1)&lt;/code&gt; per chunk records the whole stream as 1.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next.js route handlers: wrap each one
&lt;/h2&gt;

&lt;p&gt;App Router route handlers have no &lt;code&gt;app.use&lt;/code&gt;, and &lt;code&gt;middleware.ts&lt;/code&gt; runs before the handler without ever seeing the status it returns. So the check wraps the handler itself, and you pass the route template, because a route handler isn't told which file it came from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app/v1/items/[id]/route.js&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;rail&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"@/lib/usagerail"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// usageRail({ ..., waitUntil: after })&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="no"&gt;GET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/v1/items/[id]"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;loadItem&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;track&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// optional: count units instead of requests&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;middleware.ts&lt;/code&gt; still has a job if you use your own auth. The &lt;code&gt;consumer&lt;/code&gt; function has to return synchronously, so a database lookup goes in middleware, which passes the account ID on in a request header. Make the middleware overwrite that header on every request, or a caller can send their own. The work after the response moves into &lt;code&gt;after()&lt;/code&gt; from &lt;code&gt;next/server&lt;/code&gt;, since a frozen instance may never fire a timer. If the platform stops an instance before that work finishes, the unsent counts are gone. In &lt;a href="https://github.com/vercel/next.js/discussions/101024" rel="noopener noreferrer"&gt;our October 2026 tests of &lt;code&gt;after()&lt;/code&gt; on Vercel&lt;/a&gt;, every successful response was counted, including a burst of 70 requests at once that started 70 new instances.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cloudflare Workers: set up on the first request
&lt;/h2&gt;

&lt;p&gt;On Workers the code looks like the Hono version with &lt;code&gt;serverless: true&lt;/code&gt;, plus &lt;code&gt;nodejs_compat&lt;/code&gt; in the Wrangler config, because the SDK uses &lt;code&gt;node:crypto&lt;/code&gt;. Two platform rules forced real changes. Workers refuse to generate random values while the module is loading, and the SDK creates an instance ID when it starts, so with &lt;code&gt;serverless: true&lt;/code&gt; it sets itself up on the first request instead. Leave &lt;code&gt;serverless: true&lt;/code&gt; out and the Worker fails to start. And there's no process to hook, so reporting, key-list refreshes and quota lease renewals go into &lt;code&gt;c.executionCtx.waitUntil&lt;/code&gt; after each response, at most one batch every 0.5 seconds per isolate.&lt;/p&gt;

&lt;p&gt;When we &lt;a href="https://github.com/cloudflare/workers-sdk/discussions/16652" rel="noopener noreferrer"&gt;moved per-customer keys and quotas from Node.js into a Worker&lt;/a&gt;, revocation changed too. With no background sync, a request that finds the key list more than 20 seconds old tries to refresh it first, waiting up to 1 second, so a revoked key normally stops working within about 20 seconds, even on an isolate that sat idle. If UsageRail can't be reached, the isolate keeps using the list it already has, and a list more than 24 hours old isn't trusted.&lt;/p&gt;

&lt;h2&gt;
  
  
  Django REST framework: in front of DRF, not inside it
&lt;/h2&gt;

&lt;p&gt;DRF's throttles handle "N requests per minute per user" well. We needed limits that come from each customer's plan plus a quota per billing period, and putting those quota checks in a plain Django middleware in front of DRF means they run before DRF does. With &lt;code&gt;SessionAuthentication&lt;/code&gt; that's fine, because Django's &lt;code&gt;AuthenticationMiddleware&lt;/code&gt; has already set &lt;code&gt;request.user&lt;/code&gt;. With &lt;code&gt;TokenAuthentication&lt;/code&gt;, a &lt;code&gt;consumer&lt;/code&gt; that reads &lt;code&gt;request.user&lt;/code&gt; still finds an anonymous user when the middleware runs, so those requests are recorded as anonymous: not rate limited, not counted against a quota and never billed. The fix is to call the auth class from the consumer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# myapp/usage.py, set as USAGERAIL["consumer"] in settings.py&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;rest_framework&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;authentication&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;TokenAuthentication&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;rest_framework&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exceptions&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;AuthenticationFailed&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;consumer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;TokenAuthentication&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;authenticate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;AuthenticationFailed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That costs a second token lookup per request. Running first has two other effects. Refused requests never reach DRF's exception handler or renderers, so a custom error format has to be set in &lt;code&gt;on_reject&lt;/code&gt;. And router URLs are recorded as the regular expressions DRF builds, which route weights have to match exactly.&lt;/p&gt;

&lt;h2&gt;
  
  
  LiteLLM: count where you know the customer
&lt;/h2&gt;

&lt;p&gt;LiteLLM isn't a web framework, but many LLM APIs get their token counts from it. If you sell an API and call models through the LiteLLM Python SDK, LiteLLM knows the tokens and has no idea which of your customers the call was for. To count LiteLLM token usage per customer, keep the check in your API layer. Here a FastAPI middleware checks the limits and the endpoint passes in the token count:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Inside the FastAPI endpoint, after the model call&lt;/span&gt;
&lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;litellm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;acompletion&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# Prompt plus completion tokens, as reported by the provider&lt;/span&gt;
&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usagerail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;track&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_tokens&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The middleware decides whether to let the request in before the model runs, so each route needs a reservation sized for the most tokens one API request can use. For a single model call that's your &lt;code&gt;max_tokens&lt;/code&gt; plus the largest prompt you accept. If one request makes several model calls, reserve for all of them, and add the actual tokens up before calling &lt;code&gt;track()&lt;/code&gt;, because it replaces the value instead of adding to it. If you also run the LiteLLM proxy, its virtual keys can carry their own budgets and rate limits, and the two can run side by side.&lt;/p&gt;

&lt;h2&gt;
  
  
  What doesn't move
&lt;/h2&gt;

&lt;p&gt;A few things were the same in all six:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rate limit buckets live in memory&lt;/strong&gt;, per process, per instance or per isolate. On Vercel or Workers that means a customer can get the full limit on every instance unless you split it across them. Our page on &lt;a href="https://usagerail.com/api-rate-limiting" rel="noopener noreferrer"&gt;per-customer API rate limiting&lt;/a&gt; covers the token bucket and the response headers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Refused requests and 4xx or 5xx responses aren't billed.&lt;/strong&gt; They're still recorded, and they don't use up quota.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What happens at the limit is a pricing decision.&lt;/strong&gt; A plan that stops at the quota answers &lt;code&gt;429 quota_exceeded&lt;/code&gt; with &lt;code&gt;Retry-After&lt;/code&gt; set to the seconds left in the period. Prepaid credits answer &lt;code&gt;402 insufficient_credits&lt;/code&gt; once the balance is gone. A plan that bills overage keeps serving past the included units and puts the extra on the customer's invoice; rate limits still apply.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your stack puts this check somewhere else, I'd like to hear where it ended up and what broke first.&lt;/p&gt;




&lt;p&gt;I work on UsageRail. The examples use its Node.js (&lt;code&gt;@usagerail/node&lt;/code&gt;) and Python (&lt;code&gt;usagerail&lt;/code&gt;) SDKs.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I used AI to help draft this post.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>api</category>
      <category>webdev</category>
      <category>node</category>
      <category>python</category>
    </item>
  </channel>
</rss>
