DEV Community

Daniel Pertu
Daniel Pertu

Posted on

25 rate limiters, and the only question that matters is what happens when the limiter itself is down

Munchable's backend is 29 API routes behind one Expo app. Scanning a barcode, looking a product up by name, reading a label photo, contributing a label, opening a support ticket: all of them are endpoints a signed-in phone calls, and all of them need a ceiling.

The file that holds the ceilings builds 25 limiters. Most endpoints get two, a few get three, and the shape is always the same: one keyed on the account, one keyed on the IP.

productDev: mk(Ratelimit.slidingWindow(60, '1 m'), 'rl:product:dev'),
productIp:  mk(Ratelimit.slidingWindow(300, '1 m'), 'rl:product:ip'),
Enter fullscreen mode Exit fullscreen mode

The account limit is the real one. The IP limit exists because signing up is free, so an account limit on its own is a limit on patience, not on abuse. One per account stops a loop in the app; one per IP stops someone cycling accounts to get around it.

None of that is the interesting part. The interesting part is the four lines of catch that every one of these routes has to write, and the fact that they are not all the same.

The same code, two opposite answers

Every route does this:

let decision: LimitDecision;
try {
  const lim = limiters();
  decision = await enforce([
    { limiter: lim.searchDev, key: user.id },
    { limiter: lim.searchIp,  key: getClientIdentifier(request) },
  ]);
} catch {
  decision = { success: true };   // or: return 503
}
Enter fullscreen mode Exit fullscreen mode

Both branches are defensible and they are opposites. decision = { success: true } says a request the limiter could not judge is served. return 503 says a request the limiter could not judge is refused.

In this codebase, four routes take the first branch and five take the second, and the sorting rule turned out to be simple once I wrote it down: what does this route lose if it runs unchecked?

Fail open, serve the request:

  • product lookup, the scan path,
  • product search by name,
  • the ingredient vocabulary download,
  • reading your own contribution standing.

Every one of those is a read of data Munchable already publishes, keyed on nothing private, costing a cheap indexed query. An unchecked burst of them costs some database time. A refused one breaks a scan in a shop, in front of a shelf, for somebody who is not going to try again. There is no version of that trade where 503 is the right answer.

Fail closed, 503 with rate_limiter_unavailable:

  • reading a label photo,
  • contributing a label,
  • reporting a product as wrong,
  • opening a support ticket,
  • resolving words the device could not place.

Those write rows, send email, or spend money with a model provider. Running them unmetered because a Redis call timed out means the failure of a guard becomes an unbounded bill or an unbounded queue. A 503 there is honest: the app shows a retry, and nothing has happened.

The shorthand I ended up with: reads fail open, spends fail closed. If you only take one thing from this post, that is the one, and it generalises well past rate limiting.

The budget that is not a rate limit

Two routes can end up calling a model inline. They share one daily budget, and it is a cost circuit breaker rather than a product limit:

const INLINE_DAILY_CAP = 3000;        // everyone, per day
const INLINE_USER_DAILY_CAP = 40;     // one account, per day
const INLINE_JOB_DAILY_CAP = 2000;    // one background job, per day
Enter fullscreen mode Exit fullscreen mode

Three things in there were learned the hard way.

The order of the two checks. The per-principal counter is incremented and checked before the global one. Check global first and a caller already over its own limit still increments the shared counter on every rejected call, which is a cheap way for one bad client to drain the day's budget for everybody while being rejected the whole time.

A job is not an account. The per-capture resolver originally spent the budget through the user key, so every capture in the system shared one 40-a-day allowance and exhausted it long before the global cap was touched. Raising the account cap to fix that would have loosened the abuse ceiling for real accounts at the same time. Two caps, because they protect two different things.

The budget fails closed. No Redis, no model call. A read that cannot be metered can be served; a spend that cannot be metered cannot.

There was also an earlier bug worth naming, because it is the kind that reads as fine in review: each route held its own copy of the cap as a literal, while both incremented the same Redis counter. Raising one route's cap left the other rejecting at the old number, and nothing about either file looked wrong. Now the caps and the reserve function live in one module, because two routes sharing a counter is one decision and not two.

Two enforcement functions that look identical

enforce runs its checks one after another and returns the first failure. Four of these routes chain a per-minute limiter in front of a per-day limiter, and the serial loop is load-bearing: a request already rejected by the minute check must never increment the day's counter. Otherwise a client in a tight retry loop burns its daily allowance while receiving nothing but 429s.

enforceParallel issues every check at once. It exists because the scan path's two checks are independent and fail open together, so paying two sequential Redis round trips in the latency path of the one thing in this app that has to feel instant was waste.

They have the same signature and the same return contract. Swapping one for the other silently changes behaviour on any route that relies on short circuiting, which is why the comment above the parallel one is six lines long and ends with "do not repoint an existing caller at this function without checking". Sometimes the only available guard is a sentence.

Two small details that cost nothing and matter

Never key on the left-most forwarded IP.

const realIp = request.headers.get('x-real-ip');
if (realIp?.trim()) return realIp.trim();
Enter fullscreen mode Exit fullscreen mode

Falling back to the last entry of x-forwarded-for, never the first. The left-most token is whatever the client sent, so keying on it means an attacker picks their own rate-limit bucket per request and the IP limit becomes decoration.

Every limiter is constructed with analytics off.

new Ratelimit({ redis, limiter, prefix, analytics: false })
Enter fullscreen mode Exit fullscreen mode

Munchable does not send request events to a third party, which is the posture described at munchable.app/privacy: no third-party analytics, no tracking. A rate limiter is an easy place to leak that promise by accident, because the dashboard is a genuinely useful default. One flag, and the counters stay counters.

One route also deserves mention for having a different shape: opening a support ticket works without an account, so there is no account to key on. The IP carries the whole budget and the daily ceiling is shared by everyone behind it. That is worse for somebody on a shared network, and it is still better than requiring a login before you can tell me something is broken.

What the client sees

Every limited response carries RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset, and a rejection answers with a scope name, a retryAfter and a matching Retry-After header:

{ "error": "rate_limited", "scope": "support", "retryAfter": 34 }
Enter fullscreen mode Exit fullscreen mode

The scope is there so the app can say something true. "You have reached today's limit for opening tickets" and "too many requests" are different sentences, and only one of them tells somebody what to do next.

The product all of this sits under is at munchable.app: scan a barcode, get a straight answer about whether that product suits your gut. The conditions it covers are at munchable.app/conditions. None of those pages will ever show you a 429, which is sort of the point of the ceilings being on the routes the app calls rather than on the pages people read.

Top comments (0)