Five Serverless Instances, Five Counters, One Rate Limit That Didn't Exist

Parcelglance is a Shopify app that answers "where is my order" questions with an LLM. The endpoint that matters is /api/chat, and every request to it costs me money — tokens, not bandwidth. So it has a rate limit: 30 requests per minute per shop.

For a while that limit did nothing at all, and I couldn't tell, because the thing I had built to enforce it failed in the only direction that produces no error.

Version one: a Map, and a wrong mental model

The first implementation was the one everybody writes first:

const buckets = new Map();

function allow(shop, limit = 30, windowMs = 60_000) {
  const now = Date.now();
  const b = buckets.get(shop) ?? { count: 0, reset: now + windowMs };
  if (now > b.reset) { b.count = 0; b.reset = now + windowMs; }
  b.count++;
  buckets.set(shop, b);
  return b.count <= limit;
}

On a single server this is correct. On Vercel it's a coin flip dressed as a counter, because each serverless instance has its own memory. Five warm instances means five independent Maps means an effective limit of 150 requests per minute, per shop, while the code says 30 and the dashboard says 30 and I believed 30.

I proved it to myself by hitting production and logging x-vercel-id on consecutive requests:

msg4r
fsrhr
g7kxp
jdlv7
p2fsw

Five requests, five different instances. Five counters. The limit was a suggestion that each instance made independently.

Version two: Redis, and a silent death

I moved the counter to Upstash Redis over REST. That worked. Increment with INCR, set EXPIRE on the first hit, done. Cross-instance, atomic, cheap.

Then, months later, the free instance got reclaimed for inactivity. Four months of no traffic and the hostname simply stopped existing:

curl: (6) Could not resolve host: trusty-polecat-106618.upstash.io

Here's the part that actually scared me. Nothing broke. No 500s, no alert, no log line that I noticed. Because I had written the limiter fail-open — if the counter can't be reached, let the request through — a dead dependency meant the endpoint had no rate limit at all, and the only symptom was an invoice that was slightly higher than usual.

Fail-open was the right call for the wrong reason. I chose it so a Redis outage wouldn't block paying customers. I didn't think about what it does to my ability to notice that Redis is gone.

The endpoint was unlimited for an unknown period. Not hours. Probably weeks.

Version three: two dead ends before the one that worked

I tried to replace it twice before anything shipped.

Supabase. I already had a Postgres there. But the connection I had was the anon/service key against a project where I couldn't create a table, and I didn't have the database password. A rate limiter that needs a migration I can't run is not a rate limiter.

Cloudflare Zone Rate Limiting. This is the product built for exactly this. It refused to authenticate: my zone is on the Free plan, and the rulesets endpoint rejects the request for that tier. Correct behavior on Cloudflare's side, dead end on mine.

What worked: a Worker with a Durable Object. One Worker, one Durable Object namespace, SQLite-backed storage, free tier, no credit card. Each rate-limit key maps to a single DO instance, and a DO is single-threaded and strongly consistent — which is the entire property I needed. Two requests for the same shop cannot both read count = 29 and both decide they're number 30.

The Vercel side got two rules I'd set in advance and held to:

1. Isolated keys. Every key is prefixed pg: so this counter can never collide with anything else in the same namespace.
2. Fail-open, with a timeout. Remote error or slow response means allow. I'm not blocking a customer's order lookup because my counter is having a bad minute.

What the Durable Object actually is

A Durable Object is a JavaScript object with a name, guaranteed to run as a single instance, with its own SQLite storage. Two requests that address the same name cannot execute concurrently. That single property is what a counter needs, and it's the one thing serverless functions fundamentally cannot offer.

The Worker is a thin front door:

// worker: one DO per rate-limit key
export default {
  async fetch(req, env) {
    const url = new URL(req.url);
    const key = url.searchParams.get('key');
    if (req.headers.get('x-rl-key') !== env.RL_KEY) {
      return new Response('unauthorized', { status: 401 });
    }
    const id = env.LIMITER.idFromName(key);
    return env.LIMITER.get(id).fetch(req);
  },
};

// durable object: read, decide, write, no interleaving
async fetch(req) {
  const { limit, windowMs } = await req.json();
  const now = Date.now();
  await this.state.storage.transaction(async (txn) => {
    const row = (await txn.get('bucket')) ?? { count: 0, reset: now + windowMs };
    if (now > row.reset) { row.count = 0; row.reset = now + windowMs; }
    row.count += 1;
    await txn.put('bucket', row);
  });
  ...
}

idFromName(key) is what makes the limit per-shop: every request carrying the same shop domain lands on the same object. Different shops get different objects, so one shop's traffic can't delay another's.

The Vercel side is deliberately dumb. It calls the Worker with the shop key, a limit, and a window; on 429 it returns 429 to the client; on anything else — timeout, 500, DNS failure, malformed body — it lets the request through. Three seconds of timeout, then give up and serve the customer.

What the verification looked like

Unit level, against the Worker directly: no key returns 401. With limit=3, five sequential calls return 1, 2, 3 pass and 4, 5 rejected. Ten concurrent calls with limit=5 produce exactly five passes and five rejections — that's the test that matters, because it's the only one that can distinguish a real counter from a Map.

Production level, 35 sequential requests to /api/chat with a deliberately malformed payload so the app itself would answer 404 rather than spend tokens:

01–30  404   (limiter allowed, app rejected the payload)
31–35  429   (limiter rejected before the app saw it)

Requests 1 through 30 getting 404 is the proof that they got past the limiter. Then the reset check: wait 65 seconds, send three more, all three come back 404. The window actually reopens.

The number that makes me trust it: the 31st request is where the wall is, exactly, in production, on a platform where I had just demonstrated that five consecutive requests land on five different machines.

The trade-offs, both of them deliberate

Fail-open versus fail-closed. I chose availability. If the Durable Object is slow or unreachable, requests pass. The cost is concrete: an attacker who can reliably make my counter time out gets unlimited LLM calls on my bill. I accepted that because the alternative — a rate limiter outage becoming a customer-facing outage — is worse for a tool whose job is answering "where is my order" for someone's paying customer. I'd rather have a bill spike than a merchant's support flow going dark.

Per-shop objects versus one global counter. One Durable Object per shop means all of a shop's chat requests serialize through a single instance. At my volume that's invisible. But it is a real bottleneck by design: a single shop sending 500 requests a second queues behind itself. I chose this over a global counter because a global counter lets one abuser throttle every other merchant, which is a worse failure.

Latency. Every chat request now pays an extra hop to a Worker before it touches the model. On a request that spends hundreds of milliseconds in an LLM, that's noise. On a health check it would be unacceptable. I only pay it where the work is expensive anyway.

I also ruled out KV early. Its reads are eventually consistent and served from edge replicas, so counts drift. A rate limiter with drifting counts is a rate limiter with an asterisk, and the asterisk is where the abuse lives.

And I considered the obvious one: a Postgres row with UPDATE ... WHERE count < limit RETURNING count. That's genuinely correct and I had Postgres available. It lost on latency, not correctness — a round trip to a database in a single region, on every chat request, for a counter that only needs to be approximately right about time and exactly right about count. The Durable Object sits at the edge and keeps the counter next to the request. If I were already doing a database write per request for other reasons, I'd have used Postgres and skipped this entire detour.

Question

If your rate limiter fails open — and for customer-facing endpoints I think it should — how do you detect that it's silently dead? I found mine through a billing anomaly, which is not a monitoring strategy. I'm weighing "log every bypass and alert on volume" against "a synthetic request that must be rejected every minute." Which one actually fires for you, given that this failure mode is invisible by construction and the system looks healthiest right after the limiter dies?

Comments

[Alex Shev]

This is the classic difference between a per-process counter and a shared control plane. The next design question is failure behavior: if the shared store is unavailable, does the limiter fail closed for sensitive endpoints or degrade with an explicit bounded risk? Making that choice visible in metrics prevents a distributed rate limit from becoming an invisible best-effort feature.