DEV Community: Krishanu Das

Designing a Rate Limiter for an SMS API: The Mistakes I Made Along the Way

Krishanu Das — Sun, 11 Oct 2026 16:11:07 +0000

I'm working through system design as an AI engineer, and I'm writing up each problem as I go. This one is a rate limiter. It looked like the easy one. It caught me out in three places.

The problem

Picture a public API that sends SMS messages, something like Twilio. Thousands of companies call it from their servers, on free, paid and enterprise plans.

Two things keep going wrong. A buggy script on a free account loops forever and slows the API down for everyone else. And some customers send bursts to squeeze out more than they pay for.

So we need a rate limiter: a cap on how many requests a client can make in a time window. Go over the cap and the API answers with HTTP 429 Too Many Requests.

Step 1: Ask before designing

I wrote my questions down first. These are the answers I settled on:

Everything is limited together (sending and status checks), with one limit per client.
The client is the API key, not the IP address. Lots of customers can share one IP, and one customer can use many.
Over the limit gets a 429 with a Retry-After header.
Free gets 60 requests per minute, paid gets 600, enterprise gets 6,000. About 90% of active accounts are free, 9% are paid and 1% are enterprise.
100,000 accounts in total, and 20,000 of them active in the peak minute.
Many API servers behind a load balancer, and the check can only add a few milliseconds.

The question I almost skipped was "who is the client?" Nearly everything else in the design hangs off the answer.

Step 2: Estimate the worst case

I assumed every active account sits exactly at its limit.

Tier	Active accounts	Requests/min	Requests/sec
Free	18,000	1,080,000	18,000
Paid	1,800	1,080,000	18,000
Enterprise	200	1,200,000	20,000
Total	20,000	3,360,000	56,000

That's 56,000 requests per second, and the limiter checks every one of them, including the ones it ends up rejecting. A disk-based database can't take that on the hot path, so the counters have to live in memory.

One more number. Paid and enterprise accounts are only 10% of accounts, but they're 68% of the traffic. I didn't think that mattered until the very end.

Step 3: Where does the count live?

My first answer was that the limiter should "make sure the requests across servers add up to 60". That's just the requirement said again. It doesn't tell you how.

Here's what actually goes wrong. If every server keeps its own count, the load balancer spreads one customer's requests across, say, 3 servers, and each server only sees a third of them. With a local limit of 60 on each, that customer can send 180 requests a minute and nobody complains.

So the count has to live in one place that all the servers share. I'd use Redis, because it's in memory and handles this kind of load.

There are two kinds of data here, and I'd keep them apart:

Config: key X is on the free tier, so the limit is 60 a minute. This almost never changes, so each gateway can cache it in memory and refresh it every minute or so.
State: key X has used 43 so far. This changes on every request, so it lives in Redis.

I'll come back to why that split matters.

The race condition I missed

Say a key has used 59 of its 60 and two requests arrive at the same moment on two different servers. Each server reads the count (59), sees it's under 60, lets the request through and writes 60. Both got through, when only one should have.

My fix was to keep the servers in sync with each other. That doesn't work. They'd have to message each other on every request, which is slow and messy, and there's still a gap between "check" and "update" where both can decide at the same instant.

The real fix is to stop doing three steps (read, compare, write) and make it one step inside the store. Redis can add 1 to a counter and hand back the new value as a single operation. Server 1 gets 60 and goes ahead. Server 2 gets 61 and gets a 429. It's the same idea as the atomic claim that makes sure only one worker picks up a job from a message queue.

Step 4: Choosing the algorithm

Fixed window. Put the current minute in the key (X:12:00), increment it on every request, and let it expire. It's simple and cheap, one number per key.

But look at the boundary. A client sends 60 requests at 12:00:59 and another 60 at 12:01:00. That's 120 requests in about 2 seconds, and nothing gets blocked.

My first reaction was "that's fine, the window is whatever the platform defines". That's wrong. The limit exists to protect the backend, and the backend was sized for 60 a minute. If all 18,000 free accounts did this at once, it would see about 112,000 requests a second, double what it was built for.

Token bucket. Each client has a bucket that holds up to a set number of tokens. Every request takes one, and an empty bucket means a 429. Tokens drip back in at a steady rate.

The nice part is that you don't need a timer. You store two numbers per key, the tokens left and the time you last saw it, and you work out the refill only when a request shows up:

tokens_now = min(capacity, tokens_saved + (now - last_seen) x refill_rate)

Here's a small example I worked by hand (capacity 3, refill 1 token a second, bucket starts full):

Time	Tokens available	Result	Tokens left
t=0	3	Allowed	2
t=0	2	Allowed	1
t=0	1	Allowed	0
t=0	0	429	0
t=1	1	Allowed	0
t=5	3 (4 would have built up, capped at 3)	Allowed	2
t=6	3 (2 saved plus 1 refilled)	Allowed	2

I went with the token bucket because there's no boundary to exploit. What I give up is simplicity: two numbers per key instead of one, and the refill maths has to run as one atomic step, since it reads and writes two values.

Tier	Bucket size	Refill
Free	60	1 token/sec
Paid	600	10 tokens/sec
Enterprise	6,000	100 tokens/sec

One thing to know: a bucket the size of a full minute lets a client burst a whole minute's worth at once. If that's too generous, make the bucket smaller. That's a product decision more than a technical one.

The check itself

Redis runs a Lua script as one uninterrupted step, so the whole check is atomic:

local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])          -- tokens per second

local t   = redis.call('TIME')              -- Redis's own clock
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

local data   = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(data[1])
local ts     = tonumber(data[2])
if tokens == nil then tokens = capacity; ts = now end

tokens = math.min(capacity, tokens + math.max(0, now - ts) / 1000 * rate)

local allowed, retry_after = 0, 0
if tokens >= 1 then
  tokens = tokens - 1
  allowed = 1
else
  retry_after = math.ceil((1 - tokens) / rate)
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('EXPIRE', KEYS[1], math.ceil(capacity / rate) * 2)
return { allowed, math.floor(tokens), retry_after }

Two details. It uses Redis's clock, not each gateway's, because different machines have slightly different clocks and I don't want them disagreeing about "now". And idle keys expire on their own, so inactive accounts don't eat memory. A key that expired just starts with a full bucket, which is the right behaviour anyway.

The full picture

What the 429 should say

My first answer was "tell them the calls for the minute are over". That's wrong for a token bucket, because there is no minute. Tokens come back continuously, so the useful thing to tell a client is how long until one token exists:

HTTP/1.1 429 Too Many Requests
Retry-After: 1
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0

For a free key that refills 1 token a second, an empty bucket means Retry-After: 1. If it refilled 1 token every 2 seconds, it would be 2. Successful responses carry the X-RateLimit headers too, so a well-behaved client can slow down before it hits the wall.

Scaling it

56,000 checks a second is within what a single Redis node can do for simple commands, but Lua scripts cost more, so I'd want headroom. I'd spread keys across several Redis nodes with consistent hashing on the API key, so adding a node only moves about 1/N of the keys.

And a key that moves, or a counter that gets lost, isn't a disaster. The client just starts with a full bucket. Worst case, one extra burst.

Replication is asynchronous, so after a failover the last few updates can be lost and a client might briefly get a bit more than its limit. For a rate limiter I'm happy with that. It picks availability over perfect accuracy.

When the limiter itself breaks

If Redis is unreachable, the limiter can't check counts, and it has to pick a default.

Fail open: let requests through. Customers keep working, but nobody is limited.
Fail closed: reject requests. The backend is safe, but every customer gets errors.

My first answer was to split by tier: fail open for paying customers, fail closed for free ones. Then I gave a reason that pointed the opposite way ("I'd rather give paying customers an error"), which is fail closed. I'd mixed the two up.

Even after fixing that, the split doesn't hold up against the numbers. Paid and enterprise are 68% of the traffic, so failing open for them leaves two-thirds of everything unchecked. Failing closed for them means errors for the customers who pay. Failing closed for everyone means a protective component takes down the whole product.

What I'd do instead is fail open for everyone, with a safety net. If a gateway can't reach Redis, it switches to a local in-memory bucket using the tier limit divided by the number of gateways. It's rough, but it still stops a runaway script, and it raises an alert straight away.

This only works because each gateway already has the tier config cached. If tier info lived in Redis too, it would disappear in exactly the outage where I need it. That's the reason config and state live in different places.

What I took away

Ask who the client is before anything else. The rest of the design hangs off it.
A counter that each server keeps by itself isn't a limit. The count has to be shared.
"Read, check, write" is a race. Make it a single atomic step inside the store.
A fixed window leaks at the boundary. Check an algorithm against what the limit is protecting, not just against its own definition.
Your 429 response has to match your algorithm. With a token bucket, tell the client how many seconds to wait, not that the minute is over.
Decide what happens when the limiter fails before it happens, and check the decision against your own traffic numbers.

Next I'm going to try the order notifications for something like Flipkart, and this time I'll attempt the whole design on my own before I look anything up.

Designing a URL Shortener From First Principles (and the Hashing Trap I Fell Into)

Krishanu Das — Tue, 22 Sep 2026 07:14:46 +0000

I'm working through system design as an AI engineer, and I'm writing up each problem as I go. This one is the classic: design a URL shortener like bit.ly. It looks simple, but it has one design decision that catches people out, and I walked right into it.

The two operations

Shorten: submit a long URL, get back a short one
Redirect: visit the short URL, get sent to the original

POST /urls { "longUrl": "...", "expiresAt": "..." } → 201 { "shortUrl": "sho.rt/21" }
GET /{shortCode} → 302 redirect to longUrl

Step 1: Estimate before designing

I anchored on a user base and derived everything else from it, rather than guessing RPS directly.

Reads (clicks):

100M total users, 50M active per day
~5 clicks per active user per day → 250M clicks/day
250M ÷ 86,400 seconds ≈ 2,900 RPS

Writes (creating links):
Creating a link is much rarer than clicking one. My first estimate had more creates than clicks, which contradicted the read-heavy intuition, so I redid it:

~5M users create links on a given day, ~5 links each → 25M creates/day
25M ÷ 86,400 ≈ 290 RPS

That's roughly a 10:1 read/write ratio, and it shapes everything below: the design effort belongs on the read path.

Storage:
Each record holds the long URL (~100 bytes), short code (~10), creator ID (~8–16), created timestamp (~8), expiry (~8) and a click counter (~8). That rounds to ~150 bytes.

25M records/day × 365 days ≈ 9.1B records/year
9.1B × 150 bytes ≈ 1.37 TB/year

1.37 TB a year fits on a single well-provisioned database. Storage isn't the bottleneck here, so there's no reason to shard just because the problem "sounds like scale." The numbers should drive that call, not the vibe.

Latency target: 100–200 ms. A redirect is a lookup with no heavy computation in the path, so it should feel instant.

Step 2: The architecture

flowchart LR
    U[User / Browser] --> LB[Load Balancer]
    LB --> A1[App Server 1]
    LB --> A2[App Server 2]
    LB --> A3[App Server N]
    A1 & A2 & A3 --> C[("Redis cache: shortCode to longUrl")]
    A1 & A2 & A3 --> DB[("Database: urls table + sequence")]

Stateless app servers sit behind a load balancer. A cache holds hot redirects, and a single relational database stores the records and generates IDs.

Step 3: The hard part is generating the short code

This is the actual design problem. There were three options:

flowchart TD
    Q{"How do we generate the short code?"}
    Q --> H[Hash the long URL]
    Q --> R[Random string]
    Q --> K[Counter + Base62]
    H --> H1["Same URL always gives the same code: breaks per-user expiry and analytics"]
    R --> R1["Can collide: needs retry when the DB rejects a duplicate"]
    K --> K1["Unique by construction: needs one shared counter"]

The hashing trap

My first instinct was to hash the long URL. It feels natural, since hashing turns a long input into a short output.

Then consider this case. User A shortens example.com/page in January for a 30-day campaign and wants their own click analytics. User B shortens the same URL in July with a different expiry.

Hashing is deterministic: the same input always gives the same output. Both users would get the same short code, and therefore one shared record. Whose expiry wins? Whose clicks are whose?

The real requirement is that each shorten request is its own record, because ownership, expiry and analytics belong to the request, not to the URL. So the short code has to be generated independently of the URL's content.

Random vs counter

Random strings mostly work, but they can collide. The database's unique constraint catches a collision, and you retry. As the table grows into billions of rows, collisions become less rare (the birthday problem).

A counter encoded in Base62 can't collide at all. Take an ever-increasing number and encode it with [0-9a-zA-Z]:

125 = 2 × 62 + 1  →  "21"

No two counter values are the same, so no two codes are the same. There's no retry logic needed.

How long does it last? With 7 characters, 62⁷ ≈ 3.5 trillion codes. At 9.1B links a year, that's over 380 years of headroom.

The distributed counter problem

There are many app servers. If each one keeps its own counter in memory, two servers can hand out the same number at the same moment.

There are two standard fixes:

Let the database own the counter (an auto-increment column or a Postgres SEQUENCE). The database guarantees uniqueness even under concurrent load.
Hand out ranges. Each server grabs a block of, say, 1,000 IDs at a time and assigns them locally. This reduces contention on the shared counter, at the cost of some wasted IDs if a server crashes mid-block.

I went with option 1. At ~290 creates/sec, a single database sequence handles the load comfortably, so ranges would solve a problem this system doesn't have. Match the complexity to the numbers you actually calculated.

sequenceDiagram
    participant C as Client
    participant S as App Server
    participant DB as Database
    C->>S: POST /urls (longUrl, expiresAt)
    S->>DB: nextval(url_seq)
    DB-->>S: 125
    S->>S: base62(125) = "21"
    S->>DB: INSERT shortCode, longUrl, createdBy, createdAt, expiresAt
    DB-->>S: OK
    S-->>C: 201 shortUrl = sho.rt/21

Step 4: Make the read path fast

With a 10:1 read/write ratio, redirects are where performance matters.

Index: the short code is the primary key, so lookups are indexed automatically.
Cache: hot shortCode → longUrl mappings live in Redis. The cache key is just the short code, because a redirect is public and identical for everyone.

sequenceDiagram
    participant C as Client
    participant S as App Server
    participant R as Redis
    participant DB as Database
    C->>S: GET /21
    S->>R: GET 21
    alt cache hit
        R-->>S: longUrl
    else cache miss
        R-->>S: null
        S->>DB: SELECT longUrl WHERE shortCode = 21
        DB-->>S: longUrl
        S->>R: SET 21 longUrl with TTL
    end
    S-->>C: 302 redirect to longUrl

Why 302 and not 301? Browsers cache a 301 (permanent) redirect, so repeat clicks never reach the server and the click counts would be wrong. A 302 keeps every click visible.

What I'd revisit next

Guessable codes: Sequential codes mean anyone can walk through 21, 22, 23... and find other people's links. If that matters, shuffle or obfuscate the ID before encoding it.
Expiry cleanup: Expired links need a background job to purge them, and the cache TTL shouldn't outlive the link's own expiry.
Cache eviction: Decide on a policy (LRU is the usual choice) and how big the cache needs to be.

Takeaways

Derive the numbers, don't guess them. Total users, then active users, then per-user rate, then RPS. The same method works for storage.
Sanity-check your estimates against each other. My first write estimate exceeded my read estimate in a read-heavy system, and that contradiction was the signal to redo it.
Not every system needs to shard. 1.37 TB a year fits on one box.
Deterministic isn't always what you want. Hashing is ideal when identical inputs should map together. Here they shouldn't.
Match solution complexity to real scale. Use a database sequence at 290 writes/sec, and switch to ID ranges when the numbers say so.

This is part of a series where I learn system design in public. Next up: designing a rate limiter.