I’m building Cerberus, an API abuse detection service. Before it ships, I load-test the pieces. This is the story of one rate limiter bug that passed 300 tests, then failed immediately under threads.
The documented limit for authenticated customers was 600 requests per minute.
The real cap was 20.
The bug
Every request hit the unauthenticated bucket first. Then the system resolved the API token. If the token was valid, the request should not have counted against the unauthenticated bucket. But the charge had already happened.
The order was wrong.
Two SDK processes behind one NAT would exhaust the unauthenticated budget quickly. One process alone, running sequential requests, might never notice.
The bug was invisible to 300 tests because every test called the endpoint one request at a time. The failure needed 21 requests inside one minute. Sequential tests never produced that.
The fix introduced a race.
The first fix looked simple: peek the budget, resolve the token, charge on failure.
Correct sequentially.
Under threads, the overshoot equaled the thread count exactly. 128 threads got 128 attempts against a limit of 20.
The race window was between the budget check and the charge.
Roughly:
if bucket.available < 1:
reject
token = resolve(request)
if not tokenvalid:
bucket.charge(1)
One thread can read bucket. available, pass the check, and pause. Another thread does the same. By the time any thread charges the bucket, too many requests are already through.
That is not a rate limiter. That is a suggestion.
Why it never showed over HTTP
When we added concurrency, throughput went from 1 to 64 concurrent clients. Throughput only moved 1.16×.
Latency went from 101ms to 3,822ms.
That is a queue, not parallelism.
An async handler doing blocking database calls serializes everything. The serialization hid the race. If only one request is ever inside the critical section at a time, the race window never opens.
We only saw the overshoot when we tested the limiter directly under threads, without the HTTP layer hiding it.
The test that had the bug was written to catch it
There was a test designed to catch this class of bug.
It excluded functions that open a database connection. The reasoning was that those functions are the openers, not the victims.
The function that both opened a database connection and looped calling other functions was excluded.
So the test passed.
What changed
Every rate limiter test now runs with threads, not sequential calls.
We measure throughput and latency shape, not just pass/fail.
We include functions that open resources in concurrency tests.
We test the limiter directly and over HTTP.
The real lesson
Sequential tests will lie to you about anything that only fails under contention.
If a rate limiter passes 300 tests, but no test sends 21 requests in a minute or runs threads, it has not been tested.
The bug was not exotic. The order of two operations was wrong, and the test suite was structured to hide it. That is the normal failure mode for concurrency bugs.
The fix is not to add more tests. It is to make the tests actually contend.
Top comments (0)