I Built a Leak Detector for API Keys. My Synthetic Tests Caught 8 Out of 20

Cheno — Mon, 24 Aug 2026 11:31:41 +0000

The honest story of building Cerberus and why I need real traffic to finish it.

A few days ago I wrote here about a test that failed. It was a synthetic data generator that fell over on real-world noise. I got more engagement on that post than anything I'd written before, which taught me something: developers don't trust perfection. They trust someone who shows the mess.
So here's the mess I'm in now.
The Problem
I sell API access for a living. A few months ago a customer's key leaked—committed to a public repo, scraped by a bot, used from a hundred places. We found out when the bill landed. The gap between the leak and the discovery is where all the damage lives.
What I Built
Cerberus watches usage metadata per API key and looks for one signature: a key being used from many places at once, each doing very little work.
A customer scaling legitimately runs on more machines and does proportionally more work, so requests-per-machine stays constant. A leaked credential inverts that ratio. More origins, less work per origin.
The detection rule uses three signals:
• Origin count per key
• Work per origin
• Network spread (unrelated networks vs. one cloud provider)
When all three hold for three consecutive hours, it sends one Slack message. That's it. No dashboard. No daily report. No noise.
Where It Falls Apart
I ran 20 synthetic leak scenarios through it. It caught eight.
The misses are all on the most common key shape: low baseline traffic, suddenly distributed. The thresholds are guesses. Every number in the rule is a guess, and the code comments literally say so.
I need real traffic history from actual API platforms to calibrate it. Synthetic data doesn't have the weird edge cases that real customers create.
The Privacy Architecture
Before anyone asks: keys and IPs never reach my server.
The SDK computes an HMAC-SHA256 of each value under a secret salt the customer holds. I never receive that salt, so I cannot reverse a fingerprint—not for the customer, not for an attacker who breaches me, not for anyone who compels me legally.
Twelve metadata fields only. Timestamp, hashed key, route template, token counts, latency, status, three hashed network fingerprints, address family, optional cost. The ingest endpoint rejects anything else by name. There is no field for a prompt, a response, a key, an address, or a user identifier. They can't be sent even by accident.
What I'm Looking For
Three API companies willing to install it free forever in exchange for sharing two weeks of traffic history so I can tune the detection.
First installs are done by hand on a call, deliberately, so there's a person to ask when something looks wrong.
If you sell API access and have ever worried about a leaked key, hit reply

We tested our rate limiter under concurrency. 128 threads made 128 attempts against a limit of 20.

Cheno — Thu, 20 Aug 2026 06:22:19 +0000

I’m building Cerberus, an API abuse detection service. Before it ships, I load-test the pieces. This is the story of one rate limiter bug that passed 300 tests, then failed immediately under threads.

The documented limit for authenticated customers was 600 requests per minute.

The real cap was 20.

The bug
Every request hit the unauthenticated bucket first. Then the system resolved the API token. If the token was valid, the request should not have counted against the unauthenticated bucket. But the charge had already happened.

The order was wrong.

Two SDK processes behind one NAT would exhaust the unauthenticated budget quickly. One process alone, running sequential requests, might never notice.

The bug was invisible to 300 tests because every test called the endpoint one request at a time. The failure needed 21 requests inside one minute. Sequential tests never produced that.

The fix introduced a race.
The first fix looked simple: peek the budget, resolve the token, charge on failure.

Correct sequentially.

Under threads, the overshoot equaled the thread count exactly. 128 threads got 128 attempts against a limit of 20.

The race window was between the budget check and the charge.

Roughly:

if bucket.available < 1:
reject

token = resolve(request)

if not tokenvalid:
bucket.charge(1)

One thread can read bucket. available, pass the check, and pause. Another thread does the same. By the time any thread charges the bucket, too many requests are already through.

That is not a rate limiter. That is a suggestion.

Why it never showed over HTTP
When we added concurrency, throughput went from 1 to 64 concurrent clients. Throughput only moved 1.16×.

Latency went from 101ms to 3,822ms.

That is a queue, not parallelism.

An async handler doing blocking database calls serializes everything. The serialization hid the race. If only one request is ever inside the critical section at a time, the race window never opens.

We only saw the overshoot when we tested the limiter directly under threads, without the HTTP layer hiding it.

The test that had the bug was written to catch it
There was a test designed to catch this class of bug.

It excluded functions that open a database connection. The reasoning was that those functions are the openers, not the victims.

The function that both opened a database connection and looped calling other functions was excluded.

So the test passed.

What changed
Every rate limiter test now runs with threads, not sequential calls.

We measure throughput and latency shape, not just pass/fail.

We include functions that open resources in concurrency tests.

We test the limiter directly and over HTTP.

The real lesson
Sequential tests will lie to you about anything that only fails under contention.

If a rate limiter passes 300 tests, but no test sends 21 requests in a minute or runs threads, it has not been tested.

The bug was not exotic. The order of two operations was wrong, and the test suite was structured to hide it. That is the normal failure mode for concurrency bugs.

The fix is not to add more tests. It is to make the tests actually contend.

DEV Community: Cheno

I Built a Leak Detector for API Keys. My Synthetic Tests Caught 8 Out of 20

We tested our rate limiter under concurrency. 128 threads made 128 attempts against a limit of 20.