DEV Community: Cheno

Wrote myself a rule about tests that pass for the wrong reason. One turn later I wrote a test that passed with the feature removed. Holding the lesson in your head is not a control.

Cheno — Mon, 31 Aug 2026 22:28:54 +0000

Six tests that passed for the wrong reason

Cheno — Mon, 31 Aug 2026 22:10:55 +0000

My backup script told me the restore was verified. It restored ten tables with zero rows in every one, checked the schema, and printed VERIFIED.

The schema check passed because an empty database has every table. The row counts were printed but never compared to anything. A dump that lost every row would have looked exactly like a dump that worked — and I’d have found out during the incident it existed to protect me from.

That was the sixth time in one project that a check passed without checking anything. Here are the others.

A stability gate with a fork nothing held in place. A rule read “recent-quiet” where it could have read “average-rate.” The implementation was correct. But when I flipped it to the wrong reading, all six test scenarios stayed green. The test suite couldn’t tell the two implementations apart, so nothing stopped a future edit from silently inverting it.

A health-check scenario running at 200 requests an hour. It existed to prove a low-volume endpoint wouldn’t false-alarm. It was never low-volume. I’d deferred a design decision for weeks on the grounds that fixing it would break that case — a case that couldn’t test the thing I was worried about. When I built the real one, the trade I’d been avoiding turned out not to exist.

A false-positive corpus that reported zero false positives. Including one I had already measured and knew was there. My “cloud scale-out” scenario specified requests-per-address directly, which meant volume was never held constant — which is the definition of a scale-out. The corpus wasn’t exercising the condition it was named after. I only caught it because 0/25 looked too good.

A bucketing test whose assertion and input were the same constant. It asserted hour == 9, which was only ever true because the timestamp above it was a hardcoded string containing 9. Two copies of one number, written twice, checking each other.

And the one that stings. After finding all of the above, I wrote myself a rule about it. One turn later I wrote a new test that passed with the feature it tested removed. Holding the lesson in my head did not stop me repeating it.

That’s the actual finding. Intuition doesn’t catch these. Every one of them was written by someone trying to be careful, and four of the six were guards I wrote specifically to catch this class of problem.

So it has to be mechanical. One question, asked of every guard before it counts as finished:

What input makes this pass without the property holding — and is that input in the suite as a negative case?

If you can’t name the input, the guard isn’t done. Not “I can’t think of one” — you have to be able to state it and then show the test failing against it. Every one of my six had an obvious answer once I asked: an empty database, the other implementation, a genuinely low-volume endpoint, a scale-out that actually scales out, a timestamp that isn’t the assertion.

The general shape is that a check measuring something adjacent to the claim looks identical to a check measuring the claim. Both are green. The difference only shows up on the day it matters, which is the day you can least afford it.

Related and worse: a check that fails correctly and gets ignored. I had a shell command joining a verification step to a commit with ; instead of &&. The verification failed. The commit went through anyway, with a message describing a change that hadn’t applied. A verification step that can’t stop the thing it verifies is decoration.

I’m building a leaked-API-key detector, which is how I ended up with this many guards in the first place — detection rules are mostly made of conditions that suppress alerts, and every one of those is a place where passing quietly is the failure mode. But nothing above is specific to security tooling. If you have a backup you’ve never restored, you have a belief, not a backup.

I Built a Leak Detector for API Keys. My Synthetic Tests Caught 8 Out of 20

Cheno — Mon, 24 Aug 2026 11:31:41 +0000

The honest story of building Cerberus and why I need real traffic to finish it.

A few days ago I wrote here about a test that failed. It was a synthetic data generator that fell over on real-world noise. I got more engagement on that post than anything I'd written before, which taught me something: developers don't trust perfection. They trust someone who shows the mess.
So here's the mess I'm in now.
The Problem
I sell API access for a living. A few months ago a customer's key leaked—committed to a public repo, scraped by a bot, used from a hundred places. We found out when the bill landed. The gap between the leak and the discovery is where all the damage lives.
What I Built
Cerberus watches usage metadata per API key and looks for one signature: a key being used from many places at once, each doing very little work.
A customer scaling legitimately runs on more machines and does proportionally more work, so requests-per-machine stays constant. A leaked credential inverts that ratio. More origins, less work per origin.
The detection rule uses three signals:
• Origin count per key
• Work per origin
• Network spread (unrelated networks vs. one cloud provider)
When all three hold for three consecutive hours, it sends one Slack message. That's it. No dashboard. No daily report. No noise.
Where It Falls Apart
I ran 20 synthetic leak scenarios through it. It caught eight.
The misses are all on the most common key shape: low baseline traffic, suddenly distributed. The thresholds are guesses. Every number in the rule is a guess, and the code comments literally say so.
I need real traffic history from actual API platforms to calibrate it. Synthetic data doesn't have the weird edge cases that real customers create.
The Privacy Architecture
Before anyone asks: keys and IPs never reach my server.
The SDK computes an HMAC-SHA256 of each value under a secret salt the customer holds. I never receive that salt, so I cannot reverse a fingerprint—not for the customer, not for an attacker who breaches me, not for anyone who compels me legally.
Twelve metadata fields only. Timestamp, hashed key, route template, token counts, latency, status, three hashed network fingerprints, address family, optional cost. The ingest endpoint rejects anything else by name. There is no field for a prompt, a response, a key, an address, or a user identifier. They can't be sent even by accident.
What I'm Looking For
Three API companies willing to install it free forever in exchange for sharing two weeks of traffic history so I can tune the detection.
First installs are done by hand on a call, deliberately, so there's a person to ask when something looks wrong.
If you sell API access and have ever worried about a leaked key, hit reply

We tested our rate limiter under concurrency. 128 threads made 128 attempts against a limit of 20.

Cheno — Thu, 20 Aug 2026 06:22:19 +0000

I’m building Cerberus, an API abuse detection service. Before it ships, I load-test the pieces. This is the story of one rate limiter bug that passed 300 tests, then failed immediately under threads.

The documented limit for authenticated customers was 600 requests per minute.

The real cap was 20.

The bug
Every request hit the unauthenticated bucket first. Then the system resolved the API token. If the token was valid, the request should not have counted against the unauthenticated bucket. But the charge had already happened.

The order was wrong.

Two SDK processes behind one NAT would exhaust the unauthenticated budget quickly. One process alone, running sequential requests, might never notice.

The bug was invisible to 300 tests because every test called the endpoint one request at a time. The failure needed 21 requests inside one minute. Sequential tests never produced that.

The fix introduced a race.
The first fix looked simple: peek the budget, resolve the token, charge on failure.

Correct sequentially.

Under threads, the overshoot equaled the thread count exactly. 128 threads got 128 attempts against a limit of 20.

The race window was between the budget check and the charge.

Roughly:

if bucket.available < 1:
reject

token = resolve(request)

if not tokenvalid:
bucket.charge(1)

One thread can read bucket. available, pass the check, and pause. Another thread does the same. By the time any thread charges the bucket, too many requests are already through.

That is not a rate limiter. That is a suggestion.

Why it never showed over HTTP
When we added concurrency, throughput went from 1 to 64 concurrent clients. Throughput only moved 1.16×.

Latency went from 101ms to 3,822ms.

That is a queue, not parallelism.

An async handler doing blocking database calls serializes everything. The serialization hid the race. If only one request is ever inside the critical section at a time, the race window never opens.

We only saw the overshoot when we tested the limiter directly under threads, without the HTTP layer hiding it.

The test that had the bug was written to catch it
There was a test designed to catch this class of bug.

It excluded functions that open a database connection. The reasoning was that those functions are the openers, not the victims.

The function that both opened a database connection and looped calling other functions was excluded.

So the test passed.

What changed
Every rate limiter test now runs with threads, not sequential calls.

We measure throughput and latency shape, not just pass/fail.

We include functions that open resources in concurrency tests.

We test the limiter directly and over HTTP.

The real lesson
Sequential tests will lie to you about anything that only fails under contention.

If a rate limiter passes 300 tests, but no test sends 21 requests in a minute or runs threads, it has not been tested.

The bug was not exotic. The order of two operations was wrong, and the test suite was structured to hide it. That is the normal failure mode for concurrency bugs.

The fix is not to add more tests. It is to make the tests actually contend.