I Wrote 238 Tests Against My Own Auth Package and Found 4 Real Bugs

I'd already done a lot right by the time I started writing tests for Beaver-Auth. Every module had gone through multiple rounds of deliberate review. Enumeration protection, hashed tokens, refresh rotation, TOTP replay defense — the design was solid, and I knew it was solid, because I'd thought hard about every piece of it.

Then I wrote 238 tests against the actual code, and found 8 real bugs. Some of them were the kind that would have silently broken production on day one.

This post isn't about the bugs specifically — it's about the gap between "I reviewed this carefully" and "this is shippable," and why that gap is bigger than most of us assume, even when the reviewing was genuinely careful.

"Passing tests" and "shippable" are different claims

Here's the trap I nearly walked into: I'd built a solid test suite covering the core auth flows — registration, login, verification — and every test passed. It felt done. But passing tests only tell you the code does what the tests expect. If the tests were written from the same mental model as the code, they'll happily confirm a bug is correct behavior, because both the code and the test agree on the same wrong assumption.

The fix wasn't "write more tests." It was testing against the real, integrated system — not a hand-built mock of my own logic, and not testing modules in isolation from what actually calls them. A few of the bugs below only surfaced because a test exercised the real dependency chain instead of assuming it worked.

Bug 1: TypeScript let an argument-shift bug compile clean

This is the one that scared me most. Beaver-Auth dispatches background work (like sending a verification email) through a TaskDispatcher interface:

interface TaskDispatcher {
  dispatch(
    taskName: string,
    payload: unknown,
    handler: () => Promise<void>,
    onFailure?: (error: unknown) => Promise<void> | void,
  ): Promise<void>
}

The default implementation had drifted to a different signature — missing the payload parameter entirely:

// what the class actually had:
async dispatch(
  taskName: string,
  handler: () => Promise<void>,
  onFailure?: (error: unknown) => Promise<void> | void,
): Promise<void> { /* ... */ }

Every caller still invoked it with all four arguments, matching the interface's shape. Positionally, that meant the payload object — a plain data object — landed in the handler slot, where the code expected a function. The real handler function landed in onFailure's slot. The real onFailure was silently dropped.

Run this, and handler() throws TypeError: handler is not a function — on literally every dispatch, in every deployment using the default dispatcher, which is the zero-config default nearly everyone would be using. Every verification email. Every password reset email. Silently, forever.

tsc --noEmit reported zero errors. Not a warning, nothing. Why? TypeScript checks method parameters bivariantly by default — permissively enough that a function with fewer parameters can satisfy an interface expecting more, especially when one of the mismatched parameter types is unknown (which structurally accepts anything). The type system had a real hole here, and nothing about writing careful code would have caught it — only running the actual dispatch path caught it.

The fix was a one-line signature correction. The lesson was: a compiling type signature is not proof the shapes actually agree at the call site. I added a regression test that exercises this exact path through the real dispatcher (not a test double) specifically so this can never silently regress again.

Bug 2: A feature that looked complete but had no way to actually be called

LoginEngine had a logoutJwtSession(familyId: string) method — revoke a refresh-token family, kill a JWT session's ability to mint new access tokens. Clean, well-tested in isolation.

Except: nothing, anywhere in the public API, ever returned a familyId to the caller. Login returned { status: 'success-jwt', user, accessToken, refreshToken } — no familyId. A consumer integrating this package had no way to obtain the one piece of information the logout function required. The feature was fully implemented and completely unreachable.

This is the kind of gap that unit tests of logoutJwtSession in isolation will never catch — you'd construct the test by directly passing in a familyId you already know, exactly like the broken consumer flow never could. It only surfaced once I wrote an integration test that role-played an actual consumer: log in, then try to log out using only what the login response gave you. That test couldn't be written without hitting the gap immediately.

The fix: thread familyId through the LoginResult type and every place it's constructed. Small change, but only findable by testing the usage path, not the unit.

Bug 3: My own error type leaked past its own contract

ValidationEngine is documented — by its own exported ValidationError class — to be the only kind of error it throws. Underneath, it delegates to an internal payload-normalization step that has its own error type, NormalizationError, used for resource-limit violations (someone sending a maliciously deep or wide payload).

NormalizationError was never wrapped. If normalization threw, ValidationEngine's public method let it propagate raw — an internal, not even exported from the package, error type. Any consumer following the documented pattern —

try {
  engine.validateInputs(input)
} catch (err) {
  if (err instanceof ValidationError) { /* handle it */ }
}

— would have that check silently fail exactly when a malicious payload tripped the depth/size limits. The one case where catching it correctly mattered most was the one case the contract didn't actually cover.

This is a good example of a bug that's invisible from reading the "happy path" code and only shows up when a test deliberately tries to break the documented contract, not just the documented behavior. I added a test that specifically threw a bad enough payload to trigger the internal limit, and asserted the thrown error was instanceof ValidationError — it wasn't, until I fixed it.

Bug 4: A security fix that quietly stopped working under realistic load

The rate limiter's token-bucket implementation stores bucket state with a TTL, so idle buckets eventually get cleaned up. The TTL was a fixed refillRateMs * 2. That number has nothing to do with how long a bucket actually takes to refill from empty — which is maxTokens * msPerToken.

Whenever maxTokens / refillAmount exceeded 2 (a completely ordinary configuration — say, 10 tokens refilling one at a time), the cache entry expired and got evicted well before the bucket would have naturally refilled. The practical effect: an attacker who went quiet for just over the (too-short) TTL got treated as a brand-new identifier on their next request — a fresh, full bucket, essentially resetting the rate limit for free.

This one needed a very deliberately constructed test to actually prove: advance simulated time past the old, buggy TTL but nowhere near the true refill time, and check whether the bucket reports a nearly-full state (bug present) or a correctly-partial one (bug fixed). Both outcomes are plausible-looking numbers — you can't catch this by eyeballing the code, only by doing the arithmetic and asserting the exact expected value under controlled, fake time.

The pattern underneath all of these

None of these bugs were "obviously" wrong on a read-through — I'd read through all of this code multiple times already. What they had in common:

• They lived at integration boundaries, not inside a single function's logic. The dispatcher bug was an interface/implementation mismatch. The familyId bug was a missing field crossing a return-type boundary. The NormalizationError leak was one module's contract silently depending on another's internals.
• They were invisible without exercising the real call path. A hand-rolled mock of "what I assume the dispatcher does" would have passed every test, because I would have mocked it to match my own (wrong) assumption about its shape.
• The type system helped, but wasn't sufficient. TypeScript caught plenty elsewhere in this project. It specifically didn't catch the one bug that would have broken the most things, because the unsoundness in method-parameter checking is a known, documented trade-off in the language — not a tooling failure, just a gap worth knowing about.

What I'd tell someone starting this process

1. Test against the real dependency graph, not a mock of your own mental model of it. If your test double was written by the same person who wrote the bug, it will happily agree with the bug.
2. Write integration tests that role-play an actual consumer, using only what your public API actually gives them — not what you, the author, happen to already know. This is what caught the unreachable logoutJwtSession.
3. Test your contracts, not just your happy paths. "This function is documented to only throw X" is a claim that needs its own test, deliberately trying to make it throw something else.
4. When you fix something subtle, write the test that would have caught it before the fix — check it actually would have failed against the old code. A regression test that can't prove it would have caught the regression isn't actually a regression test.
5. "It compiles" and "the tests pass" are both necessary and neither is sufficient. Verified against the real, wired-together system is the only claim actually worth making before you tell people something is shippable.

238 tests, 4 real bugs, and — as far as I know right now — a package I trust considerably more than the one I had before I started. Not because I got smarter partway through. Because I finally stopped grading my own homework.

---

Beaver-Auth is an open-source TypeScript auth package built on Node's built-in crypto, with a public test suite you can read and run yourself: github.com. If you find a bug I missed, that's genuinely the best possible outcome — here's how to report it.