DEV Community: Ozoemena John

I Wrote 238 Tests Against My Own Auth Package and Found 4 Real Bugs

Ozoemena John — Wed, 19 Aug 2026 00:20:36 +0000

I'd already done a lot right by the time I started writing tests for Beaver-Auth. Every module had gone through multiple rounds of deliberate review. Enumeration protection, hashed tokens, refresh rotation, TOTP replay defense — the design was solid, and I knew it was solid, because I'd thought hard about every piece of it.

Then I wrote 238 tests against the actual code, and found 8 real bugs. Some of them were the kind that would have silently broken production on day one.

This post isn't about the bugs specifically — it's about the gap between "I reviewed this carefully" and "this is shippable," and why that gap is bigger than most of us assume, even when the reviewing was genuinely careful.

"Passing tests" and "shippable" are different claims

Here's the trap I nearly walked into: I'd built a solid test suite covering the core auth flows — registration, login, verification — and every test passed. It felt done. But passing tests only tell you the code does what the tests expect. If the tests were written from the same mental model as the code, they'll happily confirm a bug is correct behavior, because both the code and the test agree on the same wrong assumption.

The fix wasn't "write more tests." It was testing against the real, integrated system — not a hand-built mock of my own logic, and not testing modules in isolation from what actually calls them. A few of the bugs below only surfaced because a test exercised the real dependency chain instead of assuming it worked.

Bug 1: TypeScript let an argument-shift bug compile clean

This is the one that scared me most. Beaver-Auth dispatches background work (like sending a verification email) through a TaskDispatcher interface:

interface TaskDispatcher {
  dispatch(
    taskName: string,
    payload: unknown,
    handler: () => Promise<void>,
    onFailure?: (error: unknown) => Promise<void> | void,
  ): Promise<void>
}

The default implementation had drifted to a different signature — missing the payload parameter entirely:

// what the class actually had:
async dispatch(
  taskName: string,
  handler: () => Promise<void>,
  onFailure?: (error: unknown) => Promise<void> | void,
): Promise<void> { /* ... */ }

Every caller still invoked it with all four arguments, matching the interface's shape. Positionally, that meant the payload object — a plain data object — landed in the handler slot, where the code expected a function. The real handler function landed in onFailure's slot. The real onFailure was silently dropped.

Run this, and handler() throws TypeError: handler is not a function — on literally every dispatch, in every deployment using the default dispatcher, which is the zero-config default nearly everyone would be using. Every verification email. Every password reset email. Silently, forever.

tsc --noEmit reported zero errors. Not a warning, nothing. Why? TypeScript checks method parameters bivariantly by default — permissively enough that a function with fewer parameters can satisfy an interface expecting more, especially when one of the mismatched parameter types is unknown (which structurally accepts anything). The type system had a real hole here, and nothing about writing careful code would have caught it — only running the actual dispatch path caught it.

The fix was a one-line signature correction. The lesson was: a compiling type signature is not proof the shapes actually agree at the call site. I added a regression test that exercises this exact path through the real dispatcher (not a test double) specifically so this can never silently regress again.

Bug 2: A feature that looked complete but had no way to actually be called

LoginEngine had a logoutJwtSession(familyId: string) method — revoke a refresh-token family, kill a JWT session's ability to mint new access tokens. Clean, well-tested in isolation.

Except: nothing, anywhere in the public API, ever returned a familyId to the caller. Login returned { status: 'success-jwt', user, accessToken, refreshToken } — no familyId. A consumer integrating this package had no way to obtain the one piece of information the logout function required. The feature was fully implemented and completely unreachable.

This is the kind of gap that unit tests of logoutJwtSession in isolation will never catch — you'd construct the test by directly passing in a familyId you already know, exactly like the broken consumer flow never could. It only surfaced once I wrote an integration test that role-played an actual consumer: log in, then try to log out using only what the login response gave you. That test couldn't be written without hitting the gap immediately.

The fix: thread familyId through the LoginResult type and every place it's constructed. Small change, but only findable by testing the usage path, not the unit.

Bug 3: My own error type leaked past its own contract

ValidationEngine is documented — by its own exported ValidationError class — to be the only kind of error it throws. Underneath, it delegates to an internal payload-normalization step that has its own error type, NormalizationError, used for resource-limit violations (someone sending a maliciously deep or wide payload).

NormalizationError was never wrapped. If normalization threw, ValidationEngine's public method let it propagate raw — an internal, not even exported from the package, error type. Any consumer following the documented pattern —

try {
  engine.validateInputs(input)
} catch (err) {
  if (err instanceof ValidationError) { /* handle it */ }
}

— would have that check silently fail exactly when a malicious payload tripped the depth/size limits. The one case where catching it correctly mattered most was the one case the contract didn't actually cover.

This is a good example of a bug that's invisible from reading the "happy path" code and only shows up when a test deliberately tries to break the documented contract, not just the documented behavior. I added a test that specifically threw a bad enough payload to trigger the internal limit, and asserted the thrown error was instanceof ValidationError — it wasn't, until I fixed it.

Bug 4: A security fix that quietly stopped working under realistic load

The rate limiter's token-bucket implementation stores bucket state with a TTL, so idle buckets eventually get cleaned up. The TTL was a fixed refillRateMs * 2. That number has nothing to do with how long a bucket actually takes to refill from empty — which is maxTokens * msPerToken.

Whenever maxTokens / refillAmount exceeded 2 (a completely ordinary configuration — say, 10 tokens refilling one at a time), the cache entry expired and got evicted well before the bucket would have naturally refilled. The practical effect: an attacker who went quiet for just over the (too-short) TTL got treated as a brand-new identifier on their next request — a fresh, full bucket, essentially resetting the rate limit for free.

This one needed a very deliberately constructed test to actually prove: advance simulated time past the old, buggy TTL but nowhere near the true refill time, and check whether the bucket reports a nearly-full state (bug present) or a correctly-partial one (bug fixed). Both outcomes are plausible-looking numbers — you can't catch this by eyeballing the code, only by doing the arithmetic and asserting the exact expected value under controlled, fake time.

The pattern underneath all of these

None of these bugs were "obviously" wrong on a read-through — I'd read through all of this code multiple times already. What they had in common:

They lived at integration boundaries, not inside a single function's logic. The dispatcher bug was an interface/implementation mismatch. The familyId bug was a missing field crossing a return-type boundary. The NormalizationError leak was one module's contract silently depending on another's internals.
They were invisible without exercising the real call path. A hand-rolled mock of "what I assume the dispatcher does" would have passed every test, because I would have mocked it to match my own (wrong) assumption about its shape.
The type system helped, but wasn't sufficient. TypeScript caught plenty elsewhere in this project. It specifically didn't catch the one bug that would have broken the most things, because the unsoundness in method-parameter checking is a known, documented trade-off in the language — not a tooling failure, just a gap worth knowing about.

What I'd tell someone starting this process

Test against the real dependency graph, not a mock of your own mental model of it. If your test double was written by the same person who wrote the bug, it will happily agree with the bug.
Write integration tests that role-play an actual consumer, using only what your public API actually gives them — not what you, the author, happen to already know. This is what caught the unreachable logoutJwtSession.
Test your contracts, not just your happy paths. "This function is documented to only throw X" is a claim that needs its own test, deliberately trying to make it throw something else.
When you fix something subtle, write the test that would have caught it before the fix — check it actually would have failed against the old code. A regression test that can't prove it would have caught the regression isn't actually a regression test.
"It compiles" and "the tests pass" are both necessary and neither is sufficient. Verified against the real, wired-together system is the only claim actually worth making before you tell people something is shippable.

238 tests, 4 real bugs, and — as far as I know right now — a package I trust considerably more than the one I had before I started. Not because I got smarter partway through. Because I finally stopped grading my own homework.

Beaver-Auth is an open-source TypeScript auth package built on Node's built-in crypto, with a public test suite you can read and run yourself: github.com. If you find a bug I missed, that's genuinely the best possible outcome — here's how to report it.

Why Your Login Response Time Is a Security Leak

Ozoemena John — Sun, 09 Aug 2026 15:38:01 +0000

You did everything right. Your login endpoint returns a generic "invalid credentials" for both a wrong password and a nonexistent email. No information leaked — right?

Not quite. There's a channel most of us never think to close: how long the response took to come back.

The leak, made concrete

Here's roughly what a "correct-looking" login handler does:

const user = await db.findUserByEmail(email)

if (!user) {
  return { status: 'invalid-credentials' }
}

const isValid = await verifyPassword(password, user.passwordHash)

if (!isValid) {
  return { status: 'invalid-credentials' }
}

Both failure paths return the exact same response shape. Looks airtight. But watch what actually happens on the wire:

Email doesn't exist: one database lookup, then an immediate return. Fast — a few milliseconds.
Email exists, password is wrong: the database lookup, plus a full password verification. If you're using bcrypt, scrypt, or Argon2 (and you should be), that step is deliberately slow — often 50–200ms, by design, to resist brute-forcing.

An attacker doesn't need to read your response body at all. They just need a stopwatch. Submit a candidate email with any password, measure the response time, and a gap of 100+ ms tells them the account exists — invisibly, at scale, across thousands of addresses, with zero errors logged that look unusual (they're all just "failed logins").

This is a real, well-known class of attack called a timing side-channel, and it's one of the easier ones to actually pull off, because HTTP response timing is something every client can measure for free.

The instinctive fix doesn't fully work

The first thing most people try:

if (!user) {
  // "compare against something" so the timing looks similar
  await bcrypt.compare(password, DUMMY_HASH)
  return { status: 'invalid-credentials' }
}

Better! Now both branches do some expensive hashing work. But this only gets you closer, not equal — and "closer" is exactly the gap a patient attacker exploits. Real systems have jitter: network latency, GC pauses, database connection pool contention. A few milliseconds of consistent difference, averaged over a few hundred requests per candidate email, is still statistically detectable. You've raised the cost of the attack, not eliminated it.

There's also a second, easy-to-miss version of this same bug: inconsistent response shape, not just timing. If your registration endpoint returns { status: 'success', user: {...} } for a new email but { status: 'success-pending-verification' } (no user object) for one that already exists — congratulations, you've built a perfectly reliable, zero-timing-analysis-required account enumeration oracle. This is arguably worse than the timing leak, because it doesn't even require statistics — one request tells you everything.

What actually works: a response floor

The fix that closes this properly isn't "make the branches equally fast" (hard to guarantee) — it's "pad every branch up to a fixed minimum time," so the total response time is constant regardless of which internal path executed:

private async holdToFloor(start: number): Promise<void> {
  const elapsed = Date.now() - start
  const remaining = this.responseFloorMs - elapsed
  if (remaining > 0) {
    await new Promise((resolve) => setTimeout(resolve, remaining))
  }
}

Wrap this around every exit path of the endpoint — success, wrong password, nonexistent user, even unexpected server errors — and the observable timing collapses to "at least responseFloorMs," full stop. It doesn't matter if the actual work took 3ms or 80ms; the client always waits until the floor. No amount of statistical averaging recovers a timing signal that was never there.

public async executePasswordStage(rawInput, secret, context, options) {
  const start = Date.now()

  try {
    const user = await this.adapter.findUserByEmail(email)

    if (!user || !user.passwordHash) {
      // still do the expensive comparison, so this path costs roughly
      // the same as the real one — belt AND suspenders with the floor
      await this.crypto.verifyPassword(password, STATIC_DUMMY_HASH)
      await this.holdToFloor(start)
      return { status: 'invalid-credentials' }
    }

    const isValid = await this.crypto.verifyPassword(password, user.passwordHash)

    if (!isValid) {
      await this.holdToFloor(start)
      return { status: 'invalid-credentials' }
    }

    // ...success path also goes through holdToFloor before returning
  } catch (err) {
    await this.holdToFloor(start)
    return { status: 'system-error', message: '...' }
  }
}

Note the dummy-hash comparison is still there — the floor is defense in depth, not a replacement for doing real, comparable work on both paths. Belt and suspenders.

The response-shape fix, applied to registration

The same principle applies to what you return, not just when. For registration specifically, this means every branch — new email, existing email, even with enumeration protection explicitly disabled — has to converge to identical shapes:

if (existingUser) {
  if (this.protectAgainstEnumeration) {
    // do the SAME unit of real work as the new-user branch,
    // so there's no synchronous cost difference to leak either
    await this.crypto.hashPassword(password ?? DUMMY_PASSWORD)
    return { status: 'success-pending-verification' }
  }
  return { status: 'email-already-exists' }
}

// new user path
const user = await this.adapter.createUser({ ...})
// fire-and-forget the verification email — deliberately NOT awaited,
// so its cost never shows up in this response's timing at all
void this.sendVerificationEmail(user)

return { status: 'success-pending-verification' }

Both the existing-email and new-email paths return the exact same status, with no extra fields that differ. An attacker probing your /register endpoint with a list of candidate emails gets back one indistinguishable response, every time.

The trade-off you're actually making

This isn't free. A fixed response floor means every legitimate user waits at least that long too, even on your fastest possible code path. At scale, that's real, deliberately-added latency, and in serverless environments specifically, it's billed latency — you're paying for milliseconds you didn't strictly need, on every single request.

The honest way to think about it: pick the floor based on your slowest legitimate branch's realistic p95, not an arbitrary round number. If your real success path takes 60–80ms under load, a 150ms floor is padding a fixed, bounded amount — not multiplying your latency. Measure it, don't guess it.

The takeaway

"Return the same error message" is necessary but nowhere near sufficient for an auth endpoint that needs to resist enumeration. The full checklist looks more like:

Same response shape, in every branch, with no extra/missing fields.
Comparable real work on every branch (a dummy hash comparison when there's no real one to do).
A hard floor on total response time, applied after everything else, covering success and error paths alike.

Skip any one of the three and you've left a measurable signal on the table — and "measurable" is all a patient attacker needs.

This post describes design decisions from beaver-auth, an open-source TypeScript auth package built on Node's built-in crypto. If you're building auth from scratch, beaver-auth's RegistrationEngine and LoginEngine implement all three of the fixes above by default — response floor, enumeration-safe response shapes, and dummy-hash comparisons aren't opt-in extras, they're how the package behaves out of the box.