Some bugs hide in logic. This one hid in the gap between "the code looks correct" and "the code is actually working" and the only way I found it was by refusing to trust either claim without checking the live system directly.
The setup
Mid-migration between two credential-storage systems: an older secrets vault (call it the legacy vault) used by an original provider integration, and a newer, purpose-built vault for a self-hosted replacement being rolled out gradually. During the transition, both exist side by side. The code that decides which vault to check for a given user's credentials looked, on review, completely correct it classified each connection record as "legacy" or "new" based on a stored field, and looked in the matching vault.
Nothing about the code review raised a flag. Which is exactly the problem with code review as your only line of defense: it tells you the logic is internally consistent, not that the systems it's talking to still exist.
Building the audit, properly this time
Instead of trusting the classification logic in the abstract, I wrote a small audit script with one rule: don't reconstruct what a secret's name should be and assume it's there actually query the live vault and see. This sounds obvious. It's also very easy to skip, because "reconstruct the expected name, spot-check a couple" feels like it should be equivalent to "check them all live," and it really isn't.
The script pulled every connection record from the database, classified each one the same way the app's own code did, and then made a real, live call to whichever vault it should be in.
What it found
Every single legacy-vault lookup failed. Not "secret not found" a connection failure. The vault itself wasn't resolving anymore.
That's a materially different problem than a missing secret. A missing secret means "someone forgot to write this." A vault that doesn't resolve at all means the entire piece of infrastructure the code has been confidently pointing at no longer exists and every single credential-write attempt to it, for every legacy user, had been silently failing, this whole time, for however long it had been broken.
How did nobody notice?
The honest, slightly embarrassing answer: the code that wrote to this vault caught the failure, logged a warning, and moved on. Not a crash. Not an alert. A line in a log nobody was watching, for a code path nobody expected to fail, because "the vault exists" was baked in as an assumption so early in the system's life that it had stopped being something anyone thought to re-check.
This is the actual shape of the bug, more than the specific vault: a try/catch that swallows a failure gracefully is indistinguishable, from the outside, from success until someone goes looking.
The part that actually surprised me
Here's where it got more interesting than "we found a broken thing, went and fixed it." Digging into why the legacy vault mattered at all, it turned out the underlying legacy provider managed those users' actual live sessions entirely on its own side the app's cached copy of their credentials was never actually read from again in practice. The vault had been dead, silently, for a long stretch, and functionally, nothing legacy-side had actually broken because of it.
That's not a "phew, no harm done, nothing to see here" ending though. It's a different, more uncomfortable one: we got lucky that this specific dead dependency happened to be redundant. The audit didn't know that in advance, and neither did I. The only way to find out whether a silently-dead piece of infrastructure actually matters is to go looking you can't reason your way to "probably fine" from the code alone.
The same audit, run against the newer vault the one actually still in active, load-bearing use found a real, smaller, actionable problem: a couple of expected entries genuinely missing. Real incomplete setup, not infrastructure rot. Fixed directly.
What I took away from this
- Code review confirms logic. It cannot confirm that the systems the logic depends on still exist. Those are different guarantees, and it's easy to unconsciously treat the first as if it implies the second.
- A caught-and-logged failure is a blind spot by construction, not just bad luck the entire point of catching it gracefully is that nothing breaks loudly. That's good for uptime and bad for ever noticing, unless something is actually watching those logs.
- "Reconstruct the expected value and spot-check it" is not the same audit as "query the live system for every case." The first checks your mental model. The second checks reality. They can disagree, and when they do, reality wins.
- Every so often, a system this old deserves a live audit, not just a code review specifically to surface the assumptions that were true once, got baked in early, and nobody has had a reason to re-question since.
Top comments (0)