A support ticket came in that made no sense at first: a customer swore they'd briefly seen someone else's search results. Not an error, not a crash — just, for a few seconds, data that wasn't theirs. Then it "fixed itself" on a page refresh, and they moved on with their day more confused than angry. We should have been a lot more alarmed than we were.
We had a shared cache layer in front of a search endpoint, keyed on the query string and the account's pricing tier. That had worked fine for a long time, because in practice most cache keys collided safely — two customers on the same tier running a similarly-shaped query usually got results that happened to match, or close enough that nobody noticed. The cache key didn't include the dimension that actually mattered: the account's specific data scope. Two different customers could produce the exact same cache key while expecting to see completely different result sets, and whichever one populated the cache first would end up serving the other one's request for the life of that cache entry.
The bug had probably existed since the caching layer was first added. It just needed two things to line up to become visible: two customers on the same tier, with overlapping-enough queries, whose underlying data had diverged enough that a shared result would actually look wrong to one of them. Early on, with few customers and sparse data, the odds of that were low. As the customer base grew, the odds went up, and the ticket we got was really just the first time we got unlucky enough for someone to notice.
The fix, once we found it, was almost anticlimactic: audit every parameter that could change what the "correct" response looks like for a given request, and make sure all of them are part of the cache key — not just the ones that seemed obviously relevant when the cache was first built. We ended up adding an automated check as much as a code fix: a test that generates two different account contexts with the same visible query parameters and asserts that they cannot produce the same cache key. That test is the thing that actually prevents this class of bug from coming back, because it doesn't rely on a person remembering to think about it during a future refactor.
What made this bug uncomfortable wasn't the complexity of the fix, it was the shape of the failure. This wasn't a performance bug or a UI glitch — it was two customers' data touching in a way that neither of them chose and neither of them could have detected reliably from their side. That's the kind of bug that's cheap to prevent up front and expensive to explain after the fact, both in engineering time and in the harder-to-quantify cost of a customer wondering what else they can't see.
The broader lesson we took from it: a cache key isn't just a performance knob, it's a security boundary as soon as any two callers can share a cache entry non-consensually. Every time we design a new cache layer now, "what could two different callers legitimately have in common, and what must always keep them apart" is a question we answer explicitly before the cache goes live, not something we patch in after a ticket forces the question.
This is the kind of correctness work that's easy to skip when a system is small and obvious in the moment, and expensive to retrofit later — it's part of why we treat this stuff as a first-class concern in how we build backend systems at Edilec, more on that at edilec.com.
Top comments (0)