The cache that served one user's account to another

Most of my bugs cost time or money. This one nearly cost me something I couldn't get back: a user seeing another user's private data. I want to tell it plainly, because the mistake is so small and so common that if you run a cache — and you do — you have probably written it and gotten lucky.

The optimization that felt free

We had an endpoint that was slow. It assembled a user's account summary — profile, recent activity, a couple of joins that got heavier as the data grew — and it was getting hammered on every page load. The obvious fix, the one anybody would reach for, was a cache. Compute the summary once, stash it, serve the stashed copy for a minute. Textbook. It dropped the endpoint's latency through the floor and everyone was happy, including me, especially me, because I'd made a number go down in an afternoon.

Here is the cache key I used, more or less:

key = "account_summary:" + endpoint_version
value = the rendered summary

Read that and feel where your stomach drops. The key identifies what is cached — the account summary from this endpoint. It does not identify whose. There is no user ID in it. Every request for an account summary hashed to the same slot.

In my defense — and it's a weak defense, which is the point — it didn't look wrong. In development I was always logged in as one test user, so the cache always held my data and always served me my data. It worked perfectly. It passed review, because the reviewer was also picturing one user at a time, the way you do when you read code. The bug is invisible until two different people are in the system at the same moment, and code review is a fundamentally single-user activity. You read it as one story with one protagonist. Concurrency is a different genre.

Thirty minutes I'd like back

It surfaced as a support ticket, and the wording is burned into me: "Why am I looking at someone else's account?" A user had loaded the page and seen a name that wasn't theirs, activity that wasn't theirs. My first read was that it had to be a mistake, a confusion, a screenshot from the wrong tab. Then a second ticket. Then I understood, all at once, the way you understand you've left the stove on.

What was happening was exactly what the key made inevitable: user A hit the endpoint, their summary got computed and stored under the shared key. A beat later, user B hit the same endpoint, the cache had a fresh entry, so it skipped the computation and handed B the stored copy — A's data. Whoever populated the slot first, within that minute, was showing their private account to everyone who came after. Under low traffic it almost never lined up. Under real traffic it lined up constantly. The faster and more popular the endpoint got, the more it leaked, which is a sentence that still makes me a little sick.

I killed the cache in production first — just turned it off, ate the latency, stopped the bleeding before I did anything clever. That's the one instinct I'm glad I had: when you're leaking data, you don't debug the leak, you close the valve and debug it closed. Slow and correct beats fast and exposing every single time.

The fix was one word

key = "account_summary:" + user_id + ":" + endpoint_version

That's it. Put the identity in the key. Now A's summary and B's summary live in different slots and can never be handed to the wrong person, because the cache can't confuse two things it's storing separately. One missing field — user_id — was the whole bug. The fix was smaller than the ticket describing it.

But the small fix is not the lesson. The lesson is what the small fix taught me about the category.

A cache key is not a performance detail. It is a claim about identity — it says "anyone who asks this exact question may be given this exact answer." If your key doesn't encode every dimension that makes an answer specific to someone — the user, the tenant, the permission scope, the locale, whatever narrows "the answer" to "their answer" — then you have told the cache that those things don't matter, and the cache will believe you, and it will hand the wrong answer to the wrong person the instant two of them disagree. Caching doesn't add a data leak on its own. Caching plus an under-specified key adds a data leak. The key is the whole safety property.

More generally: an optimization that "feels free" almost never is. What the cache actually did was introduce shared mutable state into a request path that used to be cleanly per-user. Every request computed its own summary before; now they reached into a common box. Shared mutable state is precisely where the frightening bugs live — the races, the leaks, the heisenbugs — and I'd added a big pile of it while thinking I was just making a number go down. The latency win was real. It came with a new failure mode I hadn't priced in, and the failure mode was "show strangers each other's accounts."

The part I couldn't fully undo

Here's the honest tail. The code fix was an afternoon; living with what had already happened was not. Because the leak had run for a while before the tickets, I could not claim with certainty that no data had been exposed — the whole nature of the bug is that it served real data to real people silently. So I did the unglamorous, correct thing: assumed the worst plausible exposure, wrote it up honestly for the people whose job it is to make those calls, and let them decide on disclosure. I don't get to hand-wave that part. When you're not sure whether you leaked, you treat it as though you did.

I never found out if anyone truly saw something they shouldn't have. That uncertainty is its own small penalty, and I've decided it's a fair one, because it's the exact price of having shipped shared state without thinking about who it was shared between.

What I do now, every time

I have one rule that came directly out of this, and it's cheap: before I cache anything, I write down who the answer belongs to, and every one of those things goes in the key. If I can't name who owns the answer, that's the signal that the thing isn't safe to cache yet — not that I should cache it and hope.

And the meta-lesson, the one I trust more than any single rule: when you add a cache, a queue, a pool, any shared thing, you have not just made the system faster. You've changed how many people touch the same state at once. Ask what happens when two of them touch it in the same instant, because in production, two of them will — and the cache will do exactly, precisely, what your key told it to.