We once spent a genuinely baffling week chasing a bug that did not exist. Customers reported that a pricing change was not taking effect, so we checked the database, and the new price was right there, correct and committed. We checked the application code, and it was reading the right field. Everything we looked at said the system was doing exactly what it should. Meanwhile customers kept seeing the old number. The truth, when we finally found it, was a caching layer nobody had thought about in months, cheerfully serving a value it had memorized before the change and never been told to forget.
Caching is one of those ideas that is beautiful in the abstract and treacherous in practice. You store the answer to an expensive question so you do not have to ask it again, and your system gets dramatically faster. But you have also created a second copy of the truth, and now you have two things that must agree: the real data, and the remembered data. The moment the real data changes and the memory does not get updated, your system starts lying with total confidence, and it lies fastest of all, because that is the whole point of the cache.
The hard part was never adding the cache. The hard part is invalidation, knowing precisely when the remembered answer has gone stale and must be thrown away. Get that wrong in one direction and you clear too aggressively, losing all the speed you wanted. Get it wrong in the other and you serve outdated data to people making real decisions, with no error, no warning, and no obvious place to look. The database is right. The code is right. The stale copy sitting quietly in between is the one telling the lie.
Since that week I treat every cache as a deliberate commitment, not a free optimization I sprinkle on for speed. Before I add one, I ask exactly when its contents become wrong and exactly what event will tell it so. If I cannot answer that clearly, I do not yet understand the cache well enough to add it, because a cache without a clear invalidation story is not a performance feature. It is a stale-data incident scheduled for some unlucky future afternoon.
A cache is a promise that two copies of the truth will stay in sync. Speed is what you get when the promise holds. A week of chasing a bug that was never in your code is what you get when it quietly breaks.
– Serguey Shinder
Top comments (0)