Support pasted a screenshot at 9:40am: a customer's invoice page showed a balance of $0.00, and a second tab showed $412.60. Same account, same minute. No deploy that morning. The database was right. Our cache was wrong, and nothing in the system was responsible for telling it so.
We had a TTL. That is the part that stings. Every key expired in five minutes, which felt careful at the time, and for two years it was fine because nobody reconciled two tabs at once.
A TTL is a bet, not a correctness mechanism
A TTL says: I believe this answer stays true for N seconds. That is a guess about your write patterns, disguised as a config value. It has no idea that invoices.balance.8842 was just invalidated by a payment webhook. When a TTL-driven cache is wrong, it is not broken — it is behaving exactly as designed and answering a question about the past.
The honest framing: a cache is a stored copy of a decision you already made. The database decided the balance was $412.60. Our cache held a copy of the decision that said $0.00, made before the payment landed. The only interesting question is which writes make that copy wrong, and which code paths know about those writes.
If you want to know how much staleness you actually tolerate, measure it. Write the timestamp of the cached write into the value, and log now - write_time on every read of a key that later turns out stale. That distribution tells you what your TTL is really costing. Do not guess the number; sample it.
The write path is the only thing that knows
Reads cannot invalidate. A read has no idea whether another writer is mid-transaction. The write path — the one place that changes the underlying decision — is the only code that knows the old answer is now garbage.
We ran cache-aside: the reader did GET, missed, queried Postgres, then SETEX. It looked clean. The bug was ownership. Nobody owned invoices.balance.{id}. The reader created it. The payment service, which actually changed the balance, had never heard of the key. So the payment service wrote to Postgres and the cache kept serving its stale copy until the TTL swept it away.
Write-through inverts that. The writer updates the store and the cache in one place, so the key has an owner:
def record_payment(account_id, amount, conn):
with conn.transaction():
conn.execute(
"UPDATE invoices SET balance = balance - %s WHERE account_id = %s",
(amount, account_id),
)
row = conn.execute(
"SELECT balance FROM invoices WHERE account_id = %s", (account_id,)
).fetchone()
# The writer owns the key. Reads never create it.
redis.set(f"invoices.balance.{account_id}", row["balance"], ex=300)
redis.publish("invoice.changed", str(account_id))
The ex=300 is now a safety net for a missed write, not the correctness story. That is the difference. When invalidation is owned by the writer, the TTL becomes a backstop you hope never fires.
Be careful with delete-on-write too. DEL then re-populate races: a reader can miss, read the pre-commit row, and SETEX the old value back on top of your fresh write. Delete outside the transaction and re-populate from the committed row, or version the key.
Expiry is a coordinated attack on your database
The other failure nobody plans for: everyone's key expires at the same moment. A popular key with a 300-second TTL, written at 10:00:00 during a batch job, expires at 10:05:00 for every reader at once. All of them miss, all of them hit Postgres, and the database that was comfortably handling 200 reads per second suddenly takes the full load.
Three things helped. Jitter the TTL so keys written together do not die together. Collapse concurrent misses onto a single in-flight fetch so one request rebuilds and the rest wait on it. And for the hottest keys, refresh in the background before expiry so the miss never happens in the request path. Test this deliberately: expire your top key by hand during peak traffic and watch the database.
Make invalidation an event, not a cron
The fix that finally held was deleting the sweeper job. We had a cron that walked recent invoices and refreshed their cache entries every five minutes. It was a second source of truth about when data changed, and it was always slightly behind the writes it was chasing.
Instead, the payment service publishes invoice.changed with the account id. A subscriber deletes the balance key. The domain already had a name for this moment — a payment was recorded — so the event existed before the cache did. The cache became a consumer of a fact the business cared about, not a timer with an opinion.
That reframing is the whole job. Stop asking what TTL to pick. Ask which writes make this answer wrong, and who in your code knows they happened. If the answer is "a cron job, eventually," you have a staleness budget, not a cache.
I write about production failures in Postgres, queues, and distributed systems.
Top comments (1)
If the process dies after the database commit but before
redis.setorpublish, the old value remains until the TTL expires. Does the payment event use an outbox so that crash window cannot lose the invalidation?