DEV Community: Hancic

Cache Invalidation: Three Patterns and the Cost of Getting It Wrong

Hancic — Fri, 18 Sep 2026 13:42:16 +0000

A version-control service I worked on kept a small list in Redis: which version of the code bundle was sitting on the shared NFS volume. Every request checked the list first. Hit meant the file was local and the executor could load it. Miss meant a download from object storage.

One morning the cleanup job ran. It deletes old versions when NFS fills up. It deleted v1.42 from NFS. The list still said v1.42 existed. The cache lied, and a lying cache is worse than a missing one. A miss sends you to the source of truth. A false hit tells you the file is there, the loader opens it, and you get an error three steps later with no clear culprit.

Picking Redis or Caffeine or Memcached was not the question. The question was how to keep that list aligned with NFS. Every cache, on every system, faces the same question. Three patterns answer it, and each one hands you a different bill.

Three patterns, three bills

TTL: the default that lies until it stops

Most teams pick TTL because it costs nothing to set up. Each key gets an expiration time. Read the key. If the time has passed, treat it as a miss and reload from the source. Done.

TTL cache flow from write to read to expiry

The good part is the simplicity. The bad part is the gap between expiration and the actual re-read. Anything that writes to the source before that re-read lands will be missed. Every miss costs a trip to the source, which is fine until the source is slow, and then the cache that was supposed to take load starts generating it.

Tunables: expiration time, refresh-ahead. Neither removes the window. Shorter expiration shrinks it. Longer expiration makes the cache useful again. Pick one bill and own it.

Event-driven: the cache stays honest, the event bus stays busy

The second pattern answers the lying problem directly. Instead of letting entries time out, you tell the cache when the source changed. Every write to the source emits an event. A consumer in the cache layer picks it up and invalidates or updates the key.

Event-driven invalidation races against writes

Two failure modes show up in practice.

First, missed events. The producer crashed, the network dropped the message, the consumer was down. The cache says the old value is still right. The window is smaller than TTL because it depends on the outage, but the window is never zero.

Second, out-of-order events. Two writes happened in quick succession. The first invalidation lands, then the second one lands before the first write has been read back into the cache. A reader who arrives between them sees nothing, reloads the first write, then the second invalidation hits and evicts it. Net effect: a clean cache, one wasted reload. If the order reverses, the cache holds the older value while the source has the newer one. That is the same lying cache, just for shorter.

Messages can be lost

The bill is the event bus. A queue with delivery guarantees, idempotent consumers, retries with backoff, dead-letter handling, ordering where it matters. Each of these is real engineering, and the system that owns the source has to wire it in. Event-driven caches are not free. They are cheap to write the first time and expensive to operate.

Write-through: the cache never holds a lie

The third pattern removes the lying window by removing the path that creates it. The cache sits on the write path. Every write goes to the cache and the source. Reads go to the cache. There is no separate invalidation step because there is no separate write.

Write-through keeps cache and source in step

The tradeoff moves up the stack. Writes get slower because they pay two round-trips, and the cache becomes part of the critical path. If the cache layer fails, writes fail. The source has to be the system of record that recovers state on restart.

Pick one, or pay three bills

This works well when reads dominate writes by a wide margin and the cache layer can match the source's durability story. It works badly when the source and the cache disagree on failure modes, because now you have two systems that both believe they own the truth.

Pick one and own its failure mode

The three patterns answer the same question with three different bills.

TTL gives you a cache that lies until the timer expires. You own the gap.
Event-driven gives you a cache that stays honest when the events flow. You own the event bus.
Write-through gives you a cache that never holds a lie. You own the write path.

There is no pattern that has none of these bills. The list of caches in your system is the list of failure modes you have agreed to operate.

When the lying cache shows up in an incident report, look at which pattern the cache is using. The failure mode is not a surprise. It is the bill.

Where Distributed Systems Complexity Comes From

Hancic — Fri, 11 Sep 2026 13:54:52 +0000

Three machines in a rack are still three machines. They become a distributed system when they have to finish one job together, and that requirement is where the trouble starts.

On a single machine you get three things for free. Every thread shares the same memory, so a write lands and everyone sees it. One clock puts every event in order. One lock protects the shared state.

Move that job across a network and all three disappear.

Everything hard about distributed systems comes from three root problems: space, time, and consensus. Space shows up first and you notice it. Time is quieter, and you misread it. Consensus is what you get once the first two pile up.

Space and time become consensus

Space: the data lives in more than one place

Sharing a variable inside one process costs nothing. Run the same code on two machines and each node holds a piece of the truth. No node holds all of it.

Replication: who gets to write

Let every node write and two of them will accept conflicting writes for the same key. Neither has heard from the other yet, so both believe they are right. Now you need a rule for that conflict.

Restrict writes to one leader and you inherit a different set of problems. Then the leader dies, and you own the seconds that follow. You also have to decide which reads may go to a follower that lags behind.

Single-leader, multi-leader, and leaderless replication are three answers to the write question. Each one leaves you a different set of failures.

Who may write

Partitioning: where data goes

The dataset outgrows one disk. Split by key range and the ranges go out of balance as traffic shifts. Split by hash and each node gets a similar share of writes. Range scans get slower, and a query that needs rows from two partitions pays a network hop.

Range versus hash

The two problems multiply. One node can lead one partition and follow another, so the replication policy has to match the partition layout. The topology you draw on a whiteboard becomes a set of rules every node has to agree on.

Time: order stops being obvious

Code on one machine gives you a total order. The line a = 1 finishes before the line b = a + 1 reads it, because the hardware guarantees that.

Across machines there's no global clock. Two machines drift apart and NTP holds them within milliseconds, which is slow when you handle thousands of operations per second. The network reorders messages on top of that. You send A, then B, and the receiver gets B first.

Read a timestamp and you still can't say which event happened first.

Clocks that count events

Drop wall-clock time and ask a smaller question instead. Compare two events: did one cause the other?

If A causes B, then A is earlier. If neither causes the other, the two ran concurrently. A logical clock records that relation and gives you a partial order.

One counter carries one number, so it can't separate concurrency from causation. Vector clocks fix that. Each node keeps one counter per node. Comparing two vectors tells you whether one event caused the other, or whether the two just happened in parallel.

Causal, or concurrent

The price grows with the cluster. A 500-node system attaches 500 counters to every message, and trimming old entries means accepting wrong answers.

The consistency spectrum

Ordering uncertainty is why you choose from a spectrum of consistency models.

Linearizability makes every operation look like it took effect at one instant on one machine. Sequential consistency drops the real-time requirement and keeps one global order. Causal consistency orders causes before effects and ignores the rest. Eventual consistency promises that replicas converge once writes stop, and stays quiet about when.

Strong to weak

Each step down that ladder buys you latency and availability, and costs you a rule you can reason about.

Consensus: agreement is the hard part

Stack space on top of time and you get the problem distributed systems are famous for.

You run into consensus in a lot of disguises. A cluster elects a leader. A transaction spans two databases and has to commit on both or neither. Two clients want the same row locked.

Every one of those reduces to the same requirement. A group of nodes that don't trust each other, talking over a network that drops and delays messages, has to agree on one value.

The FLP result

In an asynchronous system, if one node can crash, no deterministic protocol can guarantee consensus.

The word that matters is deterministic. The theorem covers protocols that never give a wrong answer. Protocols that get it right with high probability sit outside it, and that gap is where Paxos and Raft live. Both use a quorum and randomized timeouts, so they finish in practice without ever being certain in theory.

Paxos, Raft, and ZAB

Paxos is correct and hard to implement. Raft splits the same problem into leader election, log replication, and safety, which makes the protocol teachable. ZAB drives ZooKeeper and takes a similar route with ordering guarantees on top.

All three need a majority to get anything done. A five-node cluster survives two failures. Split it 3 to 2 and the minority side can't serve writes.

Majority writes, minority waits

Distributed transactions

Two-phase commit asks every participant to prepare, then tells everyone to commit. That window between the two phases is the weak point. If the coordinator dies after the prepares, participants sit there holding locks.

Prepare, vote, commit

Three-phase commit adds a timeout step that shrinks the window without closing it. Saga and TCC commit each step and define a compensating action for it, which trades isolation for availability. One step fails and you spend the following week untangling a state that no single snapshot explains.

The three problems stack

Space, time, and consensus don't sit side by side. Space is the one you notice, because a node hands back stale data and the gap shows up in a request.

Time is harder to catch. The bugs it causes look like space bugs until you check clock skew or measure how far behind a follower is.

Consensus needs both before it bites. Agreement only gets hard once the data lives in several places and the messages arrive in an order you can't predict.

That explains the size of the toolbox. Vector clocks, quorums, consistent hashing, leader election, compensation protocols. Different angles on one question: several machines have to finish one job, and each machine sees only its own local state.

Next time a design review stalls on gossip versus Raft, or on two-phase commit versus Saga, name the problem you are paying for.

With replication you decide who writes. With partitioning you decide where data lives. With time you decide whether ordering matters. With consensus you decide what the group accepts.