DEV Community: Hancic

The Crash Window: Why I Picked Double-Counting Over Data Loss

Hancic — Sat, 26 Sep 2026 12:23:30 +0000

A service I worked on processed inventory messages. Millions per minute. Each message carried a list of SKU changes. The service filtered, accumulated counts, and wrote them to a distributed cache.

The message queue guaranteed at-least-once delivery. That guarantee is the problem. At-least-once means the queue redelivers a message if it thinks you did not process it. Crash during processing, ack timeout, partition rebalance. All trigger redelivery. Duplicates are not exceptional. They are the baseline.

The counts are accumulators. You add to them. Process the same message twice and the count goes too high. Wrong for 30 days, the TTL of the cache entry.

Our redelivery rate was about 0.1 percent. At millions of messages per minute, that is thousands of duplicates. Each one adds a phantom count that sticks for a month. Without deduplication, the numbers drift far enough to corrupt downstream decisions within hours.

Two operations, one window

The idempotency design splits message processing into two steps. Before handling a message, check whether a key exists in the cache. If it does, skip. If it does not, process the message, write the accumulated count, then set the key with a TTL of 10 minutes. Next time the queue redelivers, the key is there. Skip.

Writing the count and setting the key are two separate cache operations. Between them is a window. If the process crashes in that window, the service has written the count but the key is missing. The queue redelivers. The pre-check finds no key. The service processes the message again. The count goes up by one extra.

I call this the crash window. It is a few milliseconds wide. It cannot close.

The crash window between write and set key

Distributed caches do not support cross-key transactions. You cannot atomically write the count and set the idempotency key in a single operation. You could introduce an external transaction coordinator, or a distributed lock, or a two-phase commit. Each adds a new component that can fail, and each new component fails in ways that are harder to observe than a double-count.

Which failure to live with

I considered reversing the order. Set the key first, then write the count. If the process crashes between them, the key exists but the count is missing. The redelivery finds the key, skips, and the service skips the write. You lose the data.

Double-counting and data loss are both failures. They differ in one way that matters. I can see double-counting. I cannot see data loss.

Double-counting produces a number that is too high. I can compare the total against the source. I can estimate the deviation from the known redelivery rate and the crash recovery time. I can alert on it.

Data loss produces nothing. The count is too low, but there is no signal. The missing data does not trigger an alert. It does not show up in any dashboard. The downstream business discovers it days later when the numbers do not match, and by then the trail is cold.

I write the count first, then set the key. If I crash between them, I double-count. I accept that deviation because I can measure it, bound it, and catch it. I cannot do any of those things with data loss.

Two orderings, two failure modes

Bounding instead of closing

The idempotency key has a TTL of 10 minutes. Crash recovery on our infrastructure takes 1 to 3 minutes. If the process comes back within the TTL, the key is still there. The redelivery hits the pre-check, finds the key, and skips. No double-count.

The real risk is a crash where recovery takes longer than 10 minutes. This happens when the container scheduler cannot find a healthy node, or when the service mesh gets stuck on a bad configuration push. The key has expired, the pre-check misses, and the service processes the message again. The deviation is one count per message in that window.

I know the redelivery rate. I know the crash recovery distribution. I know the TTL. The product of those three numbers is the expected deviation per month. It is not zero. It is small, it is bounded, and I can tell you what it is.

I track three counters. Messages received. Idempotent keys hit, which means duplicates caught. Occupation failures after successful writes, which means the crash window fired. The third counter tells me how many times the window happened. If it spikes, something changed in the crash pattern. If it stays flat, the deviation is within the expected range.

Batch processing narrows the window. The service collects a batch of messages, aggregates them in memory, and submits one pipeline write to the cache. The crash window exists between that single write and the subsequent key occupation. A batch of 500 messages produces one window, not 500. If the batch fails, the queue redelivers the whole batch. The idempotency check catches it.

Batch narrows 500 windows to 1

The trade I made

At-least-once delivery plus accumulator semantics plus distributed cache gives you a choice. You can add transactions, locks, and external state stores to approach exactly-once. Or you can accept a known, measured deviation and keep the system simple enough to reason about.

I picked the second option. The deviation in my system is not zero. I know where it comes from, I know how large it can be, and I know when it fires. That is enough for the business. The complexity of closing the last few milliseconds was not worth the failure modes it would introduce.

Cache Invalidation: Three Patterns and the Cost of Getting It Wrong

Hancic — Fri, 18 Sep 2026 13:42:16 +0000

A version-control service I worked on kept a small list in Redis: which version of the code bundle was sitting on the shared NFS volume. Every request checked the list first. Hit meant the file was local and the executor could load it. Miss meant a download from object storage.

One morning the cleanup job ran. It deletes old versions when NFS fills up. It deleted v1.42 from NFS. The list still said v1.42 existed. The cache lied, and a lying cache is worse than a missing one. A miss sends you to the source of truth. A false hit tells you the file is there, the loader opens it, and you get an error three steps later with no clear culprit.

Picking Redis or Caffeine or Memcached was not the question. The question was how to keep that list aligned with NFS. Every cache, on every system, faces the same question. Three patterns answer it, and each one hands you a different bill.

Three patterns, three bills

TTL: the default that lies until it stops

Most teams pick TTL because it costs nothing to set up. Each key gets an expiration time. Read the key. If the time has passed, treat it as a miss and reload from the source. Done.

TTL cache flow from write to read to expiry

The good part is the simplicity. The bad part is the gap between expiration and the actual re-read. Anything that writes to the source before that re-read lands will be missed. Every miss costs a trip to the source, which is fine until the source is slow, and then the cache that was supposed to take load starts generating it.

Tunables: expiration time, refresh-ahead. Neither removes the window. Shorter expiration shrinks it. Longer expiration makes the cache useful again. Pick one bill and own it.

Event-driven: the cache stays honest, the event bus stays busy

The second pattern answers the lying problem directly. Instead of letting entries time out, you tell the cache when the source changed. Every write to the source emits an event. A consumer in the cache layer picks it up and invalidates or updates the key.

Event-driven invalidation races against writes

Two failure modes show up in practice.

First, missed events. The producer crashed, the network dropped the message, the consumer was down. The cache says the old value is still right. The window is smaller than TTL because it depends on the outage, but the window is never zero.

Second, out-of-order events. Two writes happened in quick succession. The first invalidation lands, then the second one lands before the first write has been read back into the cache. A reader who arrives between them sees nothing, reloads the first write, then the second invalidation hits and evicts it. Net effect: a clean cache, one wasted reload. If the order reverses, the cache holds the older value while the source has the newer one. That is the same lying cache, just for shorter.

Messages can be lost

The bill is the event bus. A queue with delivery guarantees, idempotent consumers, retries with backoff, dead-letter handling, ordering where it matters. Each of these is real engineering, and the system that owns the source has to wire it in. Event-driven caches are not free. They are cheap to write the first time and expensive to operate.

Write-through: the cache never holds a lie

The third pattern removes the lying window by removing the path that creates it. The cache sits on the write path. Every write goes to the cache and the source. Reads go to the cache. There is no separate invalidation step because there is no separate write.

Write-through keeps cache and source in step

The tradeoff moves up the stack. Writes get slower because they pay two round-trips, and the cache becomes part of the critical path. If the cache layer fails, writes fail. The source has to be the system of record that recovers state on restart.

Pick one, or pay three bills

This works well when reads dominate writes by a wide margin and the cache layer can match the source's durability story. It works badly when the source and the cache disagree on failure modes, because now you have two systems that both believe they own the truth.

Pick one and own its failure mode

The three patterns answer the same question with three different bills.

TTL gives you a cache that lies until the timer expires. You own the gap.
Event-driven gives you a cache that stays honest when the events flow. You own the event bus.
Write-through gives you a cache that never holds a lie. You own the write path.

There is no pattern that has none of these bills. The list of caches in your system is the list of failure modes you have agreed to operate.

When the lying cache shows up in an incident report, look at which pattern the cache is using. The failure mode is not a surprise. It is the bill.

Where Distributed Systems Complexity Comes From

Hancic — Fri, 11 Sep 2026 13:54:52 +0000

Three machines in a rack are still three machines. They become a distributed system when they have to finish one job together, and that requirement is where the trouble starts.

On a single machine you get three things for free. Every thread shares the same memory, so a write lands and everyone sees it. One clock puts every event in order. One lock protects the shared state.

Move that job across a network and all three disappear.

Everything hard about distributed systems comes from three root problems: space, time, and consensus. Space shows up first and you notice it. Time is quieter, and you misread it. Consensus is what you get once the first two pile up.

Space and time become consensus

Space: the data lives in more than one place

Sharing a variable inside one process costs nothing. Run the same code on two machines and each node holds a piece of the truth. No node holds all of it.

Replication: who gets to write

Let every node write and two of them will accept conflicting writes for the same key. Neither has heard from the other yet, so both believe they are right. Now you need a rule for that conflict.

Restrict writes to one leader and you inherit a different set of problems. Then the leader dies, and you own the seconds that follow. You also have to decide which reads may go to a follower that lags behind.

Single-leader, multi-leader, and leaderless replication are three answers to the write question. Each one leaves you a different set of failures.

Who may write

Partitioning: where data goes

The dataset outgrows one disk. Split by key range and the ranges go out of balance as traffic shifts. Split by hash and each node gets a similar share of writes. Range scans get slower, and a query that needs rows from two partitions pays a network hop.

Range versus hash

The two problems multiply. One node can lead one partition and follow another, so the replication policy has to match the partition layout. The topology you draw on a whiteboard becomes a set of rules every node has to agree on.

Time: order stops being obvious

Code on one machine gives you a total order. The line a = 1 finishes before the line b = a + 1 reads it, because the hardware guarantees that.

Across machines there's no global clock. Two machines drift apart and NTP holds them within milliseconds, which is slow when you handle thousands of operations per second. The network reorders messages on top of that. You send A, then B, and the receiver gets B first.

Read a timestamp and you still can't say which event happened first.

Clocks that count events

Drop wall-clock time and ask a smaller question instead. Compare two events: did one cause the other?

If A causes B, then A is earlier. If neither causes the other, the two ran concurrently. A logical clock records that relation and gives you a partial order.

One counter carries one number, so it can't separate concurrency from causation. Vector clocks fix that. Each node keeps one counter per node. Comparing two vectors tells you whether one event caused the other, or whether the two just happened in parallel.

Causal, or concurrent

The price grows with the cluster. A 500-node system attaches 500 counters to every message, and trimming old entries means accepting wrong answers.

The consistency spectrum

Ordering uncertainty is why you choose from a spectrum of consistency models.

Linearizability makes every operation look like it took effect at one instant on one machine. Sequential consistency drops the real-time requirement and keeps one global order. Causal consistency orders causes before effects and ignores the rest. Eventual consistency promises that replicas converge once writes stop, and stays quiet about when.

Strong to weak

Each step down that ladder buys you latency and availability, and costs you a rule you can reason about.

Consensus: agreement is the hard part

Stack space on top of time and you get the problem distributed systems are famous for.

You run into consensus in a lot of disguises. A cluster elects a leader. A transaction spans two databases and has to commit on both or neither. Two clients want the same row locked.

Every one of those reduces to the same requirement. A group of nodes that don't trust each other, talking over a network that drops and delays messages, has to agree on one value.

The FLP result

In an asynchronous system, if one node can crash, no deterministic protocol can guarantee consensus.

The word that matters is deterministic. The theorem covers protocols that never give a wrong answer. Protocols that get it right with high probability sit outside it, and that gap is where Paxos and Raft live. Both use a quorum and randomized timeouts, so they finish in practice without ever being certain in theory.

Paxos, Raft, and ZAB

Paxos is correct and hard to implement. Raft splits the same problem into leader election, log replication, and safety, which makes the protocol teachable. ZAB drives ZooKeeper and takes a similar route with ordering guarantees on top.

All three need a majority to get anything done. A five-node cluster survives two failures. Split it 3 to 2 and the minority side can't serve writes.

Majority writes, minority waits

Distributed transactions

Two-phase commit asks every participant to prepare, then tells everyone to commit. That window between the two phases is the weak point. If the coordinator dies after the prepares, participants sit there holding locks.

Prepare, vote, commit

Three-phase commit adds a timeout step that shrinks the window without closing it. Saga and TCC commit each step and define a compensating action for it, which trades isolation for availability. One step fails and you spend the following week untangling a state that no single snapshot explains.

The three problems stack

Space, time, and consensus don't sit side by side. Space is the one you notice, because a node hands back stale data and the gap shows up in a request.

Time is harder to catch. The bugs it causes look like space bugs until you check clock skew or measure how far behind a follower is.

Consensus needs both before it bites. Agreement only gets hard once the data lives in several places and the messages arrive in an order you can't predict.

That explains the size of the toolbox. Vector clocks, quorums, consistent hashing, leader election, compensation protocols. Different angles on one question: several machines have to finish one job, and each machine sees only its own local state.

Next time a design review stalls on gossip versus Raft, or on two-phase commit versus Saga, name the problem you are paying for.

With replication you decide who writes. With partitioning you decide where data lives. With time you decide whether ordering matters. With consensus you decide what the group accepts.