Why Timestamps Lie in Distributed Systems (and How Logical Clocks Fix It)

There's a line of code in almost every service that stores replicated data:

def resolve(incoming, stored):
    # keep whichever write is "newer"
    return incoming if incoming.timestamp > stored.timestamp else stored

It passes review. It passes tests. It behaves perfectly on your laptop, in CI, and in staging. Then a customer reports that a profile change they definitely saved has reverted, and there's no error anywhere in the logs. The write arrived, was processed, and was persisted. Then it was discarded on purpose by the function above.

Here's what happened. incoming.timestamp and stored.timestamp were generated on two different machines, and one machine's clock was about 90 ms ahead of the other. The write that happened later in real time carried the smaller timestamp, so "keep the newer one" kept the older data. That's last-write-wins (LWW), and it's the most common way clock skew silently corrupts a database.

The same root cause shows up in disguises: a cache entry that "expires" a moment before it was written, a distributed trace that shows a response arriving before its request, a rate limiter that resets mid-window when a node's clock steps backward, a dedupe window that drops a legitimate event. Different symptoms, one assumption underneath all of them — that timestamps from different machines can be compared. They can't, at least not at the precision this kind of logic needs.

This is about what to use instead: the happened-before relation, Lamport clocks, vector clocks, and the hybrid clocks that modern databases actually run on. None of it is exotic. Most of it is already inside systems you use every day.

Why two clocks disagree, and why it's worst when it matters

"Clocks drift" is the folklore. The specifics are what tell you how much to distrust them.

A commodity quartz oscillator drifts on the order of tens of parts per million. One ppm is one microsecond per second, so tens of ppm works out to roughly a second of error per day if nothing corrects it. NTP exists to correct it, and on a healthy network it keeps machines within a few milliseconds of a reference clock. The trouble is the word "healthy."

NTP correction is not smooth. After a network partition, a VM migration, or a long pause, NTP can step the clock — jump it forward, or backward. The backward step is the dangerous one. Any code computing t2 - t1 on a single machine can suddenly see a negative duration. Any code comparing timestamps across machines can watch events reorder.

Virtualization makes it worse. VMs pause and resume with stale clocks, containers inherit whatever the host is doing, and laptops sleep. Cloud instances are generally harder to keep in tight sync than bare metal unless you opt into the provider's dedicated time service.

The part that turns "annoying" into "dangerous" is the correlation: clocks disagree the most during partitions, failovers, and overload — which is exactly when conflict-resolution code runs hardest. The clock is least trustworthy at the precise moment your logic leans on it most. That correlation, not the average-case drift, is the real argument against trusting cross-machine timestamp comparisons in a correctness path.

There is one clock you can trust, with a caveat. A monotonic clock never runs backward:

import time

start = time.monotonic()
run_the_work()
elapsed = time.monotonic() - start   # always >= 0

Use monotonic clocks (time.monotonic(), System.nanoTime(), CLOCK_MONOTONIC) for every duration and timeout. The caveat: a monotonic clock's zero point is arbitrary — usually "since this machine booted" — so its values are meaningless across machines and often across reboots. It measures elapsed time locally. It does not order events globally. For that, you need something else entirely.

The idea that fixes your mental model: happened-before

Leslie Lamport's 1978 paper made the point that in a distributed system, the meaningful order of events isn't temporal, it's causal. Event A happened-before event B if any of these hold:

• A and B are on the same process, and A came first locally.
• A is the sending of a message and B is the receipt of that message.
• There's a chain of the above linking A to B (the relation is transitive).

If no such chain exists in either direction, the two events are concurrent. This is the word engineers most often misuse. Concurrent does not mean "at the same instant." It means the system holds no evidence that one came before the other — and therefore no correct algorithm may assume an order between them.

Physical time has nothing to do with it. A message sent from Tokyo can carry a wall-clock time later than an event in Virginia and still happen-before it, if the clocks are skewed. What your invariants care about is the causal chain, not the wall clock. A user who reads a profile and then edits it has created a causal chain: the edit happened-after the read. Two users editing the same record from different continents, with no communication between them, are genuinely concurrent — and any order you put on those two writes is a decision you're making, not a fact you discovered.

Everything below is machinery for tracking that relation, or for honestly admitting when there's no order to track.

Lamport clocks: cheap order that respects causality

The smallest useful mechanism is a single integer per process:

class LamportClock:
    def __init__(self):
        self.t = 0

    def local_event(self):
        self.t += 1
        return self.t

    def send(self):
        self.t += 1
        return self.t          # attach this value to the outgoing message

    def receive(self, msg_t):
        self.t = max(self.t, msg_t) + 1
        return self.t

The rule on receive is the whole trick: take the max of your counter and the sender's stamp, then add one. That guarantees a receiver's clock always exceeds the sender's stamp, which gives you the core property:

If A happened-before B, then L(A) < L(B).

Causality is now embedded in the numbers. Break ties (equal counters) with a process ID, and every node agrees on the same total order without reading a wall clock even once. That's enough for ordering operations in a replicated log, for a fair queue behind a distributed lock, or for consistent tie-breaking.

Here's the part people get wrong, and it's worth saying slowly because it's both a classic interview trap and a real bug generator: the implication only runs one way. L(A) < L(B) does not mean A happened-before B. The two events might be concurrent, with one process simply having counted higher. Lamport clocks respect causality; they can't detect concurrency. When you need to know whether two events were concurrent — because concurrent means conflict, and conflict means someone has to merge — one integer isn't enough.

Vector clocks: detecting concurrency, not just respecting it

Give each of N processes a vector of N counters — its view of everyone's progress. On a local event, increment your own slot. On send, attach the whole vector. On receive, take an element-wise max with the incoming vector, then increment your own slot.

Now comparison has three outcomes instead of two:

def compare(a, b):
    # a, b are vector clocks: dict of node_id -> counter
    nodes = set(a) | set(b)
    a_le_b = all(a.get(n, 0) <= b.get(n, 0) for n in nodes)
    b_le_a = all(b.get(n, 0) <= a.get(n, 0) for n in nodes)

    if a_le_b and b_le_a:
        return "equal"
    if a_le_b:
        return "a -> b"        # a happened-before b
    if b_le_a:
        return "b -> a"        # b happened-before a
    return "concurrent"        # neither dominates: a real conflict

That last branch is the entire upgrade. When neither vector is less-than-or-equal to the other across all slots, the events are provably concurrent. Vector clocks are causality detectors.

This is the machinery behind conflict detection in Dynamo-style stores. Replicas accept writes independently — availability over coordination — and when two versions of an object meet, vector comparison decides whether one version supersedes the other (discard the ancestor) or whether they're concurrent (keep both as siblings and hand them to the application to merge). The classic example is a shopping cart, where the merge is "take the union of both carts." The deeper point is that only vector-style machinery can even ask whether two writes conflict. An LWW store answers that question by silently keeping one and dropping the other — which is the profile-loss bug from the top of this article, promoted to a default setting.

Vector clocks aren't free. The vector grows with the number of writers, so N counters per object becomes real overhead when writers are numerous or membership churns. And detecting a conflict isn't resolving it — the merge logic is still yours to write, and "surface siblings to the application" is a genuine API burden. Plenty of teams try it, feel the weight, and deliberately trade back to LWW. That trade is fine when it's chosen, for data where losing a concurrent write is acceptable. The bug is never LWW itself. The bug is LWW by default on data that can't tolerate it.

Hybrid logical clocks: readable numbers that still respect causality

Pure logical clocks have a practical problem: their numbers mean nothing to a human or an external system. You can't ask "what was the state around 14:32?" and you can't garbage-collect everything "older than a week," because the counters don't map to time. Pure physical clocks have the opposite problem — they're readable, but they lie about causality.

Hybrid logical clocks (HLC) braid the two. An HLC timestamp stays close to physical time, so it's human-readable and usable for TTLs, but it updates with a Lamport-style max-and-increment rule, so causality is never violated even when the physical clocks skew. If a message arrives from a node whose clock runs ahead, the receiver's HLC jumps forward past it — the logical component absorbs the skew — instead of letting causality invert. The one-line model: an HLC is a wall clock that refuses to contradict causality. It was introduced in a 2014 paper by Kulkarni and colleagues, and it's what CockroachDB, YugabyteDB, and MongoDB's cluster time run on.

The high-end alternative is worth knowing as a contrast. Google's Spanner uses TrueTime: GPS receivers and atomic clocks in each datacenter that expose time as a bounded uncertainty interval rather than a single value. Spanner buys strong consistency by having a transaction wait out that uncertainty before committing — usually a few milliseconds. It's the exception that proves the rule. Even with the best clocks money can buy, correctness comes from explicitly modeling how wrong the clock might be, not from trusting the number it reports. Everyone without atomic clocks in the rack reaches for HLC-style compromises instead.

What real systems actually do

This machinery is hiding in tools you already run. Knowing which strategy each one uses tells you where the sharp edges are.

System	How it orders	What to remember
Kafka	Per-partition offset, no clocks	Order holds within a partition, never across. Partition-key choice is an ordering decision.
Postgres / most SQL replication	WAL log sequence numbers, single writer	A single writer dissolves the problem. Multi-writer setups are where clocks come back.
Cassandra / ScyllaDB	LWW on write timestamps, by default	Carries every skew risk here. Mitigate with client-side timestamps and tight NTP — and know this before storing must-not-lose data.
Riak / the Dynamo lineage	Vector-clock-style causality, siblings	Detects concurrent writes and hands them back for you to merge.
CockroachDB / YugabyteDB / Mongo cluster time	Hybrid logical clocks	Causality-safe ordering with human-readable timestamps.
Tracing (spans)	Parent/child causal links	Traces stay coherent across skewed hosts because they order by causality, not timestamps. Raw multi-host log interleaving doesn't.

That last row is worth internalizing for incident response. When interleaved logs from two hosts "prove" that a cache responded before the request arrived, that's clock skew, not time travel. Order by trace links, not by timestamps.

A framework for choosing

You don't need to memorize the tools. You need to ask the right questions of your design, in order:

1. Can you avoid multi-writer ordering entirely? One partition owner, one leader, one sequencer turns the whole problem into a local counter. It's the strongest and cheapest answer, and a surprising amount of good architecture is quietly this move.
2. Do you need total order, or just causal consistency? Total order over everything is consensus territory, and it's expensive. Many systems only need "effects follow their causes," which HLCs and causal broadcast give you far more cheaply.
3. When two writes conflict, who merges? If the application can merge — carts, sets, counters, collaborative text — use vector or CRDT machinery and surface or auto-merge. If one write should simply win by business rule, use LWW, chosen on purpose, for data whose loss you've explicitly accepted.
4. Does anything outside the system read your order? TTLs, humans, and cross-system joins all pull you toward HLC-style physically-meaningful timestamps rather than opaque counters.
5. What breaks if the clock steps backward right now? Ask this of every timestamp comparison in the design. It takes ten minutes and it finds the LWW data-loss bug before a customer does.

A quick word on CRDTs, since question 3 points at them. Conflict-free replicated data types are the "stop fighting the ordering problem" option: data structures whose merge is commutative, associative, and idempotent, so concurrent updates converge no matter what order or how many times they're applied. They trade expressiveness — not everything fits a CRDT — for eliminating both the ordering machinery and the merge burden. For counters, sets, flags, and collaborative documents, that's an excellent trade.

The mistakes that actually bite

• LWW by default on data you can't lose. The config is one line, the data loss is silent, and the postmortem is long.
• Cross-machine timestamp math in a correctness path. Rate limits, cache expiry, dedupe windows, and "is this newer?" checks built on cross-host comparison inherit unbounded error at the worst possible time.
• Assuming local time only moves forward. Wall-clock APIs step backward. Use monotonic clocks for durations and timeouts.
• Reading a Lamport comparison backward. L(A) < L(B) does not establish that A caused B. One-way implication only.
• Vector clocks keyed by an unbounded actor set. A per-user entry on a public API grows without limit. Key vectors per replica, not per client, or prune deliberately.

The habit that replaces all of this

The tools matter less than the reflex. Somewhere in every distributed-systems education — sometimes in a lecture, more often in a postmortem — a timestamp stops being infrastructure and becomes a claim that needs evidence. The log says 14:32:07.190. Says which clock? Synchronized to what? Stepped when?

Once you start asking that, you can't stop, and the timestamps scattered through your correctness logic start to look like what they are: unverified reports from sources known to be unreliable under load. The replacement toolkit is small — causal order as the ground truth, one counter when respecting it is enough, a vector when detecting concurrency matters, a hybrid when humans need to read the numbers, and structural order whenever you can design the problem away. The real upgrade isn't in the code. It's the question you now ask on reflex: what does this system actually know about what happened before what?