Why This Matters
In a distributed system — especially one spanning multiple regions and datacenters — there's no single global clock. A node in Virginia and a node in Frankfurt each have their own clock, and they drift by milliseconds or more. Yet we need two guarantees:
- Causal consistency — if event A caused event B, every node in every datacenter must see A before B. A user in Tokyo shouldn't see a reply before the original message, just because their nearest replica processed the reply first.
- Total ordering — even for unrelated events happening simultaneously in different regions, we need a deterministic way to put them in a single, agreed-upon order. Every node across every datacenter must agree, and they need to do it without a central coordinator — fully decentralized.
The tricky part? Neither Lamport clocks nor HLCs can actually detect concurrent events (Vector Clocks do that). But they don't need to — they solve a different problem. They assign every event a unique, sortable timestamp so you get a total order that respects causality, even across nodes in different continents that never talk to each other directly.
Lamport Clocks — Logical Order Only
A Lamport clock is just an integer counter. Two rules:
- Before any local event or sending a message, increment your counter.
- When receiving a message, set your counter to
max(yours, theirs) + 1.
Example: A node in Virginia (clock=1) sends a write to a node in Frankfurt (clock=3). Frankfurt sets its clock to max(3, 1) + 1 = 4. Now we know Frankfurt's event at 4 happened after Virginia's event at 1 — even though these nodes are 6,000 km apart and their wall clocks disagree.
Tie-breaking for total order: When two events across different datacenters have the same Lamport timestamp, they're concurrent — but we still need to order them. The standard trick is to append the node ID: (timestamp, node_id). So (5, virginia-1) always comes before (5, frankfurt-2) lexicographically. Arbitrary, but deterministic and consistent across all regions.
The catch: The timestamp "5" tells you nothing about wall-clock time. A user debugging a cross-region issue can't look at Lamport timestamps and say "this happened at 2:03 PM." You get ordering, not timing.
Hybrid Logical Clocks (HLC) — Best of Both Worlds
An HLC is a tuple: (physical_time, logical_counter, node_id).
It tracks causality like a Lamport clock and stays close to real wall-clock time — critical when your nodes are spread across regions with varying NTP sync quality.
How it works:
- On a local event: use
max(your_physical, wall_clock). If the physical part didn't advance, bump the logical counter. Otherwise reset it to 0. - On receiving a message from another region: take
max(your_physical, their_physical, wall_clock). If the max physical time didn't change, bump the logical counter. Otherwise reset to 0.
Example: A node in Virginia sends at physical time 10:00:01.000, HLC = (10:00:01.000, 0, virginia-1). A node in Singapore receives it, but Singapore's wall clock reads 10:00:00.500 (behind due to NTP drift). Instead of going backward, Singapore's HLC becomes (10:00:01.000, 1, singapore-1) — it takes the higher physical time and bumps the counter. Causality preserved, clock never goes backward, and the timestamp still reflects roughly when it happened.
HLC tie-breaking (three levels):
- Compare physical timestamps — the later wall-clock time wins.
- If physical times are equal, compare logical counters — higher counter means it saw more causal history at that timestamp.
- If both are equal, compare node IDs — deterministic, arbitrary, but guarantees a total order even for truly simultaneous events in different datacenters.
This three-part comparison means every event globally — across every region — gets a unique, sortable position. No ambiguity, no central coordinator needed.
Why production systems love this: CockroachDB and YugabyteDB use HLCs across their geo-distributed clusters because they need causal ordering without sacrificing the ability to reason about "when" something actually happened. When a user in London writes data replicated to nodes in São Paulo and Sydney, every replica agrees on the order — and an operator can still look at the timestamps and map them to real-world time.
Summary
| Feature | Lamport Clock | Hybrid Clock |
|---|---|---|
| Tracks causality? | Yes | Yes |
| Detects concurrency? | No (use Vector Clocks) | No (use Vector Clocks) |
| Reflects real time? | No | Approximately |
| Tie-breaker | Node ID | Physical time → counter → node ID |
| Total order? | Yes (with node ID) | Yes (built-in) |
| Centralized? | No | No |
| Works cross-region? | Yes, but no time context | Yes, with real-time context |
| Size | Integer + node ID | Timestamp + counter + node ID |
| Use case | Theory, simple systems | Production geo-distributed databases |
Lamport clocks tell you what depends on what. Hybrid clocks tell you that and roughly when. Neither detects concurrency — but both give you a fully decentralized total order that never violates causality, even across datacenters on opposite sides of the planet.
This post was originally published on:
Top comments (0)