Ask your two services what time it is.
They'll give you different answers.
Both will be wrong.
The Relativity Problem
In 1905, Einstein published special relativity. One of its core conclusions was uncomfortable:
There is no universal "now."
Two observers moving relative to each other will disagree about whether two events happened simultaneously. Not because of measurement error. Not because of imprecise clocks. Because simultaneity itself is relative.
Whether two things happened "at the same time" depends entirely on who's asking.
I kept thinking about this while debugging a distributed system.
Service A logs: "order submitted at 14:23:00.412"
Service B logs: "payment processed at 14:23:00.408"
Payment before submission. Four milliseconds of causality violation.
No hardware failure. No bug. Just clocks.
What NTP Actually Guarantees
Most distributed systems assume their clocks are synchronized. They use NTP. They trust the timestamps.
Here's what NTP actually gives you: clocks that are probably within 1–100 milliseconds of each other, under normal network conditions.
That's not "synchronized." That's "close enough for most things."
It is not close enough for:
- establishing event ordering across services
- audit logs that need to reconstruct causality
- distributed transactions
- anything where "which happened first" matters
Clock drift is real. Network delays are asymmetric. A server that was briefly partitioned and reconnected can have a clock that jumped forward or backward. NTP corrects this gradually — meaning for a window of time, your clock is knowingly wrong while it adjusts.
The assumption that "timestamp A < timestamp B" means "A happened before B" is false in distributed systems.
It's not an edge case. It's the default.
Einstein's Lesson: Define Your Reference Frame
In relativity, the fix isn't to find the "true" time. There is no universal reference frame. Instead, you define your frame explicitly, and work within it.
Statements like "A happened before B" are only meaningful when you specify: according to whom?
Distributed systems need the same discipline.
The question isn't "what time did this happen?" The question is:
"What is the causal relationship between these events?"
Those are different questions. And only the second one has a reliable answer.
Logical Clocks: Causality Without Clocks
Leslie Lamport solved this in 1978 with logical clocks.
The insight: you don't need wall clock time to establish ordering. You need causality.
Lamport timestamps assign a counter to each event. The rule is simple:
- Every event increments your local counter
- Every message carries the sender's counter
- On receipt, you take the maximum of your counter and the received one, then increment
The result: if A causally precedes B (A's message caused B to happen), then A's timestamp is guaranteed to be less than B's.
What it doesn't give you: if A's timestamp < B's timestamp, that doesn't mean A caused B. They might be concurrent — unrelated events that happened independently.
Vector clocks go further. Instead of a single counter, each service maintains a vector of counters, one per service. Now you can detect true concurrency: two events are concurrent if neither causally precedes the other.
This is how systems like DynamoDB, Riak, and CRDTs handle conflicting writes. Not by assuming clocks, but by tracking causality explicitly.
The Speed of Light Problem
In general relativity, nothing travels faster than light. Information takes time to propagate. Two observers far apart cannot instantly agree on a shared state.
In distributed systems, the equivalent is network latency. There is a speed-of-light limit on consensus.
This is what the CAP theorem is actually saying:
During a network partition, you cannot have both consistency and availability.
Choose consistency: all nodes agree on the current state, but you have to wait (or refuse requests) during partition.
Choose availability: every node responds, but they might disagree on what the current state is.
There is no third option. Physics doesn't have a third option either.
The engineers who understand this don't fight it. They design around it. They ask: for this specific piece of state, what consistency level do I actually need?
- User session token → strong consistency (can't be in two places)
- Shopping cart → eventual consistency (merge conflicts are recoverable)
- View count → approximate is fine
- Financial ledger → strong consistency with auditability
Different data has different consistency requirements. Treating everything as strongly consistent is expensive and slow. Treating everything as eventually consistent is dangerous.
When "Now" Actually Matters
Some systems need to know the real wall clock time. Financial transactions, audit logs, certificate issuance. You can't replace wall time with logical clocks in these cases.
The honest answer is that you need bounded uncertainty.
Google's TrueTime API (used in Spanner) is the best production example. Instead of returning a timestamp, it returns an interval: [earliest, latest]. The API commits that the true time lies somewhere in that interval, with the uncertainty being at most a few milliseconds.
Operations wait until uncertainty windows don't overlap before committing. The system is correct, but it explicitly accounts for the fact that "now" is not a point — it's a range.
Most of us don't have atomic clocks and GPS receivers in our datacenters. But the mental model applies:
When you see a timestamp, think: this is a range, not a point. What happens to your system if the true time is at the edges of that range?
Practical Consequences
Things that break when you assume wall clocks are synchronized:
Distributed locks with TTLs — If Service A acquires a lock that expires in 10 seconds, and its clock runs 50ms fast, the lock expires slightly earlier than expected. Under normal conditions, fine. Under load, with GC pauses and clock adjustments, your "held" lock has already expired.
Event sourcing and replay — If you reconstruct state by replaying events ordered by timestamp, clock skew means events can replay in the wrong order. The reconstructed state is wrong.
Log correlation — Joining logs from two services by timestamp to reconstruct a request's journey gives you a fuzzy picture at best. Events within the NTP uncertainty window cannot be reliably ordered.
Cache invalidation with TTLs — A cache entry set at T expires at T+300s. If the cache server and the origin server have drifted clocks, "expired" is ambiguous.
What To Do Instead
For ordering: use logical clocks (Lamport or vector) when causality matters. Don't use wall time.
For distributed state: embrace eventual consistency where you can. Design conflict resolution (last-write-wins, CRDTs, explicit merge functions) where you can't.
For wall time: treat timestamps as intervals, not points. Add tolerance to any comparison that crosses service boundaries.
For debugging: log both wall time and a logical sequence number. The wall time gives you human-readable context. The sequence number gives you causal ordering.
For architecture: don't design systems where correctness depends on two independent services agreeing on the current time. They won't.
Conclusion
Einstein wasn't being philosophical when he said there's no universal now.
He was being precise.
Two observers in different places, moving at different speeds, cannot agree on simultaneity. The universe doesn't provide a global clock. You have to define your reference frame and work within it.
Your distributed system operates under the same constraints.
The services don't share a clock. The network introduces asymmetric delays. "At the same time" doesn't mean anything unless you define it explicitly.
This isn't a solvable bug. It's the environment.
Design for it.
Timestamps tell you when a service thought something happened.
They do not tell you when it happened.
For that, you need causality, not clocks.
Top comments (0)