DEV Community

Cover image for Clock Synchronization in Distributed Databases
Urvish Shah
Urvish Shah

Posted on

Clock Synchronization in Distributed Databases

#architecture #backend #database #scalability

Part 1: The Physical Clock Problem, Quartz, NTP, and Why Time Lies

This is Part 1 of a 5-part series on clock synchronization in
distributed databases, covering CockroachDB and Aurora DSQL.

  • Part 1: The Physical Clock Problem (you are here)
  • Part 2: Atomic Clocks, The Source of Truth
  • Part 3: The Hybrid Logical Clock, CockroachDB's Solution
  • Part 4: Uncertainty Intervals, MVCC, and Transaction Restarts
  • Part 5: Aurora DSQL, When You Can Assume Atomic Clocks

Why Time Matters in Distributed Databases

Time seems simple. It isn't, especially in a distributed database.

When your database spans multiple nodes across regions, every transaction
needs a timestamp. That timestamp drives two critical things:

  • MVCC versioning: distinguishing before and after versions of a row
  • Serializable isolation: determining whether two concurrent transactions conflict

The challenge is that no two clocks across different machines ever read
exactly the same time. They drift. They jump. They disagree. If your
database trusts them blindly, you get subtle, hard-to-reproduce
consistency violations.

This series walks through how CockroachDB and Aurora DSQL tackle this
problem, starting from the physics of quartz crystals, through NTP and
atomic clocks, and ultimately to bounded uncertainty intervals and
transaction restarts.

The Physical Clock, What It Actually Is

Every server has a quartz crystal oscillator on its motherboard. When
electricity passes through the crystal, it vibrates at a predictable
frequency. The operating system counts those vibrations to track time.

Quartz oscillator vibrates
-> OS counts vibrations
-> System clock value (what software reads as "current time")
Enter fullscreen mode Exit fullscreen mode

Two terms you will see used often refer to the same underlying reality
at different levels of abstraction:

Term What it means
Quartz oscillator The physical silicon chip that vibrates
System clock / Physical clock / Wall clock The OS time value built by counting those vibrations

They are not two separate things. The oscillator produces the ticks.
The system clock is the counter that reads those ticks.

The Drift Problem

Quartz crystals are not perfect. Manufacturing tolerances, temperature
changes, CPU load, aging, and power fluctuations all cause the vibration
rate to shift slightly over time. Left uncorrected, every server clock
will slowly wander away from true time.

Drift rate:  roughly 1 second per 11 to 12 days

Over 30 days without correction:

  Server A clock:  12:00:02.700
  True time:       12:00:00.000
  Server C clock:  11:59:57.400

  Difference between A and C: over 5 seconds
Enter fullscreen mode Exit fullscreen mode

In a single-node system, clock drift rarely causes problems. Everything
uses the same clock, so events are still ordered correctly relative to
each other.

In a distributed system it is a different story. Two nodes writing the
same key at the "same time" may disagree by seconds about what that
means. A read on Node A may silently miss a write on Node B that
happened moments earlier, simply because Node A's clock is behind.

NTP, How the World Corrects Drift

Network Time Protocol (NTP) is the standard answer to clock drift.
Before going into how it works, it is worth being clear about what NTP
actually is:

NTP is not a clock. It is a synchronization protocol. It does not
improve the quality of your quartz oscillator. It periodically asks
an external server what time it is, then adjusts your local system
clock toward that answer.

How NTP Measures the Offset

NTP uses a 4-timestamp exchange to estimate how far your clock is from
the server's clock:

Step 1: Your server sends a request, recording its local time T1
Step 2: NTP server receives it, records its time T2
Step 3: NTP server sends a reply, recording its time T3
Step 4: Your server receives the reply, records its local time T4

Round-trip delay = (T4 - T1) - (T3 - T2)
Clock offset     = ((T2 - T1) + (T3 - T4)) / 2
Enter fullscreen mode Exit fullscreen mode

Your clock is then adjusted toward the correct time based on the
calculated offset.

Slewing vs. Stepping, How the Correction Is Applied

There are two ways NTP can apply a correction.

Slewing is a gradual adjustment:

Target correction: +50ms
Slew rate: roughly 0.5ms per second
Time to complete: about 100 seconds

The system clock is nudged slowly and continuously.
No abrupt jump. Time always moves forward.
Enter fullscreen mode Exit fullscreen mode

Stepping is an abrupt jump:

Target correction: +50ms
Applied instantly in a single adjustment.

The system clock jumps forward (or backward) immediately.
Enter fullscreen mode Exit fullscreen mode

For databases, slewing is strongly preferred. An abrupt step that moves
the clock backward, even by a few milliseconds, can cause a new write
to receive a lower timestamp than an older write on the same node.

One more important clarification before moving on:

NTP usually does not physically change the quartz crystal’s natural oscillation rate. Instead, it disciplines the software/system clock by adjusting its effective rate and offset on top of that hardware clock source.

The NTP Stratum Hierarchy

NTP organizes time sources into a hierarchy called stratums. The lower
the stratum number, the closer the source is to a physical atomic clock.

Stratum 0   Physical atomic clocks, GPS receivers
            (not directly on the network)
                |
Stratum 1   Servers directly connected to Stratum 0
                |
Stratum 2   Servers synced from Stratum 1
                |
Stratum 3   Servers synced from Stratum 2
                |
Your Server Typically Stratum 3 or 4
            Each hop adds network jitter and error
Enter fullscreen mode Exit fullscreen mode

Typical accuracy by level:

Source Typical Accuracy
Stratum 0 (atomic clock / GPS) Nanoseconds
Stratum 1 (directly connected) 1 to 10 microseconds
Stratum 2 (LAN) ~1 millisecond
Stratum 3 to 4 (internet) 1 to 50 milliseconds

Your server is generally 3 to 4 hops away from an atomic clock.
Each hop adds error that cannot be fully recovered downstream.

Why NTP Has a Hard Accuracy Ceiling

Even with a perfect atomic clock at Stratum 0, NTP cannot give you
nanosecond accuracy at your server. Several factors create a floor on
NTP error that no amount of tuning can eliminate.

Network Jitter

Each NTP measurement is a network round-trip. The time that round-trip
takes varies unpredictably:

Request 1:   5ms round-trip
Request 2:  23ms round-trip  (router was busy)
Request 3:   8ms round-trip

NTP averages and filters these, but cannot eliminate the variance.
The resulting offset estimate carries residual error from jitter.
Enter fullscreen mode Exit fullscreen mode

Path Asymmetry

NTP assumes the message takes the same time in each direction. In
practice it rarely does:

A to B: 5ms
B to A: 15ms

NTP computes: round-trip = 20ms, assumes one-way = 10ms
Actual one-way A to B = 5ms
Systematic offset error = 5ms, undetectable and uncorrectable
Enter fullscreen mode Exit fullscreen mode

This asymmetry error is silently absorbed into the offset measurement.
It is one reason why CockroachDB's default max_offset is a
conservative 500ms rather than something close to typical NTP accuracy.

Software Timestamping

NTP timestamps are added in software, after the OS has processed the
packet. OS scheduling delays, interrupt handling, and context switches
all introduce noise, typically microseconds to low milliseconds on a
busy server.

Virtualization

In a virtual machine, the guest OS clock depends on the hypervisor.
If the VM is paused for migration, or the host is under CPU pressure,
the guest clock can drift significantly and NTP corrections can lag
behind. VMware and KVM documentation both note that VM clock discipline
requires special configuration, making this a real operational concern
for any database running in virtual environments.

Leap Seconds

Occasionally a leap second is inserted into UTC to keep it aligned with
Earth's rotation. This is a 1-second discontinuity that looks like a
catastrophic clock jump to software that does not handle it carefully.

Leap second event:
  23:59:59 UTC
  23:59:60 UTC  (the extra second)
  00:00:00 UTC

Systems that assume 60 seconds per minute can crash,
produce duplicate timestamps, or show impossible log entries.
Enter fullscreen mode Exit fullscreen mode

Google and AWS handle this with leap smearing, distributing the
extra second gradually over a 24-hour window so the clock advances
continuously without any abrupt change. This approach is strongly
recommended for CockroachDB deployments. There is one important caveat
though: all nodes in a cluster must use the same smearing implementation.
Mixing smeared and non-smeared NTP sources introduces a 1-second offset
between nodes, which will trigger the cluster's self-protection
mechanisms.

Where We Stand After Part 1

The problem:
Quartz oscillators drift. Every server clock wanders from true time.
In a distributed database, nodes disagreeing on time leads directly
to consistency violations.

NTP's role:
NTP periodically corrects the system clock toward an external reference.
It corrects the software reading, not the oscillator frequency itself.
Slewing (gradual) is safe for databases. Stepping (abrupt) is not.

NTP's limits:
Network jitter, path asymmetry, software timestamping, virtualization,
and leap seconds all create a hard floor on NTP accuracy.
Typical accuracy: 1 to 50ms over the internet, ~1ms on a good LAN.

The gap:
For most applications, millisecond-level accuracy is fine.
For a distributed database where transaction ordering depends on
timestamps across nodes, that level of uncertainty requires careful
engineering to handle correctly.

In Part 2, we go to the source of truth: atomic clocks. What they
are, how they work, why they are so accurate, the different types that
exist, and how AWS built atomic clock infrastructure into every region
so that Aurora DSQL can operate with microsecond-level certainty instead
of millisecond-level guesses.

References

Next up: Part 2, Atomic Clocks, The Source of Truth

Top comments (0)