UTC Is Not Agreement: Five Clock Myths That Break Deploys, Plus a Drift Probe

Ask five developers what "now" means on a server. You will get five answers. All of them will say UTC. None of them will mention skew.

That gap is where the bugs live. A token rejected as "not yet valid." A cron job that skipped a minute. A latency graph that lies after a suspend. None of these are library bugs. They are clock-model bugs.

Here are five claims I keep hearing in review threads. Each one comes with evidence you can collect yourself, and a corrected mental model you can write down.

Myth 1: "We use UTC, so we agree on the time"

UTC is a format agreement. It is not a synchronization guarantee.

Two boxes can both print 2026-10-26T03:00:00Z and still be seconds apart. Your log formatter hides that. Your Date header does not.

Evidence to collect. Ask an external server for its Date header. Compare it to your own wall clock. The HTTP Date header is whole seconds, so treat anything under ~1.5s as noise, not as drift.

Corrected model. Time agreement is a range, not a fact. Write your tolerance down. Then assert against the tolerance, not against equality.

Myth 2: "time.time() is fine for measuring elapsed time"

Wall clocks get corrected. That is their job. Someone runs ntpd, or the host syncs, and your wall clock jumps.

If it jumps backward mid-request, your elapsed time goes negative. Your timeout math gets weird. Your dashboard shows a negative latency bar and everyone blames the chart.

Evidence to collect. Log time.time() and time.monotonic() together for an hour. Diff the deltas. Any divergence is a wall-clock correction.

Corrected model. Wall clock answers "when." Monotonic answers "how long." Never swap them.

Myth 3: "monotonic() covers suspend, restarts, and pauses"

This one is subtle and it bites free-tier boxes hard. On Linux, CLOCK_MONOTONIC does not advance while the machine is suspended. CLOCK_BOOTTIME does.

So a job that measures with monotonic() can think it ran for 3 seconds. The wall clock says 40 minutes passed. Both numbers are correct. They answer different questions.

A restart is worse. In-process clocks reset. If your only duration record lived in memory, it is gone.

Evidence to collect. Compare /proc/uptime against time.monotonic() from the same process. A small gap at boot is normal init time. A gap that keeps growing means the box slept.

Corrected model. Pick the clock family for the question. Persist deadlines in wall time. Measure intervals in monotonic time. Use boot time when suspend must count.

Myth 4: "Cron runs the minute I wrote down"

Cron is not a scheduler with catch-up. Miss a minute and that minute is gone. No retry, no apology.

Three things move the effective run time:

1. The cron daemon's timezone, which is usually the system timezone, not your shell's TZ.
2. Host clock corrections, which can skip or repeat a minute boundary.
3. % in the command, which cron treats as a newline unless escaped.

Evidence to collect. Append a Unix timestamp every minute. Compare gaps later.

# note the escaped \% : cron eats bare percent signs
(crontab -l 2>/dev/null; \
 echo '* * * * * date -u +\%s >> $HOME/cron_stamps.txt 2>&1') | crontab -

# gaps.py - run after ~30 minutes
stamps = [int(l) for l in open("cron_stamps.txt") if l.strip().isdigit()]
gaps = [b - a for a, b in zip(stamps, stamps[1:])]
print("runs:", len(stamps))
print("min gap:", min(gaps) if gaps else None)
print("max gap:", max(gaps) if gaps else None)

Pick your own threshold. Ninety seconds is a reasonable start for a one-minute job. Long gaps mean the box was down, suspended, or the daemon was reloading.

Corrected model. Cron is a heartbeat, not a contract. If you need exactly-once, cron is the wrong tool.

Myth 5: "Log timestamps tell you what happened first"

Same-second timestamps sort in insertion order. Insertion order is not event order. Two workers, one log file, and your "sequence" is fiction.

Fix. Emit three fields per line: wall time in RFC 3339, monotonic seconds, and a process-local counter.

import time, itertools, json, sys
_seq = itertools.count()

def log(event, **fields):
    rec = {
        "ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime())
              + f".{int(time.time() % 1 * 1000):03d}Z",
        "mono": round(time.monotonic(), 6),
        "seq": next(_seq),
        "event": event,
        **fields,
    }
    print(json.dumps(rec), file=sys.stderr, flush=True)

Sort by mono inside a process. Sort by ts across processes. Accept that cross-process order is approximate.

The probe: one file, read-only, no root

Save this as clock_probe.py. It writes nothing and needs no elevated access.

#!/usr/bin/env python3
"""clock_probe.py - read-only clock checks. No root. No writes."""
import email.utils
import json
import os
import time
import urllib.request


def http_date_offset(url="https://example.com", timeout=6):
    req = urllib.request.Request(url, method="HEAD",
                                 headers={"User-Agent": "clock-probe/1"})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        header = resp.headers.get("Date")
    t1 = time.time()
    if not header:
        return {"error": "no Date header"}
    server = email.utils.parsedate_to_datetime(header).timestamp()
    mid = (t0 + t1) / 2
    return {
        "offset_s": round(server - mid, 3),
        "rtt_s": round(t1 - t0, 3),
        "resolution_s": 1.0,  # HTTP Date is whole seconds
        "caveat": "offset is noise unless it exceeds resolution + rtt",
    }


def suspend_gap():
    try:
        with open("/proc/uptime") as fh:
            uptime = float(fh.read().split()[0])
    except OSError as exc:
        return {"error": str(exc)}
    mono = time.monotonic()
    return {"uptime_s": round(uptime, 1),
            "monotonic_s": round(mono, 1),
            "gap_s": round(uptime - mono, 1)}


def families():
    out = {"wall_s": round(time.time(), 6),
           "monotonic_s": round(time.monotonic(), 6)}
    if hasattr(time, "CLOCK_BOOTTIME"):
        out["boottime_s"] = round(time.clock_gettime(time.CLOCK_BOOTTIME), 6)
    return out


if __name__ == "__main__":
    print(json.dumps({
        "host_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "http_date": http_date_offset(),
        "suspend": suspend_gap(),
        "clocks": families(),
        "env": {k: os.environ.get(k) for k in ("TZ", "LANG")},
    }, indent=2))

Run it twice. Ten minutes apart is enough.

python3 clock_probe.py > a.json
sleep 600
python3 clock_probe.py > b.json
diff <(python3 -m json.tool --sort-keys a.json) \
     <(python3 -m json.tool --sort-keys b.json)

What to read in the diff:

• offset_s moving by more than a second between runs: your clock is being stepped, not slewed.
• gap_s growing: the box suspended, or /proc/uptime and monotonic disagree for another reason.
• boottime_s present and advancing while monotonic_s pauses: suspend is confirmed.
• Nothing moves: your baseline is quiet. Save both files as the reference pair.

I am not printing my numbers as benchmarks. Your box is the only box that matters here. Numbers from someone else's host tell you nothing about yours.

Where a free box and free model access actually help

You want a throwaway machine to run this drill on. That is exactly the low-stakes case for the free server option: the probe is read-only, needs no data, and a wrong answer costs you nothing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My loop is boring on purpose. Run the probe on the free server, keep the JSON, run it again later. When two reports disagree and I cannot tell whether the delta is real, I paste both into a session on the free model access and ask it to list only the fields that changed. I check every field it names against the raw files before I trust it.

The model is a second pair of eyes on a diff. It is not a time source. Never let it tell you what the offset "should" be.

Decision table

Question	Use	Do not use
How long did this call take?	time.perf_counter()	wall clock
Did the job finish before the deadline, across restarts?	persisted wall clock + offset check	in-process monotonic
How long was the box up, including suspend?	/proc/uptime or CLOCK_BOOTTIME	CLOCK_MONOTONIC
Cache TTL inside one process	time.monotonic()	wall clock
Token exp / nbf checks	wall clock with explicit leeway	exact equality
Log ordering	wall clock + monotonic + counter	wall clock alone

The leeway row deserves a line. If you validate tokens, pass a tolerance. In PyJWT that is jwt.decode(token, key, algorithms=["RS256"], leeway=60). Leeway absorbs skew at verification. It does not fix a broken clock, and it does not extend a token's life.

Limitations and who should skip this

• HTTP Date is a coarse source. Sub-second work needs a real time service, not this probe.
• Inside containers, the host owns the clock. You can observe drift. You usually cannot correct it.
• If you already run chrony with alerting on offset, you have better data than this. Do not rebuild it smaller.
• Single-process CLI tools on your laptop do not need any of this. time.monotonic() and move on.
• This probe does not test monotonic behavior across suspend. It only shows evidence that suspend happened.

What to write down

One paragraph in your runbook beats five clever code comments. Mine says: wall clock for labels, monotonic for durations, boot time for uptime, and every comparison gets a tolerance.

Which of the five myths is still in your repo right now? Grep for time.time() in a duration calculation. That is the fastest place to start.

If you want to try the drill, a disposable free box is enough. Bring your own thresholds.