Monitoring 34 NTP Synchronized Clocks from a Cron Job

There is a category of device that never appears in your observability stack: the wall clock.
It draws 12 watts, it has an IP address, and nobody ever checks whether it's right. Which is a strange thing to say about a device whose only function is being right.
Last quarter I ended up responsible for 34 of them across three buildings — 8-inch LED seven-segment displays, HH:MM:SS, purchased as NTP synchronized clocks. This post is the unglamorous story of making them observable, including the two things I got wrong.

The setup, briefly

Each unit is a display node with:

• an NTP client (default), or a fallback to GPS / BeiDou / 4G / WiFi where the LAN is unreliable
• an Ethernet port speaking TCP/IP, plus RS485 on the same unit
• PoE power, so one cable does power and data
• an HTTP interface for status and configuration
• a scheduler for power on/off, brightness and countdown The key architectural property, and the reason this whole post is short: the display does not decide what time it is. It polls a source. That makes each unit a network clock in the literal sense, and it makes the entire fleet a client-server system — and client-server systems are observable if you bother to look. If you're coming from the older world: this is the same topology the master-slave clock system used, minus the dedicated pulse wiring. One reference, many followers, correction distributed over an existing transport. The only thing that changed is that the transport is now Ethernet instead of a two-wire loop.

Step 1: sanity-check the upstream before blaming the fleet
Before writing any monitoring code, verify the thing you're syncing to is actually sane. If your local NTP server is drifting or unreachable, every downstream display will faithfully display the wrong time and every dashboard will be green.

import ntplib
from time import ctime

def check_source(host="192.168.10.5", timeout=2):
    client = ntplib.NTPClient()
    try:
        resp = client.request(host, version=3, timeout=timeout)
    except Exception as e:
        return {"host": host, "ok": False, "error": str(e)}

    return {
        "host": host,
        "ok": True,
        "server_time": ctime(resp.tx_time),
        "offset_ms": round(resp.offset * 1000, 2),
        "roundtrip_ms": round(resp.delay * 1000, 2),
        "stratum": resp.stratum,
    }

print(check_source())
# {'host': '192.168.10.5', 'ok': True, 'server_time': 'Thu Oct 22 09:14:03 2026',
#  'offset_ms': -0.41, 'roundtrip_ms': 1.8, 'stratum': 2}

Two notes that cost me an afternoon each:

• resp.offset is the difference between your machine and the server, not the server's error. If your monitoring host is itself unsynced, you're measuring your own drift.
• resp.delay is the round trip. On a congested VLAN this can hit 100 ms and make the offset look worse than it is. The offset calculation already compensates for half the round trip, but not for asymmetry.

Step 2: what "correct" means for a display
Here is the bit I got wrong first. I initially monitored each display's offset from the NTP server and alerted at > 500 ms.
Two problems:

1. An NTP synchronized clock that has lost its upstream entirely often reports an offset of zero — because the last successful sync was zero. Stale data looks like perfect health.
2. Visually, nobody can tell 500 ms from 0. The threshold that matters is "a human comparing two clocks side by side notices" — around one second — and the operational threshold is "a logged timestamp could be disputed" — a few seconds, not milliseconds.

Step 3: poll the fleet, not the protocol
Each unit exposes an HTTP status endpoint. I'm deliberately not inventing the exact JSON schema here — vendors differ, and you should read the API doc that ships with your hardware rather than trusting a blog post. The shape I normalize to internally looks like this:

{
  "id": "BLD-A-F2-REG-01",
  "ip": "192.168.20.41",
  "online": true,
  "synced": true,
  "last_sync_age_s": 37,
  "offset_ms": 3,
  "source": "ntp",
  "firmware": "1.4.2",
  "uptime_s": 982134
}

Collector:

import json, time, concurrent.futures as cf
import requests

FLEET = json.load(open("fleet.json"))   # [{"id": "...", "ip": "..."}]

def poll(dev, timeout=3):
    url = f"http://{dev['ip']}/status"   # adjust to your vendor's endpoint
    try:
        r = requests.get(url, timeout=timeout)
        r.raise_for_status()
        data = r.json()
        return {**dev, **data, "reachable": True}
    except Exception as e:
        return {**dev, "reachable": False, "error": str(e), "ts": time.time()}

with cf.ThreadPoolExecutor(max_workers=16) as pool:
    results = list(pool.map(poll, FLEET))

bad = [r for r in results
       if not r["reachable"]
       or r.get("last_sync_age_s", 0) > 600
       or abs(r.get("offset_ms", 0)) > 1000]

print(f"{len(bad)}/{len(results)} need attention")
for r in bad:
    print(r["id"], r.get("error") or r.get("offset_ms"))

34 devices, 16 threads, finishes in about 2 seconds. A thread pool is fine here; asyncio would be tidier but this runs from cron and doesn't deserve the ceremony.

Step 4: the schedule is the cheapest win
The single highest-value thing I did had nothing to do with accuracy.
The old clocks were bright 24/7 because there was no way to turn them off. The new units accept a power schedule from the management plane — off at 21:30, on at 06:00. In a building that operates roughly 15 hours a day, that removes about 37% of runtime across the fleet.

# applied per-device from the console; conceptually:
#   power_on  06:00  power_off  21:30
#   brightness_day 80%  brightness_night 0%

I keep the brightness step separate from the power step on purpose. On units in corridors with overnight staff, killing power is wrong; dimming to a low level is right. Being able to express both in the same schedule model is what made the energy argument survive contact with the night shift.

Step 5: don't build a dashboard nobody opens
I built a dashboard. It got looked at for about three weeks.
What actually worked was pushing the three bad signals into the ticketing system the facilities team already uses, as one daily digest. One line per failing device, one link to the device's config page. No new tab, no new login, no new habit to form.
If you're integrating with a BMS instead, the same idea applies: the fleet should be a data source in someone else's view, not a destination of its own. That's the entire argument for requiring a documented open interface in the spec — HTTP or RS485 or raw TCP, as long as it's documented and stable.

Two things I'd do differently

Start with mounting, not electronics. Three of my four rollout delays were mechanical — a suspended double-sided unit that needed a different ceiling anchor than the wall units, and a corridor where the existing drop was behind the paneling. The electronics took twenty minutes per device. The physical work took weeks.
Specify remote firmware update before you need it. One unit shipped with a timezone handling bug that would have required a site visit, a ladder and a laptop. It was fixed by pushing an update from the console — no disassembly, no badge access. That feature is worth paying for on day one, and it's nearly impossible to add later if it wasn't designed in.

The takeaway

A fleet of simple digital clocks is the cheapest distributed system you'll ever own, and it fails in the most public way possible: on a wall, in front of users, for months, without logging anything.
Give it three signals — reachable, last synced, current offset — and a schedule, and it stops being a source of quiet embarrassment.
[X]