My Raspberry Pi's Clock Drifted 9 Minutes. My AI Agent Silently Accepted 401s for Six Days.

For six days, my AI agent dutifully woke up at 6 AM, gathered the news, drafted my morning brief, and hit the send API. And for six days, the send API rejected every single request with a 401 Unauthorized.

I didn't notice because my monitoring said everything was fine. The process was running. The cron jobs fired. The logs were full of activity. From the outside, my little Raspberry Pi automation empire was humming along.

The bug wasn't in my code, my API key, or the model. It was in the Pi's clock — and it exposed a monitoring blind spot I'm a little embarrassed to admit I had.

The setup

I run a handful of automation agents on a Raspberry Pi 5 in my office: a morning briefing agent, an inbox triager, a nightly recon script, and a watchdog that pings me if anything dies. Nothing exotic — Python scripts, systemd timers and cron, a couple of paid APIs.

One of those APIs signs every request with a timestamp and an HMAC signature — standard practice for webhook and API security. The server rejects any request whose timestamp is more than 5 minutes away from its own clock. That tolerance window exists precisely to stop replay attacks.

Here's the thing about Raspberry Pis: they have no real-time clock battery. When the Pi loses power, it forgets what time it is. On boot, systemd-timesyncd syncs the clock over NTP — usually within seconds of getting network. Usually.

The failure

One Sunday evening, my area had a brief power flicker. The Pi rebooted itself cleanly. Everything came back up... except NTP sync quietly failed. My router's DNS was serving stale records for the NTP pool hosts, and systemd-timesyncd kept retrying against unresolvable names.

So the Pi fell back to the last time it remembered — the fake "epoch-ish" time it uses before first sync, adjusted by a saved timestamp file. It came up roughly 9 minutes fast.

Nine minutes. Just outside the API's 5-minute signature tolerance window. Every signed request my agent made was rejected as a possible replay attack. The API was right to reject them. My agent, meanwhile, logged the 401s at DEBUG level and moved on, because I had written its error handling to "be resilient" — retry a few times, then continue the pipeline rather than crash.

Resilient right into a six-day silent failure.

How I finally found it

A reader of my morning brief — a friend who'd asked to be on the list — messaged me: "haven't gotten the brief all week, everything ok?"

That's the honest part of this story: a human noticed before any of my automation did. I had built a watchdog that checked whether processes were alive, whether the Pi was reachable, whether disk and memory were okay. All green. All useless for this failure, because the failure wasn't "the agent is down." It was "the agent is up, working hard, and achieving nothing."

My first debugging move was also wrong. I checked the API dashboard, saw a wall of 401s, and immediately assumed the API key had been rotated or rate-limited. I regenerated the key, redeployed, and... still 401s. I wasted an evening suspecting my HMAC signing code, re-reading the docs, even diffing my signature logic against the reference implementation. The signature logic was perfect. It was signing the wrong time.

The clue finally showed up when I printed the request headers side by side with the API's error body. The server said timestamp too far in the future. I ran date on the Pi and date on my laptop:

pi:      Sun Sep 13 06:00:12 BST 2026
laptop:  Sun Sep 13 05:51:03 BST 2026

Nine minutes. There it was.

The fix (10 minutes) and the real fix (an afternoon)

The immediate fix was trivial:

sudo timedatectl set-ntp true
sudo systemctl restart systemd-timesyncd
timedatectl show-timesync --property=ServerName,NTPMessage

Once DNS cleared up, the clock snapped back into place and the next scheduled run went through. But a one-line fix isn't a fix — it's a reprieve. The actual problems were:

1. Nothing alerted me when time sync failed. systemd-timesyncd failing is invisible unless you go looking. I added a tiny health check to my watchdog:

import time, subprocess

def check_clock_drift(max_drift_seconds=60):
    # Compare local clock against an HTTP Date header
    import requests
    r = requests.head("https://cloudflare.com", timeout=10)
    server_time = r.headers.get("Date")
    from email.utils import parsedate_to_datetime
    remote = parsedate_to_datetime(server_time).timestamp()
    drift = abs(time.time() - remote)
    if drift > max_drift_seconds:
        alert(f"Clock drift {drift:.0f}s exceeds {max_drift_seconds}s")

Any drift over 60 seconds pages me. HTTP Date headers are second-granular, which is plenty — I care about catching 9-minute drifts, not millisecond ones.

2. My agent treated a 401 as a transient error. A 401 is not a rate limit and not a network blip. Retrying it "for resilience" just burned six days of API quota and my credibility. I changed the rule: 4xx errors (except 408/429) are fatal to the run and page me immediately. If the API says "no," no amount of trying again will make it "yes."

3. My monitoring checked liveness, not outcomes. This was the big lesson. "Is the process running?" is the laziest possible health check. What I actually care about is "did the morning brief get delivered?" So the watchdog now verifies artifacts, not processes: it checks that the brief email's message ID appears in the send-log for today, that the recon script produced a non-empty output file, that the inbox triager's counters advanced. Dead-simple assertions, but they catch silent failures that liveness checks structurally cannot.

If you take one thing from this post, make it this: monitor the outcome, not the heartbeat.

The pattern behind the failure

Looking back, this was the third time I'd been bitten by the same shape of bug: a system that fails quietly while looking healthy. A backup job that hadn't actually backed up anything. A retry loop that reported success while achieving nothing. And now a clock-skewed agent politely accepting 401s for a week.

They all share one trait: the failure signal existed (401s in the log, empty backup dirs, sync errors in the journal) but nobody — no human, no script — was wired to care about that signal. Automation without outcome checks is just a more confident way of doing nothing.

My rules now, after each of these post-mortems:

• Every automated job must write a machine-checkable "I accomplished X today" artifact.
• A watchdog verifies the artifact, not the process.
• Auth errors are loud, immediate, and fatal. Resilience is for network blips, not permission denials.
• Anything time-based on a Pi gets a drift check. Cheap insurance against a battery-less clock.

The clock drift check runs every 15 minutes and has caught exactly one real incident since — the day my router's DNS went haywire again, three weeks later. This time I got the alert in under 20 minutes instead of learning about it from a friend six days later.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.