DEV Community

Cover image for A migrated VM, a dead NTP daemon, and 67 days of silent clock drift
Dibyaprakash Pradhan
Dibyaprakash Pradhan

Posted on

A migrated VM, a dead NTP daemon, and 67 days of silent clock drift

#aws #devops #linux #debugging

Every request we made to an AWS API started failing with a 403.

Not intermittently. All of them, instantly, with the same error repeating once per second:

(403, 'InvalidSignatureException',
'Signature expired: 20260314T090925Z is now earlier than
20260314T091043Z (20260314T091543Z - 5 min.)')
Enter fullscreen mode Exit fullscreen mode

A 403 on AWS usually sends you straight to IAM. Wrong key, expired key, missing permission, bad role. I very nearly went there.

Read the whole error before you fix the error

The thing that saved me was that AWS had already done the work. That message has three timestamps in it, and they are not decoration.

  • 20260314T090925Z is when our box signed the request. That is our clock.
  • 20260314T091543Z is what AWS's clock said when it arrived.
  • 20260314T091043Z is that minus five minutes, the cutoff.

So AWS is telling you, in the error text, exactly how far apart the two clocks are. Ours was 378 seconds behind. Six minutes and eighteen seconds.

SigV4 signatures are valid for five minutes. Our signature was stale the instant it was created. It never had a chance.

The credentials were perfect. The request was perfect. We just could not agree with AWS about what time it was.

What made me confident rather than merely suspicious was that the gap never moved. Every log line, the same 378 seconds:

signed 20260314T090925Z  aws 20260314T091543Z  drift 378s
signed 20260314T090926Z  aws 20260314T091544Z  drift 378s
signed 20260314T090929Z  aws 20260314T091547Z  drift 378s
Enter fullscreen mode Exit fullscreen mode

A clock that is slowly drifting gives you a number that creeps. A clock frozen at a fixed offset means nothing is correcting it at all. That is a different problem, and a much more interesting one.

There was a second witness too. Our logs ship off the box, and each entry carries both the timestamp the application wrote and the timestamp the ingestion service wrote when it arrived. The gap between them was 377.8 seconds. Two independent time authorities telling us the same thing about our own machine.

The cliff

Here is the part worth internalising, because it explains why this felt like a sudden outage rather than a creeping problem.

There is no degradation curve with SigV4. At 4 minutes 59 seconds of drift, everything works perfectly. At 5 minutes 1 second, every single request fails. It is a cliff, not a slope.

So the clock had probably been wrong for a long time before anyone noticed. It just had not crossed the line yet. The day it crossed, a whole class of functionality vanished at once, and it looked like something had broken that morning. Nothing had broken that morning. Something had broken months earlier and only became visible that morning.

Going in

timedatectl was blunt about it:

System clock synchronized: no
              NTP service: inactive
Enter fullscreen mode Exit fullscreen mode

And then the line that reframed the whole thing:

Active: failed (Result: exit-code) since Tue 2026-01-06 09:30:20 UTC; 2 months 7 days ago
Enter fullscreen mode Exit fullscreen mode

chrony had been dead for 67 days. It started 18 seconds after a boot in January, exited, and nothing ever brought it back. The clock had been free-running for months.

Nobody knew. It is a daemon nobody looks at, failing quietly, on a box that was otherwise completely healthy.

My first theory was that chrony had refused the correction. Probing it by hand seemed to confirm it:

$ sudo chronyd -Q 'server 169.254.169.123 iburst'
System clock wrong by 377.877093 seconds (ignored)
chronyd exiting
Enter fullscreen mode Exit fullscreen mode

"Ignored." There it was. chrony saw the offset, decided a six minute jump was implausible, and gave up.

That was real, and it was also not the reason the service was down. I was pattern matching on the first plausible cause and I was wrong. The actual reason only showed up when I stopped probing and tried to start the thing properly.

The actual cause

Step the clock manually first, then start the service:

sudo chronyd -q 'server 169.254.169.123 iburst'
# System clock wrong by 377.880183 seconds (step)
Enter fullscreen mode Exit fullscreen mode

That worked. Clock correct, 378 seconds recovered, AWS calls authenticating again.

Then:

sudo systemctl start chrony
Enter fullscreen mode Exit fullscreen mode 
Could not open /dev/ptp_hyperv : No such file or directory
Fatal error : Could not open PHC
Enter fullscreen mode Exit fullscreen mode

ptp_hyperv.

Hyper-V.

We are on EC2.

Line 5 of /etc/chrony/chrony.conf, with a comment that removes all doubt:

# Use Azure platform PTP device
refclock PHC /dev/ptp_hyperv poll 3 dpoll -2 offset 0
Enter fullscreen mode Exit fullscreen mode

This is an Azure config. On an EC2 box. Pointing at a Hyper-V paravirtual clock device that does not exist here and never will. chrony treats a missing refclock as fatal, so it exited at startup. Every time. Regardless of what the clock said.

My "chrony refused the large offset" theory was a real behaviour of the manual probe, but it had nothing to do with why the service was down. The service was down because it could not start at all.

How Azure config ended up on an EC2 box

I had migrated this VM from Azure using AWS Application Migration Service (MGN).

MGN does block level replication. It lifts your disk and boots it on EC2, and it is genuinely good at that. Your application comes across intact, your packages come across intact, your config comes across intact.

Which is the whole point, and also the entire problem.

/etc/chrony/chrony.conf came across intact. It was a perfectly correct file on Azure, where /dev/ptp_hyperv is exactly the right thing to sync against. The moment it landed on EC2 it became a file that kills chrony on boot.

MGN cannot know that. It does not read your config and reason about which lines describe hardware that no longer exists. It moves your disk. Anything in that disk describing the old hypervisor is now a small landmine, armed and waiting for the next restart.

And notice the delay. The migration did not break anything visibly. chrony kept running on its existing process. It was the reboot in January that killed it, because that is when chrony next tried to start and could not find its device. Then the clock drifted quietly for two months until it crossed five minutes and AWS started saying no.

Migration, reboot, drift, cliff. Four events spread over months, and only the last one produced an error message.

The fix

Comment out the Azure device:

sudo sed -i '5s|^|#|' /etc/chrony/chrony.conf
Enter fullscreen mode Exit fullscreen mode

Then a surprise. Grepping for a replacement time source turned up nothing:

$ grep -nE '^[[:space:]]*(server|pool|makestep)' /etc/chrony/chrony.conf
30:makestep 1 3
Enter fullscreen mode Exit fullscreen mode

No server. No pool. The Azure PTP device had been the only time source in the file. Removing it left chrony with a step policy and nothing to step toward.

This matters more than it sounds. If I had just commented the line out and started the service, I would have got a running, healthy looking daemon that syncs from nothing at all. Green in systemctl status, quietly drifting back toward the same cliff. That is a worse failure than the crash, because the crash at least left a corpse.

On EC2 the answer is the link local Amazon Time Sync Service:

sudo tee -a /etc/chrony/chrony.conf >/dev/null <<'EOF'

# EC2: Amazon Time Sync Service (replaces the Azure PTP refclock above)
server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4
makestep 1.0 -1
EOF
Enter fullscreen mode Exit fullscreen mode

Two notes on that.

I appended to chrony.conf directly rather than dropping a file into conf.d/, because this config has no confdir directive. I had already written the file to conf.d/aws.conf before checking, and it would have sat there being ignored forever. Worth a grep before you assume a drop in directory is wired up.

And makestep 1.0 -1 replaces the existing makestep 1 3. The default only allows stepping during the first three updates after startup. After that chrony will only slew, and slewing 378 seconds takes effectively forever. -1 means step whenever it is needed, which is what you want on a VM that can be paused, migrated or snapshotted out from under you.

Result:

^* 169.254.169.123    3   4   77    2   +653ns[ -983us] +/- 339us
Enter fullscreen mode Exit fullscreen mode

653 nanoseconds. From 378 seconds.

One last step, because the hardware clock was still 106 seconds out and would have reintroduced skew on the next boot:

sudo hwclock --systohc
Enter fullscreen mode Exit fullscreen mode

What I would tell past me

Read the whole error. AWS had handed me the exact drift, in seconds, inside the message itself. I nearly went to IAM because 403 looks like a permissions problem and I stopped reading at the status code.

A constant offset is a different bug from a drifting one. Drift means nobody is correcting. A frozen number means something tried, failed, and stopped. That distinction took me straight to a dead daemon.

After a lift and shift migration, audit for the old hypervisor. Not just chrony. Grep your configs for hyperv, xen, vmware, virtio, waagent, anything naming a device or agent from the platform you left. MGN moved the disk faithfully. Faithfully includes the parts that are now wrong.

Reboot the migrated box on purpose, early. Our failure was armed at migration and did not fire until a reboot months later. The gap between cause and symptom was long enough that nobody connected them. A deliberate reboot the same week would have surfaced it immediately, next to the change that caused it.

Alert on clock offset. chronyc tracking gives you the number for free. Alert at 60 seconds, well below the five minute cliff, and you get warned during the slope instead of discovering it at the edge.

When you fix a daemon, verify it has something to do. Running is not the same as working. A time daemon with no time source is green in every dashboard and wrong in every way that matters.

The whole thing came down to one line of config that was completely correct in the datacentre it was written for, and quietly fatal in the one it ended up in.

Top comments (0)