The Free Server's Clock Was 47 Seconds Fast, and Every Token Expired Early

It started at 2:14 AM with a cron job that had run cleanly for over a week. The job was supposed to call a free model endpoint from a free server, summarize a log file, and post the result to a channel. Instead, every request came back with a 401, and the logs didn't even bother to explain why.

The strangest part was that the exact same script worked on my laptop. Same token, same payload, same endpoint, and the model answered within a second. How can a token be valid on one machine and rejected on another? That question turned out to be the entire debugging session in miniature.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The symptom that made no sense

Let me set the scene properly, because the details matter more than the drama. The setup used MonkeyCode's free server option to run a small scheduled job, and the job authenticated against a free model API using a short-lived token. The token was generated by a helper script, stored in a file, and refreshed every hour.

At 2:14 AM the refresh succeeded, but the actual model call failed. The error was a bare HTTP 401 with an empty body, which is about as helpful as a compiler that only prints "no." I checked the cron logs, confirmed the token file was fresh, and ran the same call manually from the server. It failed again, with the same silence.

Then I ran it from my laptop, using the same token file copied over SSH. It worked. That single contrast — identical inputs, different results — was the clue I kept circling around.

What I ruled out first

I went through the usual suspects in the usual order, and I want to show you that list because it saved me later.

• Rotated API key: I checked the dashboard, and the key was unchanged. The token was generated from the current key, so this was a dead end.
• IP allowlist: Some APIs restrict egress IPs, and free servers often share a pool. The dashboard showed no allowlist, but I verified the server's egress IP anyway.
• Expired token: I decoded the JWT locally, and the exp claim was hours away. This should have been the end of the story, but the API clearly disagreed.

Here is the exact snippet I used to inspect the token:

import jwt

token = open("token.txt").read()
claims = jwt.decode(token, options={"verify_signature": False})
print(claims)

The output looked perfectly healthy: exp was hours in the future, iss matched, and the scopes were present. I was looking at a valid token that the API refused to accept, and I had no theory left.

The 47-second lie

I took a break, and when I came back I asked myself a stupid question: what does this server believe about time? I ran date -u on the free server and compared it to the same command on my laptop. The server was 47 seconds ahead of real time, and my laptop was within a second of it.

That is a huge deal for token validation. Many APIs reject tokens whose iat (issued-at) claim is in the future, because that usually means a replay or a misconfigured client. My server's clock was ahead, so the token's iat was in the future from the API's perspective. Worse, the token's exp was computed from the same skewed clock, so the API saw it as already expired.

The token wasn't wrong. The server's clock was lying, and every token minted on that server inherited the lie.

The fix that took four minutes

The root cause was boring in the best way: the free server had no working NTP sync. The fix was to enable time synchronization and verify it.

sudo timedatectl set-ntp true
sudo systemctl restart systemd-timesyncd
timedatectl status

If your distro prefers chrony, the equivalent is:

sudo apt install chrony
sudo systemctl enable --now chrony
chronyc tracking

After that, I compared the server clock against an HTTP Date header, which is a handy trick when NTP is blocked:

curl -sI https://api.example.com | grep -i date
date -u

The drift dropped to under a second, and the next scheduled run succeeded. I also added a tiny monitoring check that alerts me if the server clock drifts more than five seconds, because this failure mode is silent until auth starts failing.

The reusable debugging checklist

This incident wasn't exotic, and that's exactly why it's worth writing down. Here is the checklist I now use whenever auth fails on one machine but not another:

1. Reproduce with the smallest possible request. A single curl with the exact failing token beats any amount of log reading.
2. Diff the environments, not just the code. Time, DNS, locale, proxy, and timezone all live outside your repository.
3. Decode the token and check every claim, not just exp. iat and nbf cause the same 401s and are easier to miss.
4. Check the server clock before blaming the API provider. date -u costs nothing and rules out a whole class of failures.
5. Fix the root cause first, and only then consider adding leeway to your validator. A broken clock will keep breaking things no matter how much tolerance you add.

Limitations and who should not use this approach

Let me be honest about the boundaries of this story, because "check the clock" is not a universal cure.

First, not every 401 is clock skew. If your token is genuinely expired, if the key was rotated, or if the API changed its signing algorithm, no amount of NTP will help. Verify the claims before touching the clock.

Second, some free servers block outbound NTP on UDP port 123. In that case, use an HTTP-based time source or ask the provider for a managed time service, and document the workaround in your runbook.

Third, do not widen JWT leeway on security-critical tokens just to mask a drifting clock. A few seconds of tolerance is normal, but a 47-second lie should be fixed, not accommodated.

Finally, this approach is mostly irrelevant if you run on managed infrastructure with proper NTP baked in. The lesson still applies, but the fix is someone else's job.

The real lesson

The model API wasn't down, and the token wasn't invalid. The free server was simply living 47 seconds in the future, and every request carried the proof. The fix took four minutes once I asked the right question: what does this machine believe about time?

If you've ever debugged a 401 that only happens in one environment, I'd love to hear what the root cause was. Mine was a clock that refused to tell the truth, and it was hiding in plain sight.