I Chased a Late Job for 48 Hours. The Timezone Database Was Missing.

Have you ever stared at a job log and decided the scheduler must be drifting, even though the host clock looked fine? I did that for the better part of two days, and I was looking at the wrong layer the entire time. The helper came from a free coding model, and it used zoneinfo instead of the old pytz crutch. That choice is correct on paper, but it still assumes a timezone database exists on the target machine.

I am writing this as forty-eight-hour field notes, not as a vendor scorecard or a fake benchmark dump. You can delete every product name below and still rerun the receipt on a laptop and a cheap VM. If you already pin tzdata in every image you ship, steal the checklist and ignore the surrounding narrative.

What I thought was broken

The job was a boring Python script that stamped events in America/New_York and wrote JSON lines to stdout. Local runs looked punctual, and the unit test compared aware datetimes, so I trusted the clock math without asking more. Then the server logs showed labels that did not match the wall clock I was watching from my chair. Was the queue delayed, was cron using another crontab, or was the box inheriting UTC while my laptop used a local zone?

I walked those questions in order, because they are the questions operators are trained to ask first during an incident. None of them were silly, and none of them named the actual fault hiding under the wrapper. The traceback I needed was being swallowed, which is how a missing data file impersonates a late scheduler for hours.

Hours 0–8: scheduler theater

I printed date on the host, then printed timedatectl where it existed, then printed date -u right beside it. I checked crontab for CRON_TZ, and I checked the systemd timer if the box even had systemd available. I looked at queue timestamps and tried to decide whether enqueue time or start time was the liar in the story. Everything said the machine believed it was in UTC, which is normal for servers, and still did not explain the labels in my JSON.

Commands I actually ran, in roughly this order, before I admitted they were not answering the right question:

date
date -u
echo "TZ=${TZ:-unset}"
python -c "import time, datetime; print(time.tzname); print(datetime.datetime.now().astimezone())"

Did any of that output actually prove the job was late according to the business timezone I care about? Those commands only proved that I could print some clock on the host, which is not the same investigation. I still had no receipt that compared two Python processes instead of two hunches.

Hours 8–20: logging theater

Could the logger be buffering and flushing a batch of lines after a delay that looked like lateness from the outside? That is a real class of bugs, and I have been burned by block buffering when stdout is not a TTY. I exported PYTHONUNBUFFERED=1, reran the job, and still saw the same zone labels in the JSON payload. I switched from print to logging with logging.Formatter and %(asctime)s, which quietly introduced a second clock.

Two clocks are not better than one, especially when neither clock mentions tzdata in the log line. Have you ever compared asctime against an ISO timestamp and then blamed NTP because the strings disagreed in format? I did, and the disagreement was naive local time versus an aware conversion that never finished constructing its zone.

Hours 20–36: interpreter theater

I then decided the second machine must be running an older Python that handled zoneinfo differently from my laptop interpreter. Does that sound familiar if you have ever mixed a random python3 with a forgotten virtualenv and trusted the prompt color? I printed sys.version and sys.executable on both machines and stared at them like they would confess. Both were 3.13-class interpreters, which meant zoneinfo was in the standard library either way.

Python 3.9 added zoneinfo, so a missing module was not the story I wanted it to be. The import succeeded on the server, and the constructor did not, which is a distinction I should have read twice. from zoneinfo import ZoneInfo can succeed while ZoneInfo("America/New_York") raises ZoneInfoNotFoundError, because the package is present and the database is not.

What actually broke

Here is the smallest job that reproduced the mess once I stopped decorating it with cron theories. Treat it as a labeled example you can save as stamp_job.py and run yourself on two interpreters.

"""Labeled example: a tiny stamper that looks correct until tzdata is missing."""
from __future__ import annotations

import json
import sys
from datetime import datetime, timezone
from zoneinfo import ZoneInfo


def stamp(now: datetime | None = None) -> dict:
    instant = now or datetime.now(timezone.utc)
    eastern = instant.astimezone(ZoneInfo("America/New_York"))
    return {
        "utc": instant.isoformat(),
        "eastern": eastern.isoformat(),
        "label": eastern.tzname(),
    }


if __name__ == "__main__":
    json.dump(stamp(), sys.stdout)
    sys.stdout.write("\n")

On my laptop this printed a normal pair of ISO timestamps and a daylight label I recognized from local news. On a minimal Linux image it exploded before a single line of JSON, or worse, it failed inside a wrapper that caught the exception and logged a vague job-failed line. Have you seen a wrapper swallow ZoneInfoNotFoundError and then page you about a timeout that never happened? I had, and that is why I wasted the first day on the scheduler.

A reproduction you can run without my machines looks like this, if you already have Docker on the laptop:

# Common failure on slim images that omit tzdata. Confirm on YOUR image.
docker run --rm python:3.13-slim \
  python -c "from zoneinfo import ZoneInfo; ZoneInfo('America/New_York')"

If that raises ZoneInfoNotFoundError, you have the same class of drift I had, not a unique cloud mystery. Installing tzdata inside the image, or pointing TZDIR at a real IANA tree, fixes the constructor without touching cron. Do not assume every free Linux server is a slim image, and do not assume every slim image will stay that way next month without checking.

The environment receipt I wish I had run at hour zero

I wanted one JSON document I could copy from the laptop and from the server, then diff without storytelling in Slack. The receipt below is the original artifact for this write-up, not a framework and not a platform. Save it as env_receipt.py and run it with the same interpreter you use for the job, because a second interpreter is how this class of bug hides.

"""Environment receipt: snapshot the clock-related facts a model will not see."""
from __future__ import annotations

import json
import locale
import os
import sys
import time
from datetime import datetime, timezone
from pathlib import Path

RECEIPT_VERSION = 1


def _zoneinfo_status(key: str) -> dict:
    try:
        from zoneinfo import ZoneInfo

        zi = ZoneInfo(key)
        now = datetime.now(timezone.utc).astimezone(zi)
        offset = now.utcoffset()
        return {
            "key": key,
            "ok": True,
            "tzname": now.tzname(),
            "utc_offset_seconds": int(offset.total_seconds()) if offset else None,
        }
    except Exception as exc:  # labeled: broad on purpose for a receipt
        return {
            "key": key,
            "ok": False,
            "error_type": type(exc).__name__,
            "error": str(exc),
        }


def build_receipt() -> dict:
    tzdir_candidates = [
        os.environ.get("TZDIR"),
        "/usr/share/zoneinfo",
        "/etc/localtime",
    ]
    existing = []
    for item in tzdir_candidates:
        if not item:
            continue
        path = Path(item)
        existing.append({"path": item, "exists": path.exists()})

    return {
        "receipt_version": RECEIPT_VERSION,
        "sys_version": sys.version,
        "sys_executable": sys.executable,
        "sys_platform": sys.platform,
        "cwd": os.getcwd(),
        "pid": os.getpid(),
        "stdout_encoding": getattr(sys.stdout, "encoding", None),
        "stderr_encoding": getattr(sys.stderr, "encoding", None),
        "stdin_isatty": bool(getattr(sys.stdin, "isatty", lambda: False)()),
        "stdout_isatty": bool(getattr(sys.stdout, "isatty", lambda: False)()),
        "preferred_encoding": locale.getpreferredencoding(False),
        "filesystem_encoding": sys.getfilesystemencoding(),
        "locale_env": {
            "TZ": os.environ.get("TZ"),
            "TZDIR": os.environ.get("TZDIR"),
            "LANG": os.environ.get("LANG"),
            "LC_ALL": os.environ.get("LC_ALL"),
            "PYTHONUNBUFFERED": os.environ.get("PYTHONUNBUFFERED"),
        },
        "time_tzname": list(time.tzname),
        "time_timezone": time.timezone,
        "time_daylight": time.daylight,
        "zoneinfo_paths": existing,
        "zones": [
            _zoneinfo_status("UTC"),
            _zoneinfo_status("America/New_York"),
            _zoneinfo_status("Etc/UTC"),
        ],
    }


if __name__ == "__main__":
    json.dump(build_receipt(), sys.stdout, indent=2, sort_keys=True)
    sys.stdout.write("\n")

Run it twice and keep the files under names that mention the machine, because future-you will attach the wrong log otherwise. I have done that mix-up, and it creates a second forty-eight-hour loop that feels productive while proving nothing.

python env_receipt.py > receipt.laptop.json
# copy the script to the other environment, then:
python env_receipt.py > receipt.server.json

I also wanted a boring differ that does not require a dashboard or a log vendor. Save this labeled example as compare_receipts.py and let it print drifted keys until you get bored of arguing.

"""Compare two environment receipts and print the keys that drifted."""
from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any


def flatten(prefix: str, value: Any, out: dict[str, str]) -> None:
    if isinstance(value, dict):
        for key, inner in value.items():
            next_prefix = f"{prefix}.{key}" if prefix else str(key)
            flatten(next_prefix, inner, out)
        return
    if isinstance(value, list):
        for index, inner in enumerate(value):
            flatten(f"{prefix}[{index}]", inner, out)
        return
    out[prefix] = json.dumps(value, sort_keys=True)


def main(left_path: str, right_path: str) -> int:
    left = json.loads(Path(left_path).read_text(encoding="utf-8"))
    right = json.loads(Path(right_path).read_text(encoding="utf-8"))
    flat_left: dict[str, str] = {}
    flat_right: dict[str, str] = {}
    flatten("", left, flat_left)
    flatten("", right, flat_right)
    keys = sorted(set(flat_left) | set(flat_right))
    drifts = []
    for key in keys:
        if key == "pid" or key.endswith(".pid"):
            continue
        if flat_left.get(key) != flat_right.get(key):
            drifts.append(
                (key, flat_left.get(key, "<missing>"), flat_right.get(key, "<missing>"))
            )
    if not drifts:
        print("receipts match on compared keys")
        return 0
    print(f"{len(drifts)} drifted keys:")
    for key, a, b in drifts:
        print(f"- {key}")
        print(f"  left : {a}")
        print(f"  right: {b}")
    return 1


if __name__ == "__main__":
    if len(sys.argv) != 3:
        print(
            "usage: python compare_receipts.py receipt.laptop.json receipt.server.json",
            file=sys.stderr,
        )
        raise SystemExit(2)
    raise SystemExit(main(sys.argv[1], sys.argv[2]))

Would I have caught tzdata with this at hour one instead of hour thirty-six, when I was already annoyed? Yes, because zones[1].ok would be false on the server receipt and true on the laptop receipt, which is a better signal than arguing with cron. The differ also surfaces encoding and TTY facts I did not need this time, and I still want them sitting in the file for the next wrong diagnosis.

Where a free model and a free server actually helped

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access to draft the first receipt script, then I ran that script on my laptop and on the free server option so I had two environments to diff. I am not claiming a particular CPU, quota, image family, duration, or model name here, because those details change and I did not measure them for this write-up. The useful part was ordinary: I needed a second interpreter that was not my laptop, and I needed a first draft of the checker that I could read line by line before trusting it.

If you already have a spare VM, use that second interpreter instead of anything I mentioned above. The receipt does not care who owns the other box, and the lesson survives if you strip the product sentence out of this section entirely.

Decision table I keep next to the job

I printed this table after the fact, because my brain kept returning to cron even when the receipt was already screaming. Keep it in the same folder as the JSON files so you have to trip over it.

• Symptom: the job looks late versus your watch. Read first: zones[*].ok and zones[*].tzname. Likely cause: missing tzdata or a wrong zone key. Next: install tzdata or emit UTC only.
• Symptom: logs appear after the job claims it finished. Read first: stdout_isatty and locale_env.PYTHONUNBUFFERED. Likely cause: block buffering when stdout is a pipe. Next: python -u or print(..., flush=True).
• Symptom: json.dump raises UnicodeEncodeError on a character the laptop accepted. Read first: stdout_encoding and preferred_encoding. Likely cause: an ASCII locale on the server. Next: PYTHONIOENCODING=utf-8.
• Symptom: the import works and the constructor fails. Read first: zoneinfo_paths and zones[*].error_type. Likely cause: the IANA database path is missing. Next: set TZDIR or install tzdata.
• Symptom: the offset is zero when you expected daylight saving. Read first: time_daylight and zones[*].utc_offset_seconds. Likely cause: Etc/UTC or a naive datetime leaking in. Next: stop mixing naive and aware clocks in one payload.

That list is not complete, and it is not a substitute for reading the traceback you actually got. It exists so I stop improvising a fourth theory before comparing receipts.

Test plan I would rerun tomorrow

1. Run stamp_job.py on the laptop and save stdout as stamp.laptop.json without editing the file by hand.
2. Run env_receipt.py on the laptop and save receipt.laptop.json beside the stamp file.
3. Copy both scripts to the second environment without “just fixing” a path that would hide the drift.
4. Run both scripts there and save stamp.server.json plus receipt.server.json using the same interpreter as the job.
5. Run compare_receipts.py and refuse to discuss lateness until zones matches or you can document why it should not.
6. If ZoneInfo("UTC") works but ZoneInfo("America/New_York") fails, install tzdata and rerun before you touch cron.
7. Add a unit test that calls ZoneInfo("America/New_York") at import time in CI, so the image cannot ship without the database.

A tiny CI guard looks like this labeled example, and it is allowed to be boring:

# tests/test_tzdata.py — labeled example
from zoneinfo import ZoneInfo


def test_new_york_zone_is_available():
    zone = ZoneInfo("America/New_York")
    assert zone.key == "America/New_York"

If CI uses a slim image, this test will fail for the right reason instead of paging you after a customer already compared two clocks. That is the kind of failure I want at minute five, not at hour forty when the wrapper has already renamed it as a timeout.

What I would repeat, and what I would not

I would repeat the receipt before I repeat the narrative, because the narrative is how I talked myself into cron. I would repeat storing UTC internally and converting only at the edge, because named zones need data files that minimal images skip without apology. I would repeat teaching the model a constraint in the prompt: emit UTC ISO strings, and do not call ZoneInfo unless the receipt says that key is ok.

I would not repeat mixing logging asctime with my own zone-aware stamps in the same JSON line. I would not repeat catching bare Exception in a job wrapper that hides ZoneInfoNotFoundError from the person who has to wake up. I would not repeat assuming a free server shares my laptop timezone files, because every slim Dockerfile is already trying to tell us that story is false.

Why do we keep assuming the second machine is a slightly slower copy of the first, when the second machine is often a smaller filesystem with different locale defaults? I needed that question at hour zero, and I asked it only after I had already rewritten a scheduler that was doing its job.

Limitations, and who should skip this

This receipt is a snapshot of one process, not a lockfile, and not a security audit you can wave at a reviewer. It will not tell you whether tzdata is current with a last-minute daylight-saving law change in a jurisdiction you serve. It will not fix historical timestamps you already stored as naive local strings, and it will not help if a human simply chose the wrong business timezone.

Skip this approach if you already build every image from a base that installs tzdata and you convert at the edge in UTC. Skip it if your runtime is Windows with a full IANA database and you never deploy to minimal Linux. Skip it if you need legal-grade zoned history; use a maintained time library and a process for database updates instead of a JSON dump I wrote for a weekend job.

The free model will happily generate ZoneInfo("America/New_York") because that string is common in examples and looks responsible in a code review. The free server will happily run whatever interpreter is on PATH, including one that imported zoneinfo from the standard library with no tzdata beside it. Neither side knows the other side's timezone package, and that gap is the whole story I spent forty-eight hours refusing to see.

Would I start with cron again the next time a stamp looks late on a machine I do not sit in front of? Not until the receipts match, or I can explain every drifted key out loud without guessing.