Have you ever watched a timestamp parse on your laptop and explode on the worker with the same bytes? I just burned forty-eight hours on that exact mismatch, and the culprit was one trailing Z. The payload looked like every other ISO-8601 timestamp I had already shipped through this service. This is the field notebook I wish I had opened at hour zero instead of blaming the broker.
Hour 0–8: I treated it like a network problem
Why would a datetime field fail only after the job crossed a queue boundary? The producer logged a clean ISO string, the consumer logged a ValueError, and both hosts claimed they were “on Python 3.” I assumed clock skew, a truncated body, or a proxy that rewrote headers in flight. None of those guesses survived the first packet capture, because the JSON on the wire was intact.
Here is what the producer actually emitted, copied from the dead-letter payload at hour three:
{
"event_id": "6c2f1e0a-9b44-4d1c-8a77-2f0b91c4aa10",
"occurred_at": "2026-09-20T14:07:33Z"
}
And here is the one-liner that stayed green on the laptop:
from datetime import datetime
raw = "2026-09-20T14:07:33Z"
print(datetime.fromisoformat(raw))
On 3.12 that prints a timezone-aware value and you keep walking. On 3.10 the same call raises ValueError: Invalid isoformat string: '2026-09-20T14:07:33Z'. Did I check python -V on both sides before hour eight? I did not, and that is the whole story.
Hypotheses I wasted a morning on
- The broker was stripping the
Tor the offset during re-encode. -
json.loadswas feeding me bytes and I was parsing arepr. - A middleware defaulted missing timestamps to
datetime.utcnow()and then re-serialized badly. - The worker used a C extension that pinned an older datetime module.
- CI “green” meant the parser tests had never seen a Zulu suffix at all.
Notice the pattern? Every guess assumed the language runtime was identical. It was not.
Hour 8–24: pin the interpreter, then pin the string
I finally printed versions next to the exception, which is the cheapest debug print I skipped:
python -V
python -c "import sys,datetime; print(sys.version); print(datetime.datetime.fromisoformat.__doc__.splitlines()[0])"
The laptop answered 3.12. The worker image, frozen months earlier, answered 3.10. That is not a rumor; Python 3.11’s what’s-new notes expanded datetime.fromisoformat(), and the 3.11 library docs are explicit that more ISO-8601 shapes are accepted, including Z. The 3.10 docs still describe the older, narrower parser.
Would strptime have saved me? Only if I put a literal Z in the format string, which then rejects +00:00. Would dateutil.parser.isoparse have saved me? Yes, at the cost of another dependency and a looser grammar than I wanted. I wanted one function, two interpreters, and a loud failure on naive values.
What actually broke
fromisoformat() is not “the ISO-8601 parser.” It is “the parser for strings that isoformat() would emit,” plus whatever extra shapes that exact CPython version decided to teach it. A trailing Z is UTC in every API I consume, and it is still not UTC on 3.10’s stdlib path. Mixing those two facts in one codebase is how you get a forty-eight hour ghost.
There is a second footgun hiding behind the first one. Even after you accept the string, a naive datetime will compare and subtract without protest, then explode later in a timezone-aware pipeline. So the bug is not only “3.10 rejects Z.” The bug is “the happy path never asserted tzinfo.”
The shim I would repeat
I now parse at the boundary, normalize Z/z to +00:00 (valid since 3.7), and refuse naive values. I also keep timezone.utc instead of datetime.UTC, because the latter only arrived in 3.11 and would reintroduce the skew I was fixing.
# parse_api_datetime.py
from __future__ import annotations
from datetime import datetime, timezone
class TimestampParseError(ValueError):
"""Raised when an API timestamp cannot be normalized to UTC."""
def parse_api_datetime(value: str) -> datetime:
if not isinstance(value, str) or not value.strip():
raise TimestampParseError(f"empty timestamp: {value!r}")
text = value.strip()
if text.endswith(("Z", "z")):
text = text[:-1] + "+00:00"
try:
parsed = datetime.fromisoformat(text)
except ValueError as exc:
raise TimestampParseError(f"unreadable timestamp: {value!r}") from exc
if parsed.tzinfo is None:
raise TimestampParseError(f"naive timestamp rejected: {value!r}")
return parsed.astimezone(timezone.utc)
Is the z branch overkill? Maybe, but I have already seen one partner emit lowercase Zulu. Is replacing only the last character enough for 2026-09-20 14:07:33Z with a space? No, and I do not pretend it is. On 3.10, fromisoformat still wants T, so that input must fail loudly unless I also normalize the separator. I chose loud failure over a second silent rewrite.
Reproducible test file, not a screenshot of a green bar
Drop this next to the shim and run it on every interpreter you still ship.
# test_parse_api_datetime.py
from datetime import datetime, timezone, timedelta
import unittest
from parse_api_datetime import parse_api_datetime, TimestampParseError
class ParseApiDatetimeTests(unittest.TestCase):
def test_zulu_becomes_utc(self):
got = parse_api_datetime("2026-09-20T14:07:33Z")
self.assertEqual(got, datetime(2026, 9, 20, 14, 7, 33, tzinfo=timezone.utc))
def test_offset_preserved_then_folded(self):
got = parse_api_datetime("2026-09-20T16:07:33+02:00")
self.assertEqual(got, datetime(2026, 9, 20, 14, 7, 33, tzinfo=timezone.utc))
def test_fractional_seconds(self):
got = parse_api_datetime("2026-09-20T14:07:33.250Z")
self.assertEqual(got.microsecond, 250000)
def test_naive_is_rejected(self):
with self.assertRaises(TimestampParseError):
parse_api_datetime("2026-09-20T14:07:33")
def test_space_separator_is_rejected_on_purpose(self):
with self.assertRaises(TimestampParseError):
parse_api_datetime("2026-09-20 14:07:33Z")
def test_empty_is_rejected(self):
with self.assertRaises(TimestampParseError):
parse_api_datetime(" ")
if __name__ == "__main__":
unittest.main()
Commands I actually re-run, instead of trusting a single pytest from my PATH:
python3.10 -m unittest test_parse_api_datetime.py -v
python3.12 -m unittest test_parse_api_datetime.py -v
If you only have one interpreter locally, run the file in a second environment before you call the bug “fixed.” That is the whole matrix. Two greens beat one green with a story attached.
Decision table I keep above the parser
| Input | 3.10 fromisoformat
|
3.11+ fromisoformat
|
This shim |
|---|---|---|---|
2026-09-20T14:07:33Z |
ValueError |
aware UTC | aware UTC |
2026-09-20T14:07:33+00:00 |
aware UTC | aware UTC | aware UTC |
2026-09-20T14:07:33 |
naive | naive | rejected |
2026-09-20 14:07:33Z |
ValueError |
often accepted | rejected on purpose |
2026-W38-6 |
ValueError |
may parse | rejected |
If a row in that table ever surprises you, the test file is wrong, not the API partner.
Hour 24–48: a second environment, then a version gate
Around hour twenty I wanted another interpreter that was not my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access to review the shim for 3.10-only traps such as datetime.UTC, and I ran the same test file through the free server option so I was not debugging against a single Python. That did not replace CI. It only stopped me from declaring victory after one green local run.
What did the models miss until I pasted the 3.10 traceback? They kept suggesting datetime.fromisoformat as if the current docs were the only docs. That is a useful limitation, not a scandal. Stdlib behavior is versioned, and an assistant that reads today’s page will happily hide yesterday’s worker.
I then added an explicit gate so the next image rebuild cannot drift back to “whatever the distro shipped”:
python -c "import sys; assert sys.version_info >= (3, 10), sys.version"
If your fleet is already 3.11+, raise that floor and delete the Z rewrite. Do not keep a compatibility shim as folklore once every runtime has the parser you wanted.
What I would repeat, and what I would not
I would repeat these steps in this order, because skipping any of them reopened the incident in my notes:
- Print
sys.versionnext to every parse error, on producer and consumer. - Save one raw payload byte-for-byte; do not retype the timestamp from memory.
- Run the same unittest module on every interpreter you still ship.
- Reject naive datetimes at the boundary, even when
fromisoformatwould accept them. - Prefer
timezone.utcin shims that must run on 3.10. - Delete the shim once the lowest runtime is 3.11 and the tests still pass.
I would not “fix” this with default=str on json.dumps, because that writes 2026-09-20 14:07:33+00:00 with a space and trains the next parser to accept junk. I would not call date.isoformat() and then slap a Z on a naive date. I would not normalize week dates or ordinal dates in this function; that is a different grammar.
Limitations, and who should not use this
This approach is a boundary parser for API timestamps shaped like YYYY-MM-DDTHH:MM:SS[.fff][Z|+HH:MM]. It is not a full ISO-8601 implementation, and it will not read 2026-W38-6, 2026-263, or bare dates with a UTC assumption. If you need that grammar, use a dedicated library and still assert tzinfo.
Who should skip the shim entirely? Anyone who already requires 3.11+ in every image and proves it in CI. Anyone parsing log lines that mix locales, timezones without offsets, or human dates. Anyone tempted to copy this into a browser runtime and call it “the same Python.” It is not.
The forty-eight hour lesson was smaller than my ego wanted it to be. The letter Z was valid UTC. The laptop was a newer CPython. The worker was not. After that, every green parse without a version next to it looks like a rumor. If you keep a spare environment around, even a free one, run the same test file there before you trust a local interpreter that has never seen the worker’s ValueError.
Top comments (0)