There's a specific, recurring pain point in production incidents that has nothing to do with the actual bug: reconstructing what order things happened in across services that log in different timezones, or worse, in local time that shifts twice a year with no indication in the log line itself.
The problem with local-time logs
A log line that reads 2026-03-08 02:30:15 - job started looks precise until you need to correlate it with a log line from a different service, running in a different timezone, or generated on a server whose timezone configuration silently changed at some point in its history. Without an explicit UTC offset baked into every timestamp, you're relying on institutional knowledge, someone remembering which timezone that particular server logs in, which is exactly the kind of tribal knowledge that doesn't survive team turnover or a service migration.
This gets actively dangerous around DST transitions specifically. A local-time log timestamp of 02:30 on a fall-back date is genuinely ambiguous, it could be either of two distinct instants an hour apart, and nothing in a bare local-time log line tells you which one actually happened. During an incident, when you're trying to establish a precise sequence of events across multiple systems, that ambiguity can send you down the wrong path entirely, especially if the incident happens to span the transition itself.
UTC removes an entire category of ambiguity
UTC timestamps have no DST transitions and no ambiguous hours, which means a UTC log timestamp always refers to exactly one instant, unambiguously, regardless of what timezone any given reader or system happens to be in. This is the entire argument for logging in UTC everywhere: not that UTC is more "correct" in some abstract sense, but that it's the one timezone every system in your stack can agree on without any additional context, which makes cross-service correlation during an incident dramatically faster.
The practical pattern is straightforward: store and transmit all timestamps in UTC, and only convert to local time at the very last step, in whatever UI or report a human is actually reading. Any conversion that happens earlier than that, especially anything that happens before the data is stored or logged, reintroduces the exact ambiguity UTC was supposed to eliminate.
Where teams get this wrong in practice
The most common mistake isn't malicious or even really a mistake in judgment, it's inconsistency. One service logs in UTC because someone set it up carefully. Another logs in whatever the server's default OS timezone happens to be, because nobody explicitly configured it either way. A third converts to local time somewhere in its logging middleware, for readability, without anyone realizing that conversion is happening at all. None of these individually look wrong in isolation. Together, during an incident that spans all three services, they produce a picture of "what happened when" that's actively misleading, and debugging that mismatch consumes exactly the kind of time you don't have during a live incident.
A concrete audit worth running this week
Pick any three services in your stack that log independently and pull a sample of log lines from each. Check whether the timestamps are explicit about their offset (either Z for UTC or an explicit +00:00 / -05:00 style offset) or whether they're bare local timestamps with no timezone information attached at all. Bare timestamps without an offset are the actual risk, since there's no way to disambiguate them after the fact if the underlying server's timezone configuration ever changes, or if you're reading the log from a different region than where it was generated.
Structured logging makes this a one-time fix, not an ongoing discipline problem
If your logging goes through a structured logging library rather than raw string formatting, this is usually a one-line configuration change at the library level rather than something every individual log call needs to remember to do correctly. Fixing it at that layer converts "everyone needs to remember to use UTC" into "the logging library only knows how to produce UTC," which is a meaningfully more durable fix than a team convention that depends on everyone remembering it correctly, indefinitely, across every new hire and every new service.
Libraries like structlog make this explicit by letting you attach a UTC timestamp processor once, at configuration time, so every log call downstream inherits the same behavior automatically. The point isn't that any particular library is required, it's that pushing the decision into shared configuration removes it from the set of things individual engineers have to get right on every single log call they write.
Clock skew turns "correct" timestamps into a false picture
Even when every service in your stack logs in UTC correctly, you can still get a misleading sequence of events if the underlying system clocks disagree with each other. A client machine whose clock is four seconds ahead of a server's clock will produce a UTC timestamp for its request that is, correctly, four seconds ahead of when the server actually received it. Neither timestamp is wrong. They're just measuring against two different physical clocks that have drifted apart.
This matters most when you're reconstructing causality during an incident, deciding whether event A on one machine happened before or after event B on another. If the two machines disagree by even a couple of seconds, a genuinely sequential pair of events can appear to happen in the wrong order, or even appear to overlap when they didn't. Most production environments run NTP or a similar time-sync service specifically to keep this drift small, but it's worth confirming that's actually enabled and healthy on every host, not just assumed, since a host that silently stopped syncing will drift further every day it goes unnoticed.
Log aggregation tools can quietly reintroduce the ambiguity
A UTC timestamp stored correctly at the source can still confuse an on-call engineer once it reaches the log viewer, because most aggregation dashboards convert timestamps to the viewer's local browser timezone for display by default. That's a reasonable convenience for a single person browsing logs at their desk, but it becomes a real problem the moment two engineers in different timezones are looking at the same incident and reading their timestamps out loud to each other on a call.
The underlying data isn't ambiguous, only the display layer is. The practical fix is to explicitly switch the viewer to UTC display mode during any incident that involves people in more than one timezone, and to make that switch part of your incident response checklist rather than something someone has to remember to ask for. It costs nothing and it removes an entire class of "wait, are we talking about the same event" confusion mid-incident.
A concrete failure pattern this prevents
One specific incident type this discipline heads off is a race condition that only becomes visible when comparing timestamps across two microservices. Imagine a payment service writes a "charge succeeded" event and, moments later, an inventory service writes a "stock reserved" event that's supposed to happen only after the charge clears. If those two services log in different timezones, or one of them silently drifts to local time during a deploy, an engineer reviewing the incident afterward can misread the actual order of operations and conclude the race condition doesn't exist, when it does, or conclude it exists when the systems actually behaved correctly. UTC timestamps alone don't fix the race condition itself, but they make it possible to establish the true order of events quickly enough to trust the rest of the investigation.
This connects to a broader scheduling reliability problem worth knowing about if your systems run recurring jobs: daylight saving transitions cause a related but distinct class of scheduling bugs, where a job scheduled against local wall-clock time can run twice, or not at all, on the two transition dates each year. The UTC-everywhere logging discipline described here doesn't fix that scheduling problem directly, but it makes diagnosing it dramatically faster when it does happen, since you're no longer fighting timestamp ambiguity on top of the actual bug.
For more on the standards involved, RFC 3339's specification for date and time on the internet defines the explicit-offset timestamp format that most structured logging libraries default to today, and the ISO 8601 standard overview on Wikipedia covers the broader family of timestamp formats this all builds on if you want the fuller picture.
At 137Foundry, UTC-first logging is one of the smaller, less glamorous conventions we build into every client system from day one, precisely because it's cheap to set up correctly at the start and genuinely painful to retrofit after a few years of inconsistent logs have already accumulated.
Top comments (0)