Here at Nylas, we’re building an API to help developers build apps on top of email, calendar & contacts. As it turns out, doing so teaches you a lot about time changes in the Western Soviet Union. I joined Nylas just a few months back, and learned early on that one important aspect of setting up an event in a calendar is understanding what timezone that event is in, relative both to the creator of the event, as well as to any other participants. Naturally, this involves lots of timezone math. Most of this is done by the pytz library, and this works well enough for us. Why do yourself what someone else has already done, right?
Enter Microsoft. A lot of companies use Microsoft Exchange for their email and work calendar. Historically, Exchange has used a protocol called Exchange Active Sync (EAS). As with any Microsoft protocol, their attitude towards their calendar API seems to have been, “let’s provide all the raw data and make the user do something useful with it.” What they provide is:
- The offset from UTC in minutes
- The date during that calendar year when the timezone transitions to Daylight Saving Time
- The date during that calendar year when the timezone transitions to Standard Time
- The offset during Daylight Saving Time
…From which you are to derive a friendly name to show the user. It’s like if I asked you for the time, and you responded by telling me about Earth’s prograde rotation and tilt, the eccentricity of Earth’s orbit, the tidal effects of the moon on the rotational velocity, and how to adjust for leap seconds. I mean, sure, but can’t you just tell me it’s 5:30?*
Pytz has a fairly sparse public API (where “public” is defined using the Python idiom of classes and methods not prefixed by an underscore). You, the programmer, instantiate a timezone object from a friendly string like US/Eastern, and you can use the localize, normalize, and utcoffset methods to work with it. Pytz takes care of keeping track of DST transitions and offsets. From a normal user’s perspective, this makes sense. Pytz’s whole value proposition is to abstract away the pain that is dealing with timezones and the bureaucracy and politics about changing offsets. But ok, Redmond, game on.
We decided to use pytz internals to build a lookup table to store these. We need to use the information we get from Microsoft as the key, and the timezone name as the value. For each timezone, pytz stores an attribute _utc_transition_times
>>> tz = pytz.timezone('US/Eastern')
>>> pprint(tz._utc_transition_times[-5:])
[datetime.datetime(2035, 11, 4, 6, 0),
datetime.datetime(2036, 3, 9, 7, 0),
datetime.datetime(2036, 11, 2, 6, 0),
datetime.datetime(2037, 3, 8, 7, 0),
datetime.datetime(2037, 11, 1, 6, 0)]
Oh cool, so we can use this to build the lookup table., right? We’ll take the first transition of a given year as the transition to DST, the second transition of a given year as the transition to ST, pick a time outside of DST and look up the UTC offset, and store a tuple of this as the key for our table!
>>> daylight_time = tz._utc_transition_times[n]
>>> standard_time = tz._utc_transition_times[n+1]
>>> utc_offset = tz.utc_offset(date_not_in_dst)
>>> timezone_table[(daylight_time, standard_time, utc_offset)] = timezone_name
This worked well, until we got a bug report that calendar events in Auckland were losing timezone information. Looking into this, it appeared that our lookup table had the reverse daylight_time and standard_time entries. How could this be? Well, it turns out that we made a very bad assumption when we assumed that the first transition of a year is a transition to DST. Because the Southern Hemisphere’s summer is from December to March, they go into DST around November and exit around April of the following year. Maybe the Cartographers for Social Equality were right! We hadn’t realized this difference between the Northern and Southern Hemispheres, and were flawed because of it. Back to the drawing board.
Digging into Pytz
Ok, so a little more digging into Pytz is warranted. How exactly does this library work? Turns out, it’s really fascinating. Bundled with the codebase is the Olson database, otherwise known as the Zoneinfo or tz database. The whole database is written Unix-style, with plaintext files describing the timezone names and historical offsets for each geographic region. But even more prominent than the actual data are all the historical anecdotes and context scattered throughout as comments. Some of these are big, some small, and some seemingly only there as inside jokes between the maintainers.
For instance, on Alaska:
From Paul Eggert(2017-06-15):
...Many of Alaska's inhabitants were unaware of the US acquisition of Alaska, much less of any calendar or time change. However, the Russian-influenced part of Alaska did observe Russian time, and it is more accurate to model this than to ignore it. The database format requires an exact transition time; use the Russian salute as a somewhat-arbitrary time for the formal transfer of control for all of Alaska. Sitka's UTC offset is -9:01:13; adjust its 15:30 to the local times of other Alaskan locations so that they change simultaneously.
From Paul Eggert (2014-07-18):
One opinion of the early-1980s turmoil in Alaska over time zones and daylight saving time appeared as graffiti on a Juneau airport wall:
"Welcome to Juneau. Please turn your watch back to the 19th century."
See: Turner W. Alaska's four time zones now two. NY Times 1983-11-01 http://www.nytimes.com/1983/11/01/us/alaska-s-four-time-zones-now-two.html
And on Michigan:
From Paul Eggert (1999-03-31):
Shanks writes that Michigan started using standard time on 1885-09-18, but Howse writes (pp 124-125, referring to Popular Astronomy, 1901-01) that Detroit kept local time until 1900 when the City Council decreed that clocks should be put back twenty-eight minutes to Central Standard Time. Half the city obeyed, half refused. After considerable debate, the decision was rescinded and the city reverted to Sun time. A derisive offer to erect a sundial in front of the city hall was referred to the Committee on Sewers. Then, in 1905, Central time was adopted by city vote.
This story is too entertaining to be false, so go with Howse over Shanks.
Timezones are political—who knew? Jokes aside, I wanted to see how pytz actually deals with time. This happens in the localize method on the timezone class. localize attempts to cover all the bases for weird timezone transitions. To do this, it bisects _utc_transition_times to find where the given time falls in the list of transitions. Then, it retrieves information about the transition from _transition_info.
>>> tz = pytz.timezone('US/Eastern')
>>> pprint(tz._utc_transition_times[:5])
[datetime.datetime(1, 1, 1, 0, 0),
datetime.datetime(1901, 12, 13, 20, 45, 52)
datetime.datetime(1918, 3, 31, 7, 0),
datetime.datetime(1918, 10, 27, 6, 0),
datetime.datetime(1919, 3, 30, 7, 0)]
>>> pprint(tz._transition_info[:5])
[(datetime.timedelta(-1, 68640), datetime.timedelta(0), 'LMT'),
(datetime.timedelta(-1, 68400), datetime.timedelta(0), 'EST'),
(datetime.timedelta(-1, 72000), datetime.timedelta(0, 3600), 'EDT'),
(datetime.timedelta(-1, 68400), datetime.timedelta(0), 'EST'),
(datetime.timedelta(-1, 72000), datetime.timedelta(0, 3600), 'EDT')]This tells a much better story. For each entry in _utc_transition_times, there is a corresponding entry in _transition_info. The DST offset in that entry tells us whether it is a transition to ST (if it’s zero) or DST (if it’s not). Further, the UTC offset is there to use directly, rather than requiring a reference time to test against.
So what are the edge cases here? Well, it turns out that when you use information about a time zone to identify a name, there are multiple possible entries. It may not surprise you entirely to learn that there are different names for the time zone representing both Moscow and Turkey. But if you’re not careful, it’s easy to overwrite “Europe/Moscow” or “Europe/Istanbul” (the common representations of this time zone) with “W-SU” (for Western Soviet Union).
Additionally, just looking at the localize method shows a whole host of edge cases. There is the seemingly obvious case of trying to deal with a time within a transition, e.g. is 1:30 AM on the day of a transition from DST to ST the time before or after the transition? But there are other cases as well, when countries wind their clocks back outside of a DST transition. This can cause problems for anyone trying to do real time conversions between timezones.
Luckily, that is the sort of thing we can rely on pytz for. While not perfect, pytz covers about as many edge cases as possible when dealing with one of the worst issues to program around. While we need to work with the finer points of timezone transitions to decipher Exchange events, we can usually just use the fine pytz API elsewhere.
If you’d like to learn more about the Nylas Calendar API, you can get a free API key by registering here.
*I put this analogy in specifically so that I could glare at the first person to ask me what time zone it’s 5:30 in.
Top comments (0)