DEV Community

Sergey Shinder
Sergey Shinder

Posted on

We moved our cut-off service to a scratch image and it started working in UTC

Our delivery promise service is a small Go program that answers one question for the website: if this order is placed now, which day will it arrive. The answer depends on the depot's cut-off, five o'clock UK time. In June we moved it from a Debian based image to FROM scratch with a statically linked binary. The image went from ninety megabytes to eleven, and every package our scanner had been complaining about disappeared.

The service loads Europe/London at start with time.LoadLocation. Go looks for the time zone database in the operating system's zoneinfo directories, among a few other places, and a scratch image has no operating system. LoadLocation returned an error. Years earlier someone had written the obvious defensive code around that call: log a warning and fall back to UTC so the service can still start. It started, the health check passed, and the warning went into a log nobody read.

In June the UK is on British Summer Time, an hour ahead of UTC. Five o'clock in UTC is six o'clock in London. For three weeks, every order placed between five and six in the evening was promised next day delivery from a depot that had already closed for the day. Roughly four hundred customers a week were told tomorrow and received their parcel the day after. Customer service found the pattern before we did, because every complaint came from the same hour.

The fix inside the binary was one line. Importing the time/tzdata package, available since Go 1.15, embeds the time zone database in the program for a few hundred kilobytes. We also removed the fallback. A service whose whole job is local cut-off times now refuses to start if it cannot load its zone, because a crash on deploy is found in minutes and a wrong answer took three weeks. The cut-off tests now run inside the built image, with a fixed clock set once to a summer date and once to a winter date.

A minimal image is minimal about everything, including the files your code assumed every machine would have.

– Sergey Shinder

Top comments (0)