Our booking service decides which carriers it can book with from a file, carriers.json, that lists each carrier's endpoint and the services it offers. To keep one image for every environment, the file is not in the image. When we moved the service to Kubernetes, the pod spec gained a postStart hook that fetched it from our configuration service into an emptyDir volume:
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "curl -sf $CONFIG_URL/carriers.json -o /etc/booking/carriers.json"]
The service reads the file once, when it starts. If the file is missing, it logs a warning and falls back to a copy of the list built into the image, which was meant for local development and had last been updated in the spring.
In October we added Evri. The configuration service had the new list on a Tuesday morning, a rollout restarted every pod, and over the next two days about six hundred parcels failed to book with Evri and were labelled by hand in the warehouse. The warehouse system retries a failed booking twice, so most Evri parcels went through. The ones that did not were the ones that landed on the same kind of pod three times.
Seven of our twenty eight pods had started with the built in list. Their first log line said so, at warning level.
Kubernetes runs a postStart hook immediately after it creates the container, and the documentation is plain that there is no guarantee the hook runs before the container's entrypoint. The two start together. For a year the download took under a tenth of a second and the service read the file about a second after it started, so the hook always won. In September the configuration service moved behind our internal gateway, which added a token exchange, and the download began to take anything from half a second to two. On most pods it still finished first. On about one in four it did not.
A hook that fails gets its container killed, so a download that broke would have been loud. A slow one simply lost the race, and the pod went on to pass every probe, because a booking service that does not know about Evri is still a healthy booking service.
The download is now an init container. Init containers run to completion, one after another, before any application container starts, and the service does not start until the download has succeeded. The built in list is gone, and the service refuses to start without the file. Its readiness endpoint reports a hash of the carrier list it loaded, and the deploy fails if the pods report more than one.
postStart sounds like after the start. It means at the same time as the start, and we had used it for a step that had to come first.
– Sergey Shinder
Top comments (0)