After every deploy of our returns service in May, somewhere between ten and forty returns went missing. The customer had a label, the courier had the parcel, and we had no record of either.
The service accepts a return, answers the customer, and writes the record to Kafka through a producer that batches messages in memory. On shutdown it catches SIGTERM, stops taking requests, waits for those in flight and flushes the producer, which in the worst case takes about fifteen seconds. Its pods had the default terminationGracePeriodSeconds of thirty, and for two years that was plenty.
In April we had added a preStop hook that sleeps for twenty five seconds. It is common advice, and it fixed a real problem. Kubernetes removes a terminating pod from its Service endpoints at the same time as it starts shutting the pod down, so a pod that stops at once still receives requests for a few seconds and refuses them. The sleep keeps it serving until the rest of the cluster has stopped sending it traffic.
What we had missed is that the grace period is one budget for everything. The thirty seconds starts when the pod is marked for deletion, before the preStop hook runs, and the container only receives SIGTERM after the hook has finished. Our sleep used twenty five seconds, the application got SIGTERM with five left, and the kubelet sent SIGKILL when they ran out. Returns accepted in those last seconds had been confirmed to the customer and were still sitting in the producer's buffer. The pod status recorded exit code 137 for every terminating container, and we had read it as a slow shutdown rather than a killed one.
We found it by comparing a week of the courier's manifests with our own records and lining the gaps up against our deploy times.
The grace period is now sixty seconds, the preStop sleep plus the longest shutdown we have measured plus a margin, and a test in our chart fails if that sum stops fitting whenever either number changes. The return is now flushed to Kafka and acknowledged before the customer is told it is booked. And any container that exits with 137 during a rollout raises an alert.
We gave the load balancer more time and took it from the application without noticing. A pod has one clock for leaving, and everything it does on the way out shares it.
– Sergey Shinder
Top comments (0)