The sidecar left before the application had finished

We put a service mesh in and acquired three failure modes I had not thought to look for, all of them about the order in which two processes inside one pod start and stop.

The first showed up at every deploy as a small band of 502s in the last few seconds of each pod's life. On termination the kubelet sends SIGTERM to every container at once. Our application handled it properly: stop accepting new work, finish what is in flight, exit. The proxy handled it properly too, by shutting down straight away, which meant the requests our application was carefully draining had nothing left to leave through. Every graceful shutdown we had written was draining into a closed door.

The second was the same problem reversed. The application container starts calling a dependency immediately, and for the first two or three seconds the proxy has not received its configuration, so those calls get connection refused. Long lived services retry and nobody notices. Something that runs once and exits does not retry.

Which is the third. Our nightly reconciliation job finished its actual work in four minutes and then sat there. A Job is complete when its containers exit, and the proxy never exits. Pods accumulated, the concurrency policy blocked the following run, and reconciliation silently stopped for two days while the dashboard showed the job as Running.

All three fixes are lifecycle, not networking. The proxy has a preStop hook that sleeps long enough for the application to drain, so the order of departure matches the order of dependence. The application waits for the proxy's readiness endpoint before its first outbound call, which on our version is a mesh setting rather than code. Jobs call the proxy's quit endpoint from a wrapper once the main process returns, and where the cluster version allows it we moved the proxy to a native sidecar container, which the kubelet stops after the main container is done.

We also alert when a Job runs past twice its historical median, which would have found the third problem in an hour rather than in two days.

Two containers in a pod share an address, not a lifecycle. Somebody has to decide who starts first and who is allowed to leave last.

– Sergey Shinder