On the first working day of October about forty thousand of our customers received their monthly statement email twice, and a few thousand received it three times. The statement job had not been changed in a year. Our customer base had grown by a quarter since the spring.
The job runs once a month as a Kubernetes Job. It reads every account, renders a PDF, and calls our email provider for each customer in turn. Its pod had a memory limit of two gigabytes, which had been plenty when the account list was smaller. It also kept a summary of every statement sent in memory for a report at the end, and that grew with the customer count. In October it reached the limit about seventy percent of the way through. The kernel killed the container, the pod exited with code 137 and was marked failed.
A Job exists to reach a number of successful completions, and failure is something it is built to recover from. backoffLimit defaults to six, so when a pod fails the Job controller creates a new one, waiting ten seconds before the first retry and doubling the wait each time. Our new pod had no idea what the old one had done. It started at the first account, sent seventy percent of the statements again and was killed at the same point. The third attempt was most of the way through the same customers when somebody in support asked why their own statement had arrived twice, and we deleted the Job.
Nothing in that was a fault in Kubernetes. A Job's promise is that the work will finish, not that it will run once, and we had written something that was only safe to run once.
The job now records each sent statement in a table with a unique key on account and month, and checks it before sending, so a second attempt resumes where the first stopped. backoffLimit is set to zero on every Job in our cluster that talks to customers, until someone can show that running it twice is harmless. Batch jobs now alert on memory against their limit, as services do.
Retry is the default for a Job. If the work cannot be repeated, the code has to know where it stopped or the Job has to be told not to try.
– Sergey Shinder
Top comments (0)