One Sunday in July no dispatch emails went out for seven hours. Parcels left the warehouse as usual, and customers who expected to be told about them rang support instead. The service that sends those emails was running the whole time, answering health checks and serving its admin pages.
The notifier runs as a single pod because it owns a schedule: every minute a task picks up newly dispatched orders and sends their emails. The same JVM serves a small admin API, and at ten past two someone ran its yearly shipments export, which loaded every row into a list. The heap ran out.
An OutOfMemoryError does not stop the JVM by default. It is thrown in whichever thread failed to allocate, like any other error, and if nothing catches it that thread ends and the rest carry on. It reached the export request, two HTTP worker threads and, at that moment, the scheduled task. The export's list was then collected, memory recovered, and the HTTP pool replaced its lost threads. The scheduler did not. A task run with scheduleAtFixedRate that throws is never run again, as its documentation says, and the error is stored in a ScheduledFuture that our code never read, so it was not even logged. The health endpoint checked the database and the disk and reported UP. Kubernetes had no reason to do anything, because nothing had died.
We found it at nine, from a thread dump with no notifier thread in it, and restarted the pod. The backlog went out within ten minutes.
The JVM now starts with ExitOnOutOfMemoryError, so the first such error ends the process and Kubernetes starts a fresh one in seconds. HeapDumpOnOutOfMemoryError writes a dump to a mounted volume before it exits, so we keep the evidence. The scheduled task catches everything, logs it and keeps its schedule. The health check fails if no run has finished in the last five minutes, which ties liveness to the work the service exists to do. And the export moved to its own job, reading in pages.
Our tooling was built for services that crash, and this one would have been back in seconds if it had. A process that loses part of itself and keeps answering is harder to deal with than one that dies.
– Sergey Shinder
Top comments (0)