At twelve forty on a Thursday in November our delivery quote service stopped answering. Requests went in and nothing came out. CPU was at two percent, memory was normal, the database was idle, and the health check still returned 200. Restarting the pods bought nine minutes.
A quote is calculated per basket. The service receives the basket, splits it into lines, works out each line's carrier options in parallel and then combines them. Until October the request handler did the splitting and a dedicated executor did the lines. A refactor that month tidied our thread pools into one shared, fixed executor of forty threads, configured in one place. It passed review and a load test.
After the refactor the basket work itself also ran on that executor. Each basket task submitted its line tasks to the same pool and then called get on each future, waiting for them. With a few baskets at a time that works: there are always free threads for the lines. At lunchtime, when trade customers build large baskets, there were more than forty baskets in flight. Every thread in the pool was a basket task waiting for its lines, and every line task was in the queue behind them, waiting for a thread. Nothing could finish, so nothing would ever free a thread. A thread dump showed forty threads parked in FutureTask.get and eleven hundred tasks queued behind them.
The health check runs on the web server's own threads, so it never noticed. Our load test had sent small baskets at a steady rate and never had forty large ones open at once.
We went back to two pools within the hour, one for baskets and one for lines, so a waiting task can never occupy the threads it is waiting for. The code has since stopped blocking altogether: line results are combined with CompletableFuture composition instead of get. Every remaining blocking wait has a timeout. Each executor exports its active threads, queue length and the age of the oldest queued task, and we alert when that age passes five seconds. When the alert fires, the pod writes a thread dump before anyone restarts it. The load test now includes the lunchtime mix of basket sizes.
A bounded pool is a promise that work will wait its turn. When the work waits for other work in the same pool, the turn never comes.
– Sergey Shinder
Top comments (0)