Our delivery service sends customers a text with their two hour delivery window on the morning of delivery. The text is sent asynchronously: the service marks a method @async and Spring runs it on an executor we define as a bean, a ThreadPoolTaskExecutor with setMaxPoolSize(50) and a thread name prefix. We had sized that number carefully against our SMS provider's rate limit.
On a Tuesday in October the provider had a slow morning, with each call taking about three seconds instead of a hundred and fifty milliseconds. By ten o'clock the last texts for the morning window were going out at half past eleven, after many drivers had already knocked and left a card. A thread dump showed one thread with our prefix. It had always been one.
ThreadPoolTaskExecutor defaults to a core size of one, a maximum of Integer.MAX_VALUE and a queue capacity of Integer.MAX_VALUE. A Java thread pool only adds threads above the core size when its queue is full and refuses a task. Our queue had no limit, so it never refused one, and the maximum of fifty was never reached. On a normal day one thread at a hundred and fifty milliseconds a call sends four hundred texts a minute, which is more than we need, so nothing ever showed it. At three seconds a call it sent twenty.
That morning we set the core size to fifty and restarted, and the queue drained in about ten minutes. Then we looked at what else the queue had been hiding. It lived in memory, so a restart during a backlog would have dropped every text in it without a trace. Notifications are now written to a RabbitMQ queue and sent by consumers whose concurrency we set explicitly. The executors that remain have equal core and maximum sizes and a bounded queue, and they are registered with Micrometer, so active threads and queue length are on the service dashboard with an alert on the age of the oldest queued task.
The fifty we had configured was a maximum, and the pool would only grow towards it once the queue was full. We had given it a queue that could never fill.
– Sergey Shinder
Top comments (0)