Description:
A Spring Boot service slowed down under load because API calls and database calls shared one thread pool. How the team found it, fixed it, and why virtual threads would not have helped.
Load went up, and the service slowed down with it. p99 latency climbed from 400 ms to 18 s. CPU sat around 20%. Connection timeout errors started showing up in the logs.
A slow service usually has something working too hard. This one did not. Requests were piling up, yet very little real work was getting done. Most of the time was going to waiting, and the engineers had to find out what for.
What the service was doing
Each unit of work was a short chain of blocking calls. Some went to external APIs. Some went to the database. A single unit could touch both, more than once, one after another.
The service ran many units at the same time with CompletableFuture. Every stage of every unit ran on one shared thread pool of 100 threads. Database access went through a Hikari connection pool with 20 connections.
Both kinds of calls block the thread that makes them. A thread that sends an HTTP request sits idle until the response comes back. A thread that needs the database first asks Hikari for a connection. If none is free, it waits. Then it waits again for the query. While a thread waits, it does no useful work. It is also unavailable for anything else.
So the 100 threads were one shared budget for two different kinds of waiting. When the APIs were fast and connections were free, the budget was enough. When either side slowed down, it spent the budget for both.
That design was the team's own choice, and it was easy to set up. The problem was that 100 did not describe any real limit in the system. It was just a number.
Finding the cause
The team took thread dumps during the slowdown and lined them up with Hikari's metrics. Nearly every pool thread was blocked. Some were waiting on HTTP responses. Many more were parked inside getConnection. Hikari showed all 20 connections in use and up to 60 threads waiting for one.
A thread could get stuck in two ways, and each made the other worse.
The first was slow API calls. Under load, external calls took longer and held their threads longer. That left fewer threads for every other stage, including database work, which sat in the executor's queue.
The second was the database. With 100 threads and only 20 connections, most of the pool could end up waiting inside Hikari. A thread waiting for a connection is just as unavailable as one waiting on HTTP. API stages then queued behind them.
Because both kinds of work drew from the same threads, slowness on one side became delay on the other. Every unit of work waited in the same queue, whatever resource it needed next.
The root cause was a mismatch between thread demand and available connections. The thread pool and the connection pool had been sized as two separate settings. They should have been one decision. This also explained the low CPU. Blocked threads use almost none.
Transaction scope made it worse. One code path opened a transaction, called an external API, and wrote the result before committing. The connection stayed checked out for the whole API call. In that path, every slow response took a connection out of the pool while no query was running.
The fix: one executor per resource
The team split the shared pool into two executors. One runs database work. The other runs external API calls. Each stage now runs on the executor for the resource it blocks on.
@Configuration
class ExecutorConfig {
// 18 = Hikari maximum-pool-size (20) minus 2 for request-path queries
@Bean ThreadPoolTaskExecutor dbExecutor() { return pool("db-", 18); }
// 80 = ~200 calls/s x 0.3 s p95 latency = 60, plus headroom
@Bean ThreadPoolTaskExecutor apiExecutor() { return pool("api-", 80); }
private static ThreadPoolTaskExecutor pool(String prefix, int size) {
var ex = new ThreadPoolTaskExecutor();
ex.setCorePoolSize(size);
ex.setMaxPoolSize(size);
ex.setThreadNamePrefix(prefix);
return ex;
}
}
The DB executor was resized to align with the Hikari pool. A database thread without a connection can only wait, so threads beyond the connection count add waiting and nothing else. With the sizes aligned, extra database work waits in the executor queue, where it holds no thread and no connection. The executor got 18 threads rather than 20, because a few request-path queries use the same Hikari pool outside the executor.
The API executor needed a different method, and Little's Law gives it. In plain words, the calls in flight equal the calls started per second multiplied by the seconds each call takes. At peak, the service handled about 100 units of work per second, each making two API calls. That is about 200 calls per second. The API's p95 latency under load was around 300 ms, so about 60 calls are in flight at any moment. The team added headroom and set the executor to 80 threads.
The total thread count barely changed, from 100 to 98. What changed was which work could use which threads.
Latency is a live input here. If the API gets twice as slow, the same throughput needs twice as many threads. A call with no timeout breaks the math completely, because it can hold a thread for as long as the remote side likes.
The stages then hop between executors explicitly. A simplified version:
CompletableFuture<Void> process(Job job) {
return CompletableFuture
.supplyAsync(() -> externalApi.fetch(job), apiExecutor)
.thenApplyAsync(data -> repository.save(job, data), dbExecutor)
.thenApplyAsync(saved -> externalApi.confirm(saved), apiExecutor)
.thenAcceptAsync(reply -> repository.markDone(job, reply), dbExecutor);
}
The Async variants matter. Plain thenApply can run the next stage on whatever thread completed the previous one. In this chain, that would quietly put database work back on an API thread.
The split also assumes each stage touches one resource. The transactional path broke that rule, so the team moved its API call out of the transaction. The transaction now covers only the writes. They also added dashboards and alerts on pending connections and on each executor's queue depth.
After the fix, p99 at peak load settled around 450 ms. Connection timeout errors stopped. Peak throughput went from about 70 units per second during the incident back to the expected 100.
It worked because the two kinds of waiting stopped sharing a budget. A slow API can fill its own executor, but it cannot take threads from database work. A busy database cannot stall API stages. A slow API still lowers overall throughput, since every unit needs both. It just no longer drags the database side down with it.
Would virtual threads have prevented this?
Java 21 made virtual threads a standard feature. They are cheap, and a service can run a very large number of them. When a virtual thread blocks on I/O, it usually releases the platform thread underneath it. Spring Boot 3.2 added a spring.threads.virtual.enabled property that moves several of its own components onto virtual threads on Java 21 or later.
The incident was a story about blocked threads, so the question is fair. The answer is no. Virtual threads make threads cheap. They do nothing about connections, and there would still be only 20 of those.
Picture the same service with one virtual thread per task, using Executors.newVirtualThreadPerTaskExecutor(). The 100-thread limit is gone, and every unit starts
Top comments (1)
The split is the right fix, and the 18 vs 20 detail matters more than it looks. A DB thread without a connection can only wait inside getConnection, so capping dbExecutor below the Hikari max moves the wait to the executor queue, where it holds no thread and no connection. That is the key change.
For a regression test I would make the mismatch tiny and deterministic: Hikari max 2, dbExecutor 3, apiExecutor 2, one stubbed API with a fixed 200 ms delay, then N parallel units. With the shared pool the test stalls; with split executors it completes. Give the test an outer deadline and on timeout capture both executor queue depths plus Hikari pending threads - that turns a hanging suite into a readable starvation report instead of a frozen build.
Two follow-ups worth keeping: a timeout on every external call (the Little's Law sizing breaks with one unbounded call), and keeping the API call outside the transaction as you did - holding a connection across an API call reintroduces the same coupling through the back door.