TL;DR
The threads were waiting on external I/O performed inside transaction boundaries. A synchronous Spring event listener published to ActiveMQ before the transaction committed — and the same pattern repeated around Firebase, S3, and SMS.
Recap
The infrastructure investigation in Part 1 narrowed the problem to a precise shape: threads were holding Hikari connections without doing database work. The database itself was healthy. The application was the problem.
There was one more correlation that turned out to be the anchor for everything that followed. During the same 15-minute window when Hikari was failing, the ActiveMQ broker had been unavailable. The broker's connection failures started just before the Hikari errors and recovered at the same moment Hikari recovered.
That was the clue. Whatever the application was doing, it was doing it to the broker, in a way that kept JDBC connections checked out from the pool while the broker call was in flight. As soon as we framed it that way, the question became: where in our code does a database transaction enclose a call to the broker (or any other external system)?
The Trigger: Synchronous Publishing Inside a Transaction
The smoking gun looked almost reasonable at first glance. A simplified version:
@Transactional
fun saveOrder(request: OrderRequest): OrderResponse {
val saved = orderRepository.save(request.toEntity()) // grabs JDBC connection
eventPublisher.publish(OrderUpdatedEvent(this, saved)) // synchronous: blocks here
return saved.toResponse()
}
Bad shape: @Transactional → save entity → publish event → synchronous listener → external I/O.
The intent is clean: do the write, build a response, broadcast a domain event so other parts of the system can react. Three things you're supposed to do.
The problem is what eventPublisher.publish(...) actually does. The publisher is a thin wrapper around Spring's ApplicationEventPublisher, and Spring's default @EventListener runs synchronously on the publishing thread. If a listener does I/O — say, sending a JMS message to ActiveMQ so other app instances can fan it out as SSE events — that I/O happens before publish(...) returns.
Now look at the call site again. publish(...) is being called from inside an @Transactional method. Spring's JpaTransactionManager opens a transaction at method entry and grabs a JDBC connection from Hikari. The transaction is still open when publish(...) runs. That connection is still checked out from Hikari while we're waiting on a network round-trip to a message broker.
When the broker is healthy, the round-trip takes single-digit milliseconds and nobody notices. When the broker has a bad afternoon — broker pod evicted, network partition, head-of-line blocking on a TCP socket — every write request in the system parks on a JMS send. JDBC connections aren't released until publish(...)returns and the transaction commits. The pool drains. New requests wait, time out after 30 seconds, and surface as SQLTransientConnectionException.
This is the pattern the infrastructure metrics had been pointing at: many TCP connections from app to database (because Hikari hands them out before requests park), low query throughput at the database (because parked requests aren't sending queries), elevated thread count and non-heap memory in the JVM (because every parked request is a Tomcat thread sitting on a JMS send). All of it consistent with one root cause: a transaction boundary that encloses a network call to a system the database doesn't know about.
The Family of Sibling Bugs
Once we knew what to look for, we found more of it. The general shape is the same — a database transaction held open across a call to an external system — but each variant fails for a different reason and hits a different upstream.
It's worth describing each one in its own terms, because the failure modes are not interchangeable.
1. Push notifications via Firebase
Certain business writes produce a domain event — for example, an entity being assigned to a user. A listener picks the event up and calls Firebase Cloud Messaging so the mobile app gets a notification.
FirebaseMessaging.send(...) is a synchronous HTTP call to Google's servers. It's not exotic — there's no special timeout configured, no rate limit you'd hit in normal operation. It just makes an HTTPS request, waits for the 200, and returns.
When that listener runs inside an @Transactional(REQUIRES_NEW)boundary — which it did — every push notification holds a JDBC connection for the entire HTTPS round-trip. On a normal day, that's tens of milliseconds. During a Firebase regional incident, that's seconds-to-minutes per send, and we'd see the same connection-pool symptom from Part 1, but with Firebase as the trigger instead of ActiveMQ.
The kicker is that the listener doesn't actually need a database transaction at all. It reads a couple of associations to decide who to notify, then calls Firebase. Each of those reads is a tiny query that could happen in its own short transaction. The wrapping transaction was there because someone had wanted "REQUIRES_NEW so the failure can't roll back the original commit" — a defensible instinct that turned the connection pool into Firebase's hostage.
2. Object storage uploads to S3
File uploads went through the application: the user's HTTP request hit a controller, the controller wrote a metadata row to Postgres, and inside the same transaction it called s3Client.putObject(...) to push the bytes to S3.
This one is worse than it looks. The AWS SDK's default apiCallTimeout is unset — meaning the SDK is willing to wait indefinitely for a slow PUT. A multi-megabyte upload to a slow region holds a Postgres connection for the entire transfer, and the SDK won't bail on its own. A user uploading a 50 MB file from a flaky mobile network is a connection on loan to S3 for a minute or more.
The transactional intent here was "if the S3 upload fails, roll back the metadata row so we don't have a database row pointing at a non-existent object". Reasonable in principle, dangerous in practice: while the upload is in flight, that connection is unavailable to anyone else, and the database has no way to know whether to wait or not. It's just patient.
3. SMS sending through SMS gateway
Outbound SMS goes through a per-tenant scheduled dispatcher. The dispatcher wakes up periodically, finds messages in SCHEDULED status, and sends them through whichever gateway the tenant has configured.
The original dispatcher loop opened one transaction per tenant and processed every pending message for that tenant inside it:
transactionTemplate.execute {
repository.findPendingMessages().forEach { message ->
smsGateway.send(message) // synchronous HTTPS call
repository.markAsSent(message)
}
}
smsGateway.send is a synchronous HTTPS call to the upstream provider. The transaction was open across every one of those calls for every pending message in the batch. During a regional outage at the upstream — which has happened — a batch of 50 pending messages would hold one Hikari connection for the duration of 50 sequential HTTPS round-trips, while every other request in the system competed for the remaining 49.
The unifying pattern
Three different upstream services, three different failure modes, one recipe.
A transaction boundary that encloses I/O to a system the database doesn't know about.
The database is patient. It will happily hold your connection open for as long as you want, and it has no opinion about whether the thing you're waiting on is its problem. Every external system you touch inside that boundary is a potential 30-second (or longer) lien on a connection — and during a real outage, "30 seconds" is "until the upstream comes back".
The transactional integrity you think you're preserving by including the external call in the transaction is, in most of these cases, an illusion. If the HTTPS call to Firebase succeeds and the database commit fails, you've already sent the notification. There is no "rollback" for an SMS that has already been delivered. The transaction boundary is doing nothing for correctness; it's just keeping a connection on hold.
The fix, in every case, is some flavor of the same shape: do the database work first, finish it, release the connection, then do the external work. Part 3 is about the variations of that shape, why a single one-size-fits-all answer doesn't work for us, and what each fix actually looks like.
About me: Olga Ermolaeva is a technical solution owner on the Tourfold team at OpenResearch. https://openresearch.com/
Top comments (1)
Great deep dive into a pattern thats surprisingly easy to miss in production. The unifying diagnosis — transaction boundary enclosing external I/O — is the real takeaway here. One thing worth adding: the same pattern can also surface when a framework auto-wraps public methods in transactions by default, making it invisible unless you trace the call stack. We ran into a variant where a seemingly innocent @Transactional on a service method hid a cache-warming HTTP call behind a repository method. Took the same metrics-driven approach you laid out here to find it. Looking forward to Part 3 on the fix patterns.Great deep dive. The unifying diagnosis -- transaction boundary enclosing external I/O -- is the real takeaway here. One thing worth adding: the same pattern surfaces when a framework auto-wraps public methods in transactions by default. We ran into a variant where @Transactional hid a cache-warming HTTP call. Looking forward to Part 3.