DEV Community: Olga Ermolaeva

Part 3. How to Fix External I/O Inside Transactions Without Overengineering Everything

Olga Ermolaeva — Tue, 28 Jul 2026 11:48:15 +0000

TL;DR

There's no universal fix — each call needs the pattern that matches its delivery guarantees. Post-commit handoff where event loss is acceptable (SSE). No transaction at all where the database doesn't need to know (FCM). Short DB phases bracketing the network call where state has to be tracked (SMS). Presigned POSTs to keep large uploads off the application entirely (S3). An outbox where delivery has to survive a crash (anything that matters).

A Diagnostic Checklist

When the pool is exhausted but the database itself looks healthy, work through these in order:

Is HikariCP at max active while DB query throughput is down? If yes, your connections aren't doing database work — they're held by something else. Skip ahead.
Are app threads blocked on HTTP, JMS, S3, Firebase, or SMS calls? Take a thread dump (jstack, async-profiler, your APM's "blocked threads" view). Group by stack frame. Anything not in java.sql / org.postgresql / com.zaxxer.hikari is a candidate.
Do any @Transactional methods enclose external I/O? Search for @Transactional annotations and trace what they call. The bad shape is: @Transactional → service method → external client (HTTP, JMS, AWS SDK, Firebase). The boundary is wrong — move the I/O out, not the annotation.
Are @EventListener methods synchronous and called from inside a transaction? Default Spring listeners run on the publishing thread. If the listener does I/O, that I/O is inside the publisher's transaction. Switch to @TransactionalEventListener(phase = AFTER_COMMIT).
Is the upstream that triggered this incident correlated in time with the pool exhaustion? If yes, that's your culprit. If no, look for any external system whose latency spiked.

The order matters: 1–2 confirm the shape of the problem, 3–4 find the cause, 5 names the trigger. Without all four pieces it's easy to "fix" the wrong layer.

The Underlying Pattern

This is a common production trap in Spring systems: a @Transactional method makes a synchronous call to an external system — a message broker, an HTTP API, an SDK that wraps one — and a slow upstream drains the database pool while the database itself sits idle. We found it the hard way, traced it across metrics and code, and this part is the field guide that came out of it: the patterns to apply, sized to the delivery guarantees each call actually needs.

Part 1 walked through the metrics that pointed away from the database. Part 2 named the shape: external I/O inside transaction boundaries — synchronous calls to ActiveMQ, Firebase, S3, and an SMS gateway, each one holding a Hikari connection until it returned. This part is how to handle each variant.

The Constraint That Shapes the Solutions

There's one piece of infrastructure context that decides which solution fits where: the ActiveMQ deployment is a single broker instance, not a cluster. This isn't something the application layer can change — it's a given. The right move is to treat it as such.

(Yes, clustering has been considered. The cost is high, and today ActiveMQ isn't relied on as a system for durability — the loads that go through it are all loss-tolerant. If that shifts and a real event queue is needed, clustering becomes the answer then. Until that day, the transactional outbox pattern (covered below) gives stronger durability guarantees on the paths that actually need them, with less infrastructure to operate.)

That constraint has a clean implication. ActiveMQ is fine as a fan-out bus for messages whose loss is recoverable: a single-instance broker can be restarted, and any message in flight at the moment of the restart is gone. It is not fine as the durability layer for messages that absolutely must reach their destination. Those need a different mechanism.

That's why this isn't one fix — it's several, sized to the criticality of each path.

SSE: The Smallest Possible Annotation Change

Server-Sent Events are how the frontend gets near-real-time updates. The flow is: a write commits in the API, an SSE event is published to ActiveMQ, every API instance receives it on the topic, and each instance forwards it to its locally-connected SSE clients. If the broker drops a message, the client doesn't get the update via that path — but the SSE client always reconnects with a Last-Event-Id header, and the API replays missed events from an in-memory ring buffer (EventHistory). Loss is tolerable because the client recovers.

Spring's docs note that with @TransactionalEventListener, transactional resources may still be active and accessible after commit. "After commit" doesn't guarantee that the connection has already been returned to the pool. So a synchronous broker call inside an AFTER_COMMIT listener can still hold the JDBC connection while it waits.

That means two changes are needed:

Move the broker call outside the transaction boundary — switch from @EventListener to @TransactionalEventListener(phase = AFTER_COMMIT), so the publish happens after the commit completes rather than mid-transaction.
Make the broker call itself non-blocking on the listener thread — hand the publish off to a separate executor, so even if the listener thread is still associated with transactional resources, those resources can be released while the broker call proceeds elsewhere.

Together, these guarantee that a slow or unreachable broker can no longer pin a Hikari connection.

// Before — runs on the publishing thread, inside the transaction.
@EventListener
fun onOrderUpdated(event: OrderUpdatedEvent) { ... }
// After — runs on a different thread, and only after the transaction commits.
@TransactionalEventListener(phase = AFTER_COMMIT)
fun onOrderUpdated(event: OrderUpdatedEvent) {
    ssePublishExecutor.execute {
        jmsPublisher.publish(event.toMessage())
    }
}

This works for SSE because SSE losses are tolerable. It's not enough for anything that must not be lost.

FCM: Remove the Wrapping Transaction, Keep the Reads Short

The push notifications listener was already on @TransactionalEventListener(AFTER_COMMIT). The remaining problem was an inner transaction wrapping the listener body:

@TransactionalEventListener(AFTER_COMMIT)
@Transactional(propagation = Propagation.REQUIRES_NEW)   // ← the problem
fun onOrderAssigned(event: OrderAssignedEvent) {
    val recipients = userService.findRecipientsFor(event.orderId)
    pushClient.sendPush(recipients, event.payload)        // asynchronous HTTPS
}

The REQUIRES_NEW opens a fresh transaction at the start of the listener and holds it until the FCM HTTP call returns. Remove it. The listener body now runs with no transaction. Each method it calls handles its own transactional needs:

@Transactional(readOnly = true)
fun findRecipientsFor(orderId: UUID): List<User> { ... }

The flow now looks like: listener fires after commit → short read transaction → connection returned → short read transaction → connection returned → pushClient.sendPush returns immediately and the HTTPS work runs on FCM's own async executor. A Firebase regional incident now blocks neither a Hikari connection nor a Tomcat thread.

This is a deliberately conservative change. FCM isn't behind an outbox or on a queue — push notifications are best-effort by nature, and the existing code already makes that bargain. The point is just to stop letting Firebase's bad afternoons drain the database pool.

SMS: Split the Dispatcher Into Phases

SMS sending was already half-fixed by design. When a feature wants to send an SMS, it doesn't call the gateway directly — it inserts a row into a scheduled_messages table in the caller's transaction and returns. The actual HTTPS call is made later by a separate scheduled dispatcher. The user-facing request never hits the upstream provider.

That part was already correct. The problem was inside the dispatcher itself. The original loop opened one transaction per batch and held it across every gateway call. The fix splits the work into three short transactions and one transaction-free HTTP call:

@Scheduled(fixedDelayString = "...")
fun dispatchPendingMessages() {
    // Phase 1: short transaction, read pending IDs only.
    val pendingIds: List<UUID> = transactionTemplate.execute {
        repository.findPendingMessageIds()
    }
    // Phase 2: process each ID outside any DB transaction.
    pendingIds.forEach { id ->
        processor.processMessage(id)
    }
}

processMessage itself is the second important piece. It opens a REQUIRES_NEW transaction only to flip status from SCHEDULED to SENDING and to read a snapshot of the fields the gateway needs. It returns a plain value object — not a JPA entity. The gateway HTTPS call then runs with no transaction at all. A second REQUIRES_NEW transaction writes the final status once the gateway returns.

fun processMessage(messageId: UUID) {
    val snapshot = messageService.markAsSending(messageId) ?: return    // short tx
    try {
        val result = smsGateway.send(                                   // no tx
            snapshot.toNumber, snapshot.text, snapshot.fromNumber,
        )
        messageService.updateFinalStatus(messageId, result)             // short tx
    } catch (ex: Exception) {
        messageService.markAsFailed(messageId, ex.message)              // short tx
    }
}
data class MessageSnapshot(
    val toNumber: String,
    val text: String,
    val fromNumber: String?,
)

The snapshot matters. If markAsSending returned the JPA entity, that entity would be detached the moment its transaction commits, and any lazy-loaded field accessed later (during the gateway call, for example) would throw LazyInitializationException. The snapshot carries exactly the fields the gateway needs and nothing else, which makes it physically impossible to accidentally drag a JPA session into the HTTP call.

An upstream regional outage now holds, at worst, a single in-flight HTTPS call per concurrent dispatcher tick. It does not hold a Hikari connection.

S3: Don't Carry the Bytes Through the Application

For object storage, the fix isn't a transactional change at all — it's an architectural one. Instead of streaming the user's bytes through the application server and into S3, issue a presigned upload URL and let the browser upload directly to the bucket.

The flow:

Client calls POST /upload-intent with mime type and max size.
Backend writes a draft file-metadata row inside its (short) transaction, asks the storage SDK to generate a presigned upload URL constrained by storage key, mime type, content-length range, and a TTL, and returns the URL plus any required form fields to the client.
Client uploads the file directly to the storage endpoint using the presigned URL. The application server is not in the data path.
When the client confirms the upload, the backend (in another short transaction) flips the file's status from PENDING_UPLOAD to UPLOADED.

Two things this gets you. First, the application never holds a database connection while bytes move over the wire — there are no bytes moving through the application. Second, a slow user network or a slow storage region affects only that user (their progress bar moves slowly), not other unrelated requests.

This was already the design for new client-driven uploads. The discipline is to make sure no feature builds new uploads on the "stream through the server" path, and to fail closed if anyone tries to introduce one.

Webhooks: The Outbox Pattern

Webhooks are the case where SSE-style "loss is tolerable" is wrong. If a customer is subscribed to a business event and delivery silently fails, their downstream system is out of sync. A single-instance broker can't carry durability here.

The standard fix is the transactional outbox pattern. The mechanism, briefly, for readers who haven't seen it:

When a business operation needs to publish a message that must not be lost, the producing transaction inserts the message into an outbox table in the same database, in the same transaction as the business write. Either both commit or neither does — the message is now durable on the same storage as the business state. A separate background poller reads the outbox table, dispatches the messages downstream (to a broker, to an HTTP endpoint, wherever), and marks them processed.

Because the message is in the database, no broker outage can lose it. The poller can crash, restart, fall behind — the messages stay in the table until they're successfully delivered. Retries, dead-lettering, and ordering are properties of the table, not of the broker.

The producer side is one extra line inside the existing business transaction:

@Transactional
fun createOrder(request: OrderRequest): Order {
    val order = orderRepository.save(request.toEntity())     // business write
    outboxService.enqueue(                                   // outbox write (same tx)
        aggregateId = "order:${order.id}",
        eventType = "order.created",
        payload = objectMapper.writeValueAsString(order),
    )
    return order
}

outboxService.enqueue(...) is annotated @Transactional(MANDATORY)— it refuses to run outside an existing transaction. It writes a row to the outbox table and returns. The business write and the outbox row commit together, atomically, on the same connection.

The consumer side is a scheduled poller that runs every 5 seconds. It claims a batch of pending records using SELECT ... FOR UPDATE SKIP LOCKED, processes each in its own transaction (so one bad message doesn't block the others), and lets a registered handler decide what "processing" means. For webhooks, the handler resolves the subscribed endpoints for the event type and publishes one broker queue message per subscriber. From there, a separate consumer makes the actual HTTP call to the customer's endpoint with retry, signing, and dead-lettering.

The chain of guarantees:

Database → outbox row: atomic, transactional, can't be lost.
Outbox row → broker queue: at-least-once. The poller commits "processed" only after the broker accepts the message. If the broker is down, the row stays PENDING and the next poll tries again.
Broker queue → customer HTTP endpoint: at-least-once with explicit retry counters and dead-letter routing. A single-instance broker outage delays delivery; it does not lose messages, because the canonical state is in the outbox row, not the broker.

The cost is straightforward and acceptable. Latency for webhook delivery is bounded below by the poller's 5-second tick. The outbox table grows and is cleaned up on a schedule (hourly, 7-day retention by default). And because the poller uses the same short-transaction shape as everything else, it composes naturally with the rest of the system.

Preventing Recurrence

Finding and fixing the bugs is half the work. The other half is making sure the next instance of the same pattern can't ship quietly.

HikariCP configuration.

spring:
  datasource:
    hikari:
      maximum-pool-size: 50
      leak-detection-threshold: 25000  # gives stack trace before pool exhaustion

Enable Hikari's leak detection in staging, and in production too if you can tolerate the log noise. Pick a threshold longer than your typical worst-case connection hold (the p99 — the time 99% of healthy connections finish within), so it only triggers on real outliers. Treat it as a diagnostic alarm, not as proof of a leak: when it fires, you get the stack trace of the thread still holding the connection — exactly the evidence that's missing during the incident itself.

An architecture rule. ArchUnit (or your equivalent) enforces the "no external I/O inside @Transactional" rule at build time:

@ArchTest
val transactionalCodeMustNotCallExternalIo: ArchRule =
    methods()
        .that().areAnnotatedWith(Transactional::class.java)
        .or().areDeclaredInClassesThat().areAnnotatedWith(Transactional::class.java)
        .should(notTransitivelyCall(
            FirebaseMessaging::class.java,
            S3Client::class.java,
            JmsTemplate::class.java,
            RestTemplate::class.java,
            // ...whatever else moves bytes off-server
        ))

The exact DSL depends on your version of ArchUnit and how strict you want to be about transitive calls. The point is: name the external clients explicitly, and make the build fail when one of them appears inside a transaction. Two-minute lookup at PR time, instead of a 15-minute outage at 2am.

An alert that catches it in production. Even with config and rules in place, you want to know before users do:

Alert on hikaricp_connections_pending > 0 sustained for more than 30 seconds. Pending connections are threads waiting for a connection; sustained pending means the pool is saturated. This will fire long before the 30-second timeout starts surfacing as user errors.
Alert on hikaricp_connections_active / hikaricp_connections_max > 0.8 sustained for more than 1 minute. Saturation, not exhaustion — but it's the warning sign.

Both are off-the-shelf Micrometer metrics. The thresholds are starting points; tune them to your traffic.

What This All Adds Up To

This isn't a one-bug problem and it isn't a one-fix problem. It's a category of bugs — external I/O inside a transaction — that quietly accumulates in any sufficiently old Spring codebase, every instance harmless until an upstream has a bad day. And the fixes form a menu, not a single recipe:

For SSE: tolerate loss, use @TransactionalEventListener(AFTER_COMMIT) with handoff to a separate executor. Cheap, sufficient.
For FCM: remove the wrapping transaction, make inner reads short and read-only. Best-effort by nature, no database connection held during the HTTP call.
For SMS: split the dispatcher into phases, snapshot the row into a DTO, send outside any transaction. A scheduled_messages table covers durability; the dispatcher just keeps connections short.
For S3: don't carry the bytes. Presigned POST, client uploads directly.
For webhooks: full transactional outbox. Durable on the database, survives broker outages.

The unifying principle is small enough to fit on a sticky note: a database transaction must not enclose a network call to a system the database doesn't manage. The size of the fix depends on how much you care if the message is lost, how big the message is, and what infrastructure you can rely on. Pick the smallest pattern that matches your delivery guarantees — and enforce it at build time, because the next instance of this bug will look harmless on review.

For Your Runbook

When HikariCP is exhausted but database metrics look healthy, the diagnostic checklist at the top of this article is the fastest path to the cause.

The fixes are sized to delivery guarantees:

Loss-tolerant (SSE) → post-commit listener with executor handoff.
Best-effort (FCM) → remove the wrapping transaction; short read-only inner transactions; HTTP call outside.
Durable but not user-facing (SMS) → short-phase dispatcher; snapshot DTOs; HTTP call between transactions.
Large payloads (S3) → presigned upload directly from client to storage.
Must-not-lose (webhooks) → transactional outbox; poller dispatches; broker becomes optional.

Pick the smallest pattern that matches the guarantee you actually need. Enforce "no external I/O inside @Transactional" at build time with ArchUnit. Alert on hikaricp_connections_pending > 0 sustained for 30 seconds. Set leak-detection-threshold above your p99 connection-hold time.

About me: Olga Ermolaeva is a technical solution owner on the Tourfold team at OpenResearch. https://openresearch.com/

Part 2. The Spring @Transactional Pattern That Drained Our Hikari Pool

Olga Ermolaeva — Tue, 28 Jul 2026 11:28:06 +0000

TL;DR

The threads were waiting on external I/O performed inside transaction boundaries. A synchronous Spring event listener published to ActiveMQ before the transaction committed — and the same pattern repeated around Firebase, S3, and SMS.

Recap

The infrastructure investigation in Part 1 narrowed the problem to a precise shape: threads were holding Hikari connections without doing database work. The database itself was healthy. The application was the problem.

There was one more correlation that turned out to be the anchor for everything that followed. During the same 15-minute window when Hikari was failing, the ActiveMQ broker had been unavailable. The broker's connection failures started just before the Hikari errors and recovered at the same moment Hikari recovered.

That was the clue. Whatever the application was doing, it was doing it to the broker, in a way that kept JDBC connections checked out from the pool while the broker call was in flight. As soon as we framed it that way, the question became: where in our code does a database transaction enclose a call to the broker (or any other external system)?

The Trigger: Synchronous Publishing Inside a Transaction

The smoking gun looked almost reasonable at first glance. A simplified version:

@Transactional
fun saveOrder(request: OrderRequest): OrderResponse {
    val saved = orderRepository.save(request.toEntity())     // grabs JDBC connection
    eventPublisher.publish(OrderUpdatedEvent(this, saved))   // synchronous: blocks here
    return saved.toResponse()
}

Bad shape: @Transactional → save entity → publish event → synchronous listener → external I/O.
The intent is clean: do the write, build a response, broadcast a domain event so other parts of the system can react. Three things you're supposed to do.

The problem is what eventPublisher.publish(...) actually does. The publisher is a thin wrapper around Spring's ApplicationEventPublisher, and Spring's default @EventListener runs synchronously on the publishing thread. If a listener does I/O — say, sending a JMS message to ActiveMQ so other app instances can fan it out as SSE events — that I/O happens before publish(...) returns.

Now look at the call site again. publish(...) is being called from inside an @Transactional method. Spring's JpaTransactionManager opens a transaction at method entry and grabs a JDBC connection from Hikari. The transaction is still open when publish(...) runs. That connection is still checked out from Hikari while we're waiting on a network round-trip to a message broker.

When the broker is healthy, the round-trip takes single-digit milliseconds and nobody notices. When the broker has a bad afternoon — broker pod evicted, network partition, head-of-line blocking on a TCP socket — every write request in the system parks on a JMS send. JDBC connections aren't released until publish(...)returns and the transaction commits. The pool drains. New requests wait, time out after 30 seconds, and surface as SQLTransientConnectionException.

This is the pattern the infrastructure metrics had been pointing at: many TCP connections from app to database (because Hikari hands them out before requests park), low query throughput at the database (because parked requests aren't sending queries), elevated thread count and non-heap memory in the JVM (because every parked request is a Tomcat thread sitting on a JMS send). All of it consistent with one root cause: a transaction boundary that encloses a network call to a system the database doesn't know about.

The Family of Sibling Bugs

Once we knew what to look for, we found more of it. The general shape is the same — a database transaction held open across a call to an external system — but each variant fails for a different reason and hits a different upstream.

It's worth describing each one in its own terms, because the failure modes are not interchangeable.

1. Push notifications via Firebase

Certain business writes produce a domain event — for example, an entity being assigned to a user. A listener picks the event up and calls Firebase Cloud Messaging so the mobile app gets a notification.

FirebaseMessaging.send(...) is a synchronous HTTP call to Google's servers. It's not exotic — there's no special timeout configured, no rate limit you'd hit in normal operation. It just makes an HTTPS request, waits for the 200, and returns.

When that listener runs inside an @Transactional(REQUIRES_NEW)boundary — which it did — every push notification holds a JDBC connection for the entire HTTPS round-trip. On a normal day, that's tens of milliseconds. During a Firebase regional incident, that's seconds-to-minutes per send, and we'd see the same connection-pool symptom from Part 1, but with Firebase as the trigger instead of ActiveMQ.

The kicker is that the listener doesn't actually need a database transaction at all. It reads a couple of associations to decide who to notify, then calls Firebase. Each of those reads is a tiny query that could happen in its own short transaction. The wrapping transaction was there because someone had wanted "REQUIRES_NEW so the failure can't roll back the original commit" — a defensible instinct that turned the connection pool into Firebase's hostage.

2. Object storage uploads to S3

File uploads went through the application: the user's HTTP request hit a controller, the controller wrote a metadata row to Postgres, and inside the same transaction it called s3Client.putObject(...) to push the bytes to S3.

This one is worse than it looks. The AWS SDK's default apiCallTimeout is unset — meaning the SDK is willing to wait indefinitely for a slow PUT. A multi-megabyte upload to a slow region holds a Postgres connection for the entire transfer, and the SDK won't bail on its own. A user uploading a 50 MB file from a flaky mobile network is a connection on loan to S3 for a minute or more.

The transactional intent here was "if the S3 upload fails, roll back the metadata row so we don't have a database row pointing at a non-existent object". Reasonable in principle, dangerous in practice: while the upload is in flight, that connection is unavailable to anyone else, and the database has no way to know whether to wait or not. It's just patient.

3. SMS sending through SMS gateway

Outbound SMS goes through a per-tenant scheduled dispatcher. The dispatcher wakes up periodically, finds messages in SCHEDULED status, and sends them through whichever gateway the tenant has configured.

The original dispatcher loop opened one transaction per tenant and processed every pending message for that tenant inside it:

transactionTemplate.execute {
    repository.findPendingMessages().forEach { message ->
        smsGateway.send(message)          // synchronous HTTPS call
        repository.markAsSent(message)
    }
}

smsGateway.send is a synchronous HTTPS call to the upstream provider. The transaction was open across every one of those calls for every pending message in the batch. During a regional outage at the upstream — which has happened — a batch of 50 pending messages would hold one Hikari connection for the duration of 50 sequential HTTPS round-trips, while every other request in the system competed for the remaining 49.

The unifying pattern

Three different upstream services, three different failure modes, one recipe.

A transaction boundary that encloses I/O to a system the database doesn't know about.

The database is patient. It will happily hold your connection open for as long as you want, and it has no opinion about whether the thing you're waiting on is its problem. Every external system you touch inside that boundary is a potential 30-second (or longer) lien on a connection — and during a real outage, "30 seconds" is "until the upstream comes back".

The transactional integrity you think you're preserving by including the external call in the transaction is, in most of these cases, an illusion. If the HTTPS call to Firebase succeeds and the database commit fails, you've already sent the notification. There is no "rollback" for an SMS that has already been delivered. The transaction boundary is doing nothing for correctness; it's just keeping a connection on hold.

The fix, in every case, is some flavor of the same shape: do the database work first, finish it, release the connection, then do the external work. Part 3 is about the variations of that shape, why a single one-size-fits-all answer doesn't work for us, and what each fix actually looks like.

About me: Olga Ermolaeva is a technical solution owner on the Tourfold team at OpenResearch. https://openresearch.com/

Part 1. Our Database Looked Overloaded. The Real Problem Was @Transactional.

Olga Ermolaeva — Tue, 28 Jul 2026 11:26:58 +0000

TL;DR
HikariCP was exhausted and Aurora's dashboard screamed overload — but the database was actually doing half its usual work. Connections were checked out; queries weren't running. The threads holding those connections were blocked on something else entirely, and finding what meant looking outside the database.

This three-part series walks through the diagnostic path that pinned the cause to the application side (Part 1), the code patterns we found once we knew what to look for (Part 2), and the per-path fixes we shipped (Part 3).

The Symptom

Our Spring Boot application, deployed on AWS ECS and backed by Aurora PostgreSQL, started throwing this exception:

java.sql.SQLTransientConnectionException: OrdersApi - Connection is not available, 
request timed out after 30000ms (total=50, active=50, idle=0, waiting=6)
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:709)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:188)
    ...
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:420)

The message is unambiguous: HikariCP had all 50 of its connections checked out, six more threads were queued waiting, and after 30 seconds it gave up. Requests started failing.

Then, exactly 15 minutes later, the errors stopped on their own. No deployment, no restart, no human intervention. The application appeared to recover.

The natural first instinct was to look at the database. That's where the investigation started — and that's where it almost ended in the wrong place.

Step 1: Looking at the Database (And Finding Confusing Signals)

The first stop was the Aurora cluster's monitoring tab in the AWS console. The numbers there told a dramatic story:

DatabaseConnections jumped from ~12 to 73 during the incident
DBLoad spiked from ~0 to 62
DBLoadNonCPU reached 62
DBLoadRelativeToNumVCPUs jumped to 30

But there are no deadlocks according to metrics.

This looked like a textbook database overload. DBLoadNonCPU = 62 in particular is a strong signal: it means dozens of sessions were waiting on something that isn't the CPU — typically locks, I/O, or buffer waits. With one application connecting to one database, seeing 62 average active sessions waiting on non-CPU resources is the kind of thing that makes you sit up.

The PostgreSQL logs, however, told a different story. During the entire 15-minute window — exactly when the application was screaming about timeouts — the database logs were silent. No errors, no slow query warnings, nothing. Then, at the precise moment the application stopped throwing exceptions, Postgres logged:

connection interrupted by peer
unexpected EOF on client connection

This was the first contradiction. The database had no problem during the incident, and only complained at the very end — and what it complained about was the client hanging up. Not the database failing. The application closed the sockets.

Step 2: Performance Insights Says the Database Was Idle

The next step was AWS Performance Insights, which provides much finer-grained visibility into what Aurora is actually doing. We pulled up the exact incident window and looked at the standard panels:

Top SQL (the Load by waits view) — the top entry was the application's normal, most-frequent query: a very simple request we run thousands of times per minute, that individually completes in well under a millisecond. Exactly what you'd expect to see at the top on any healthy day, and nothing capable of holding a connection long enough to drain the pool.
Queries per second - dropped.
Sessions / Idle in transaction — reached 34.
Transactions in progress — spiked to 80.
Queries per second - dropped from ~450 to ~230
Tuple reads - dropped from ~350K to ~190K
Transactions per second - dropped from ~100 to ~60
IO cache hits - dropped from ~11K to ~5K pages/sec

Performance Insights said it was almost idle. Both views came from the same database, the same time window — yet they disagreed completely on whether anything was wrong.

At this point we considered three possibilities:

We were looking at the wrong time window in one of the tools (we weren't — timestamps matched).

One of the metrics was lying (neither was).

Both numbers were correct, and the discrepancy itself was a clue.

It turned out to be #3. But before we could understand why, we needed more evidence.

Step 3: The Workload Metrics Tell the Real Story

We then looked at the database's workload metrics during the incident — not "how busy is the database" but "what work is the database actually doing":

This was the moment the investigation pivoted.

If the database had been overloaded, we'd have seen queries piling up, transaction durations climbing, blocked transactions appearing. Instead, the database was doing roughly half as much work as usual. It wasn't slow. It was underused. The database had spare capacity and was waiting for work that wasn't arriving.

Combined with everything else, the picture became clear:

93 TCP connections existed at the network level - much more than usual.

Real query throughput dropped 50% because half the application's effective capacity wasn't reaching the database

The two views from AWS were measuring different layers of the same lifecycle. In our case, the gap meant: connections exist, but they aren't doing anything.

The problem was on the application side.

Step 4: Confirming the Application Side With JVM Metrics

We turned to the application's JVM metrics, scraped via Micrometer into our monitoring stack. The relevant queries:

jvm_threads_live_threads
jvm_threads_daemon_threads
jvm_threads_states_threads
jvm_memory_used_bytes{area="nonheap"}

All four spiked during the incident. That alone was consistent with the database being slow — more requests waiting means more threads parked.

What We Knew at the End of the Infrastructure Investigation

By this point, we had ruled out:

Database overload (workload metrics dropped, not climbed)
Lock contention (zero blocked transactions, zero deadlocks)
Slow queries (queries_started ≈ queries_finished)
Idle-in-transaction abuse (peaked at 117 ms)
I/O bottleneck (cache reads dropped, didn't spike)
Database CPU pressure (CPU wait at 0.05 AAS)

And we had confirmed:

The database was healthy and underutilized during the incident.
The application's connection pool was being drained by threads holding connections without using them.

The remaining questions were entirely application-side:

Which code paths were holding connections?
Why were threads getting stuck instead of failing fast?
What changes would prevent this from recurring?

These weren't questions infrastructure metrics could answer. They required reading the code.

The Handoff to Code Analysis

We turned the infrastructure findings into a code-review checklist and went looking for these patterns:

@Transactional methods making blocking calls to external systems (HTTP, message brokers, AWS services) while holding a database connection
JDBC and HTTP client configurations missing explicit timeouts (Java's default is "wait forever")
HikariCP configuration gaps — specifically missing keepalive-time and leak-detection-threshold
Unbounded thread pools that could grow without limit during traffic spikes
Resource leaks: connections, streams, or entity managers not properly closed
Self-invocation of @Transactional methods that bypass Spring's proxy and silently lose transactional behavior

The database was not the bottleneck. It was the victim. The next question was: what code path can hold a JDBC connection while doing no SQL?

That's Part 2.

About me: Olga Ermolaeva is a technical solution owner on the Tourfold team at OpenResearch. https://openresearch.com/