Part 1. Our Database Looked Overloaded. The Real Problem Was @Transactional.

TL;DR
HikariCP was exhausted and Aurora's dashboard screamed overload — but the database was actually doing half its usual work. Connections were checked out; queries weren't running. The threads holding those connections were blocked on something else entirely, and finding what meant looking outside the database.

This three-part series walks through the diagnostic path that pinned the cause to the application side (Part 1), the code patterns we found once we knew what to look for (Part 2), and the per-path fixes we shipped (Part 3).

The Symptom

Our Spring Boot application, deployed on AWS ECS and backed by Aurora PostgreSQL, started throwing this exception:

java.sql.SQLTransientConnectionException: OrdersApi - Connection is not available, 
request timed out after 30000ms (total=50, active=50, idle=0, waiting=6)
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:709)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:188)
    ...
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:420)

The message is unambiguous: HikariCP had all 50 of its connections checked out, six more threads were queued waiting, and after 30 seconds it gave up. Requests started failing.

Then, exactly 15 minutes later, the errors stopped on their own. No deployment, no restart, no human intervention. The application appeared to recover.

The natural first instinct was to look at the database. That's where the investigation started — and that's where it almost ended in the wrong place.

Step 1: Looking at the Database (And Finding Confusing Signals)

The first stop was the Aurora cluster's monitoring tab in the AWS console. The numbers there told a dramatic story:

• DatabaseConnections jumped from ~12 to 73 during the incident
• DBLoad spiked from ~0 to 62
• DBLoadNonCPU reached 62
• DBLoadRelativeToNumVCPUs jumped to 30

But there are no deadlocks according to metrics.

This looked like a textbook database overload. DBLoadNonCPU = 62 in particular is a strong signal: it means dozens of sessions were waiting on something that isn't the CPU — typically locks, I/O, or buffer waits. With one application connecting to one database, seeing 62 average active sessions waiting on non-CPU resources is the kind of thing that makes you sit up.

The PostgreSQL logs, however, told a different story. During the entire 15-minute window — exactly when the application was screaming about timeouts — the database logs were silent. No errors, no slow query warnings, nothing. Then, at the precise moment the application stopped throwing exceptions, Postgres logged:

connection interrupted by peer
unexpected EOF on client connection

This was the first contradiction. The database had no problem during the incident, and only complained at the very end — and what it complained about was the client hanging up. Not the database failing. The application closed the sockets.

Step 2: Performance Insights Says the Database Was Idle

The next step was AWS Performance Insights, which provides much finer-grained visibility into what Aurora is actually doing. We pulled up the exact incident window and looked at the standard panels:

• Top SQL (the Load by waits view) — the top entry was the application's normal, most-frequent query: a very simple request we run thousands of times per minute, that individually completes in well under a millisecond. Exactly what you'd expect to see at the top on any healthy day, and nothing capable of holding a connection long enough to drain the pool.
• Queries per second - dropped.
• Sessions / Idle in transaction — reached 34.
• Transactions in progress — spiked to 80.
• Queries per second - dropped from ~450 to ~230
• Tuple reads - dropped from ~350K to ~190K
• Transactions per second - dropped from ~100 to ~60
• IO cache hits - dropped from ~11K to ~5K pages/sec

Performance Insights said it was almost idle. Both views came from the same database, the same time window — yet they disagreed completely on whether anything was wrong.

At this point we considered three possibilities:

We were looking at the wrong time window in one of the tools (we weren't — timestamps matched).

One of the metrics was lying (neither was).

Both numbers were correct, and the discrepancy itself was a clue.

It turned out to be #3. But before we could understand why, we needed more evidence.

Step 3: The Workload Metrics Tell the Real Story

We then looked at the database's workload metrics during the incident — not "how busy is the database" but "what work is the database actually doing":

This was the moment the investigation pivoted.

If the database had been overloaded, we'd have seen queries piling up, transaction durations climbing, blocked transactions appearing. Instead, the database was doing roughly half as much work as usual. It wasn't slow. It was underused. The database had spare capacity and was waiting for work that wasn't arriving.

Combined with everything else, the picture became clear:

93 TCP connections existed at the network level - much more than usual.

Real query throughput dropped 50% because half the application's effective capacity wasn't reaching the database

The two views from AWS were measuring different layers of the same lifecycle. In our case, the gap meant: connections exist, but they aren't doing anything.

The problem was on the application side.

Step 4: Confirming the Application Side With JVM Metrics

We turned to the application's JVM metrics, scraped via Micrometer into our monitoring stack. The relevant queries:

jvm_threads_live_threads
jvm_threads_daemon_threads
jvm_threads_states_threads
jvm_memory_used_bytes{area="nonheap"}

All four spiked during the incident. That alone was consistent with the database being slow — more requests waiting means more threads parked.

What We Knew at the End of the Infrastructure Investigation

By this point, we had ruled out:

• Database overload (workload metrics dropped, not climbed)
• Lock contention (zero blocked transactions, zero deadlocks)
• Slow queries (queries_started ≈ queries_finished)
• Idle-in-transaction abuse (peaked at 117 ms)
• I/O bottleneck (cache reads dropped, didn't spike)
• Database CPU pressure (CPU wait at 0.05 AAS)

And we had confirmed:

• The database was healthy and underutilized during the incident.
• The application's connection pool was being drained by threads holding connections without using them.

The remaining questions were entirely application-side:

• Which code paths were holding connections?
• Why were threads getting stuck instead of failing fast?
• What changes would prevent this from recurring?

These weren't questions infrastructure metrics could answer. They required reading the code.

The Handoff to Code Analysis

We turned the infrastructure findings into a code-review checklist and went looking for these patterns:

• @Transactional methods making blocking calls to external systems (HTTP, message brokers, AWS services) while holding a database connection
• JDBC and HTTP client configurations missing explicit timeouts (Java's default is "wait forever")
• HikariCP configuration gaps — specifically missing keepalive-time and leak-detection-threshold
• Unbounded thread pools that could grow without limit during traffic spikes
• Resource leaks: connections, streams, or entity managers not properly closed
• Self-invocation of @Transactional methods that bypass Spring's proxy and silently lose transactional behavior

The database was not the bottleneck. It was the victim. The next question was: what code path can hold a JDBC connection while doing no SQL?

That's Part 2.

About me: Olga Ermolaeva is a technical solution owner on the Tourfold team at OpenResearch. https://openresearch.com/