Part 3. How to Fix External I/O Inside Transactions Without Overengineering Everything

TL;DR

There's no universal fix — each call needs the pattern that matches its delivery guarantees. Post-commit handoff where event loss is acceptable (SSE). No transaction at all where the database doesn't need to know (FCM). Short DB phases bracketing the network call where state has to be tracked (SMS). Presigned POSTs to keep large uploads off the application entirely (S3). An outbox where delivery has to survive a crash (anything that matters).

A Diagnostic Checklist

When the pool is exhausted but the database itself looks healthy, work through these in order:

1. Is HikariCP at max active while DB query throughput is down? If yes, your connections aren't doing database work — they're held by something else. Skip ahead.

2. Are app threads blocked on HTTP, JMS, S3, Firebase, or SMS calls? Take a thread dump (jstack, async-profiler, your APM's "blocked threads" view). Group by stack frame. Anything not in java.sql / org.postgresql / com.zaxxer.hikari is a candidate.

3. Do any @Transactional methods enclose external I/O? Search for @Transactional annotations and trace what they call. The bad shape is: @Transactional → service method → external client (HTTP, JMS, AWS SDK, Firebase). The boundary is wrong — move the I/O out, not the annotation.

4. Are @EventListener methods synchronous and called from inside a transaction? Default Spring listeners run on the publishing thread. If the listener does I/O, that I/O is inside the publisher's transaction. Switch to @TransactionalEventListener(phase = AFTER_COMMIT).

5. Is the upstream that triggered this incident correlated in time with the pool exhaustion? If yes, that's your culprit. If no, look for any external system whose latency spiked.

The order matters: 1–2 confirm the shape of the problem, 3–4 find the cause, 5 names the trigger. Without all four pieces it's easy to "fix" the wrong layer.

The Underlying Pattern

This is a common production trap in Spring systems: a @Transactional method makes a synchronous call to an external system — a message broker, an HTTP API, an SDK that wraps one — and a slow upstream drains the database pool while the database itself sits idle. We found it the hard way, traced it across metrics and code, and this part is the field guide that came out of it: the patterns to apply, sized to the delivery guarantees each call actually needs.

Part 1 walked through the metrics that pointed away from the database. Part 2 named the shape: external I/O inside transaction boundaries — synchronous calls to ActiveMQ, Firebase, S3, and an SMS gateway, each one holding a Hikari connection until it returned. This part is how to handle each variant.

The Constraint That Shapes the Solutions

There's one piece of infrastructure context that decides which solution fits where: the ActiveMQ deployment is a single broker instance, not a cluster. This isn't something the application layer can change — it's a given. The right move is to treat it as such.

(Yes, clustering has been considered. The cost is high, and today ActiveMQ isn't relied on as a system for durability — the loads that go through it are all loss-tolerant. If that shifts and a real event queue is needed, clustering becomes the answer then. Until that day, the transactional outbox pattern (covered below) gives stronger durability guarantees on the paths that actually need them, with less infrastructure to operate.)

That constraint has a clean implication. ActiveMQ is fine as a fan-out bus for messages whose loss is recoverable: a single-instance broker can be restarted, and any message in flight at the moment of the restart is gone. It is not fine as the durability layer for messages that absolutely must reach their destination. Those need a different mechanism.

That's why this isn't one fix — it's several, sized to the criticality of each path.

SSE: The Smallest Possible Annotation Change

Server-Sent Events are how the frontend gets near-real-time updates. The flow is: a write commits in the API, an SSE event is published to ActiveMQ, every API instance receives it on the topic, and each instance forwards it to its locally-connected SSE clients. If the broker drops a message, the client doesn't get the update via that path — but the SSE client always reconnects with a Last-Event-Id header, and the API replays missed events from an in-memory ring buffer (EventHistory). Loss is tolerable because the client recovers.

Spring's docs note that with @TransactionalEventListener, transactional resources may still be active and accessible after commit. "After commit" doesn't guarantee that the connection has already been returned to the pool. So a synchronous broker call inside an AFTER_COMMIT listener can still hold the JDBC connection while it waits.

That means two changes are needed:

1. Move the broker call outside the transaction boundary — switch from @EventListener to @TransactionalEventListener(phase = AFTER_COMMIT), so the publish happens after the commit completes rather than mid-transaction.

2. Make the broker call itself non-blocking on the listener thread — hand the publish off to a separate executor, so even if the listener thread is still associated with transactional resources, those resources can be released while the broker call proceeds elsewhere.

Together, these guarantee that a slow or unreachable broker can no longer pin a Hikari connection.

// Before — runs on the publishing thread, inside the transaction.
@EventListener
fun onOrderUpdated(event: OrderUpdatedEvent) { ... }
// After — runs on a different thread, and only after the transaction commits.
@TransactionalEventListener(phase = AFTER_COMMIT)
fun onOrderUpdated(event: OrderUpdatedEvent) {
    ssePublishExecutor.execute {
        jmsPublisher.publish(event.toMessage())
    }
}

This works for SSE because SSE losses are tolerable. It's not enough for anything that must not be lost.

FCM: Remove the Wrapping Transaction, Keep the Reads Short

The push notifications listener was already on @TransactionalEventListener(AFTER_COMMIT). The remaining problem was an inner transaction wrapping the listener body:

@TransactionalEventListener(AFTER_COMMIT)
@Transactional(propagation = Propagation.REQUIRES_NEW)   // ← the problem
fun onOrderAssigned(event: OrderAssignedEvent) {
    val recipients = userService.findRecipientsFor(event.orderId)
    pushClient.sendPush(recipients, event.payload)        // asynchronous HTTPS
}

The REQUIRES_NEW opens a fresh transaction at the start of the listener and holds it until the FCM HTTP call returns. Remove it. The listener body now runs with no transaction. Each method it calls handles its own transactional needs:

@Transactional(readOnly = true)
fun findRecipientsFor(orderId: UUID): List<User> { ... }

The flow now looks like: listener fires after commit → short read transaction → connection returned → short read transaction → connection returned → pushClient.sendPush returns immediately and the HTTPS work runs on FCM's own async executor. A Firebase regional incident now blocks neither a Hikari connection nor a Tomcat thread.

This is a deliberately conservative change. FCM isn't behind an outbox or on a queue — push notifications are best-effort by nature, and the existing code already makes that bargain. The point is just to stop letting Firebase's bad afternoons drain the database pool.

SMS: Split the Dispatcher Into Phases

SMS sending was already half-fixed by design. When a feature wants to send an SMS, it doesn't call the gateway directly — it inserts a row into a scheduled_messages table in the caller's transaction and returns. The actual HTTPS call is made later by a separate scheduled dispatcher. The user-facing request never hits the upstream provider.

That part was already correct. The problem was inside the dispatcher itself. The original loop opened one transaction per batch and held it across every gateway call. The fix splits the work into three short transactions and one transaction-free HTTP call:

@Scheduled(fixedDelayString = "...")
fun dispatchPendingMessages() {
    // Phase 1: short transaction, read pending IDs only.
    val pendingIds: List<UUID> = transactionTemplate.execute {
        repository.findPendingMessageIds()
    }
    // Phase 2: process each ID outside any DB transaction.
    pendingIds.forEach { id ->
        processor.processMessage(id)
    }
}

processMessage itself is the second important piece. It opens a REQUIRES_NEW transaction only to flip status from SCHEDULED to SENDING and to read a snapshot of the fields the gateway needs. It returns a plain value object — not a JPA entity. The gateway HTTPS call then runs with no transaction at all. A second REQUIRES_NEW transaction writes the final status once the gateway returns.

fun processMessage(messageId: UUID) {
    val snapshot = messageService.markAsSending(messageId) ?: return    // short tx
    try {
        val result = smsGateway.send(                                   // no tx
            snapshot.toNumber, snapshot.text, snapshot.fromNumber,
        )
        messageService.updateFinalStatus(messageId, result)             // short tx
    } catch (ex: Exception) {
        messageService.markAsFailed(messageId, ex.message)              // short tx
    }
}
data class MessageSnapshot(
    val toNumber: String,
    val text: String,
    val fromNumber: String?,
)

The snapshot matters. If markAsSending returned the JPA entity, that entity would be detached the moment its transaction commits, and any lazy-loaded field accessed later (during the gateway call, for example) would throw LazyInitializationException. The snapshot carries exactly the fields the gateway needs and nothing else, which makes it physically impossible to accidentally drag a JPA session into the HTTP call.

An upstream regional outage now holds, at worst, a single in-flight HTTPS call per concurrent dispatcher tick. It does not hold a Hikari connection.

S3: Don't Carry the Bytes Through the Application

For object storage, the fix isn't a transactional change at all — it's an architectural one. Instead of streaming the user's bytes through the application server and into S3, issue a presigned upload URL and let the browser upload directly to the bucket.

The flow:

1. Client calls POST /upload-intent with mime type and max size.

2. Backend writes a draft file-metadata row inside its (short) transaction, asks the storage SDK to generate a presigned upload URL constrained by storage key, mime type, content-length range, and a TTL, and returns the URL plus any required form fields to the client.

3. Client uploads the file directly to the storage endpoint using the presigned URL. The application server is not in the data path.

4. When the client confirms the upload, the backend (in another short transaction) flips the file's status from PENDING_UPLOAD to UPLOADED.

Two things this gets you. First, the application never holds a database connection while bytes move over the wire — there are no bytes moving through the application. Second, a slow user network or a slow storage region affects only that user (their progress bar moves slowly), not other unrelated requests.

This was already the design for new client-driven uploads. The discipline is to make sure no feature builds new uploads on the "stream through the server" path, and to fail closed if anyone tries to introduce one.

Webhooks: The Outbox Pattern

Webhooks are the case where SSE-style "loss is tolerable" is wrong. If a customer is subscribed to a business event and delivery silently fails, their downstream system is out of sync. A single-instance broker can't carry durability here.

The standard fix is the transactional outbox pattern. The mechanism, briefly, for readers who haven't seen it:

When a business operation needs to publish a message that must not be lost, the producing transaction inserts the message into an outbox table in the same database, in the same transaction as the business write. Either both commit or neither does — the message is now durable on the same storage as the business state. A separate background poller reads the outbox table, dispatches the messages downstream (to a broker, to an HTTP endpoint, wherever), and marks them processed.

Because the message is in the database, no broker outage can lose it. The poller can crash, restart, fall behind — the messages stay in the table until they're successfully delivered. Retries, dead-lettering, and ordering are properties of the table, not of the broker.

The producer side is one extra line inside the existing business transaction:

@Transactional
fun createOrder(request: OrderRequest): Order {
    val order = orderRepository.save(request.toEntity())     // business write
    outboxService.enqueue(                                   // outbox write (same tx)
        aggregateId = "order:${order.id}",
        eventType = "order.created",
        payload = objectMapper.writeValueAsString(order),
    )
    return order
}

outboxService.enqueue(...) is annotated @Transactional(MANDATORY)— it refuses to run outside an existing transaction. It writes a row to the outbox table and returns. The business write and the outbox row commit together, atomically, on the same connection.

The consumer side is a scheduled poller that runs every 5 seconds. It claims a batch of pending records using SELECT ... FOR UPDATE SKIP LOCKED, processes each in its own transaction (so one bad message doesn't block the others), and lets a registered handler decide what "processing" means. For webhooks, the handler resolves the subscribed endpoints for the event type and publishes one broker queue message per subscriber. From there, a separate consumer makes the actual HTTP call to the customer's endpoint with retry, signing, and dead-lettering.

The chain of guarantees:

• Database → outbox row: atomic, transactional, can't be lost.
• Outbox row → broker queue: at-least-once. The poller commits "processed" only after the broker accepts the message. If the broker is down, the row stays PENDING and the next poll tries again.
• Broker queue → customer HTTP endpoint: at-least-once with explicit retry counters and dead-letter routing. A single-instance broker outage delays delivery; it does not lose messages, because the canonical state is in the outbox row, not the broker.

The cost is straightforward and acceptable. Latency for webhook delivery is bounded below by the poller's 5-second tick. The outbox table grows and is cleaned up on a schedule (hourly, 7-day retention by default). And because the poller uses the same short-transaction shape as everything else, it composes naturally with the rest of the system.

Preventing Recurrence

Finding and fixing the bugs is half the work. The other half is making sure the next instance of the same pattern can't ship quietly.

HikariCP configuration.

spring:
  datasource:
    hikari:
      maximum-pool-size: 50
      leak-detection-threshold: 25000  # gives stack trace before pool exhaustion

Enable Hikari's leak detection in staging, and in production too if you can tolerate the log noise. Pick a threshold longer than your typical worst-case connection hold (the p99 — the time 99% of healthy connections finish within), so it only triggers on real outliers. Treat it as a diagnostic alarm, not as proof of a leak: when it fires, you get the stack trace of the thread still holding the connection — exactly the evidence that's missing during the incident itself.

An architecture rule. ArchUnit (or your equivalent) enforces the "no external I/O inside @Transactional" rule at build time:

@ArchTest
val transactionalCodeMustNotCallExternalIo: ArchRule =
    methods()
        .that().areAnnotatedWith(Transactional::class.java)
        .or().areDeclaredInClassesThat().areAnnotatedWith(Transactional::class.java)
        .should(notTransitivelyCall(
            FirebaseMessaging::class.java,
            S3Client::class.java,
            JmsTemplate::class.java,
            RestTemplate::class.java,
            // ...whatever else moves bytes off-server
        ))

The exact DSL depends on your version of ArchUnit and how strict you want to be about transitive calls. The point is: name the external clients explicitly, and make the build fail when one of them appears inside a transaction. Two-minute lookup at PR time, instead of a 15-minute outage at 2am.

An alert that catches it in production. Even with config and rules in place, you want to know before users do:

• Alert on hikaricp_connections_pending > 0 sustained for more than 30 seconds. Pending connections are threads waiting for a connection; sustained pending means the pool is saturated. This will fire long before the 30-second timeout starts surfacing as user errors.
• Alert on hikaricp_connections_active / hikaricp_connections_max > 0.8 sustained for more than 1 minute. Saturation, not exhaustion — but it's the warning sign.

Both are off-the-shelf Micrometer metrics. The thresholds are starting points; tune them to your traffic.

What This All Adds Up To

This isn't a one-bug problem and it isn't a one-fix problem. It's a category of bugs — external I/O inside a transaction — that quietly accumulates in any sufficiently old Spring codebase, every instance harmless until an upstream has a bad day. And the fixes form a menu, not a single recipe:

• For SSE: tolerate loss, use @TransactionalEventListener(AFTER_COMMIT) with handoff to a separate executor. Cheap, sufficient.
• For FCM: remove the wrapping transaction, make inner reads short and read-only. Best-effort by nature, no database connection held during the HTTP call.
• For SMS: split the dispatcher into phases, snapshot the row into a DTO, send outside any transaction. A scheduled_messages table covers durability; the dispatcher just keeps connections short.
• For S3: don't carry the bytes. Presigned POST, client uploads directly.
• For webhooks: full transactional outbox. Durable on the database, survives broker outages.

The unifying principle is small enough to fit on a sticky note: a database transaction must not enclose a network call to a system the database doesn't manage. The size of the fix depends on how much you care if the message is lost, how big the message is, and what infrastructure you can rely on. Pick the smallest pattern that matches your delivery guarantees — and enforce it at build time, because the next instance of this bug will look harmless on review.

For Your Runbook

When HikariCP is exhausted but database metrics look healthy, the diagnostic checklist at the top of this article is the fastest path to the cause.

The fixes are sized to delivery guarantees:

• Loss-tolerant (SSE) → post-commit listener with executor handoff.
• Best-effort (FCM) → remove the wrapping transaction; short read-only inner transactions; HTTP call outside.
• Durable but not user-facing (SMS) → short-phase dispatcher; snapshot DTOs; HTTP call between transactions.
• Large payloads (S3) → presigned upload directly from client to storage.
• Must-not-lose (webhooks) → transactional outbox; poller dispatches; broker becomes optional.

Pick the smallest pattern that matches the guarantee you actually need. Enforce "no external I/O inside @Transactional" at build time with ArchUnit. Alert on hikaricp_connections_pending > 0 sustained for 30 seconds. Set leak-detection-threshold above your p99 connection-hold time.

About me: Olga Ermolaeva is a technical solution owner on the Tourfold team at OpenResearch. https://openresearch.com/