The Silent 350-Second Killer: How an AWS NAT Gateway Cost Us $50k

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

It was 2:17 AM when the support Slack channel exploded. Three vendors had just been paid twice for their monthly bulk payouts. By morning, the number was twelve. By the end of the week, we had accidentally disbursed over $50,000 in duplicate payments. Our CFO was furious, and our banking partner was starting to ask uncomfortable questions.

I had that familiar, sickening pit in my stomach—the kind you get when you realize the bug isn't a simple logic error, and the clock is ticking while real money keeps leaking.

Our architecture was standard: Django, Celery, and PostgreSQL hosted on AWS. When a payout batch was approved, a Celery worker picked up the task and made a synchronous HTTP POST request to our legacy banking partner's API. We had explicitly set the requests timeout to 10 minutes (timeout=600) because the bank's maximum processing SLA was 8 minutes. We assumed we had a safe 2-minute buffer.

The Idempotency Trap

The first thing we investigated was idempotency. Why was the bank processing retries as new payouts? I checked the codebase.

headers = {'X-Idempotency-Key': str(uuid.uuid4())}
response = requests.post(BANK_URL, json=payload, headers=headers, timeout=600)

There was our first, embarrassing mistake. We were generating a new UUID directly inside the task execution instead of deriving it from the database record. When Celery retried a failed task, it generated a fresh key. The bank happily accepted it as a brand new batch.

We quickly hotfixed this to use a deterministic hash based on the database ID (uuid.uuid5). That stopped the bleeding, but it didn't give me any peace of mind. We had patched the symptom, not the disease. I still had absolutely no idea why the tasks were failing and retrying in the first place.

We blamed PostgreSQL locks. I spent hours analyzing pg_stat_activity, but the database was healthy.

We blamed the bank. We got on a tense call with their engineers. They pulled up their logs and proved that they were receiving our requests and returning a 200 OK.

"Your endpoint is just incredibly slow," I argued. "Some of these batches take over 6 minutes."
"Yes," their lead engineer replied. "It's a legacy mainframe. It takes 6 minutes. But we always return the success response."

The Investigation

If the bank was sending the response, why was our Python worker reporting a ReadTimeout after 10 minutes? Where was the data going?

I stopped looking at application logs and started looking at the network. I SSHed into a production worker, installed tcpdump, and waited for a large bulk payout to trigger.

When it finally happened, I opened the packet capture in Wireshark. The transaction told a very specific story:

10:15:00.000 IP worker.local:43912 > bank.api.com:443: Flags [P.], seq 1:1500, ack 1, length 1500 (HTTP POST)
10:15:00.124 IP bank.api.com:443 > worker.local:43912: Flags [.], ack 1500 (TCP ACK)
... [ABSOLUTE SILENCE] ...
10:25:00.000 Python worker hits 600s timeout, closes socket.

The bank acknowledged receiving our POST payload. Then, for exactly 10 minutes, there was dead silence. No response payload. No FIN. No RST.

But wait. If the bank's mainframe finished processing the payout around the 6-minute mark, why didn't I see their HTTP 200 OK packet in my trace?

The Silent Killer

Then I remembered our infrastructure diagram. Our worker nodes were in a private AWS subnet. All outbound traffic went through an AWS NAT Gateway.

I dug into the AWS documentation and found the culprit: AWS NAT Gateways have a fixed idle timeout of 350 seconds.

The timeline matched perfectly.

1. At 10:15:00, our worker sent the HTTP POST. The NAT Gateway tracked the connection.
2. The bank's mainframe started processing. Because it was synchronous, no packets flowed in either direction while it chugged away.
3. At 10:20:50 (exactly 350 seconds later), the NAT Gateway's idle timer expired. It silently deleted the connection state.
4. At 10:21:15 (around the 6-minute mark), the bank finished and sent the HTTP 200 OK response.
5. The NAT Gateway received the bank's packet. But since its state table had expired, it considered the packet invalid. It dropped the response and sent a TCP RST packet back to the bank.
6. Our worker never saw that RST packet. It saw absolutely nothing. It just sat there, blindly waiting for data until its own 10-minute application timeout finally fired.

Two systems silently disagreed about whether a TCP connection still existed. That disagreement cost us $50,000.

The Fix

The fix wasn't an architectural overhaul. We just needed to trick the NAT Gateway into keeping the state alive.

We needed TCP Keepalive. This is a low-level socket option that tells the Linux kernel to send an empty TCP probe packet if the connection is idle. By sending a probe before the 350-second mark, we could reset the NAT Gateway's timer.

We injected the socket options directly into urllib3 via the requests adapter:

import socket
from requests.adapters import HTTPAdapter

# Enable keepalive, probe every 60 seconds
keepalive_options = [
    (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
    (socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60),
    (socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10),
    (socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 5)
]

# We explicitly avoid max_retries here to prevent automatic replays on side-effecting payment operations
adapter = HTTPAdapter()
adapter.poolmanager.connection_pool_kw['socket_options'] = keepalive_options

session.mount('https://', adapter)

We deployed the configuration and watched the metrics.

• Timeout rate before the patch: 4.2% of bulk payouts.
• Timeout rate after the patch: 0%.
• Duplicate payments: 0.

We are currently working with the bank to migrate this flow to an asynchronous webhook architecture so we never have to hold a connection open for 6 minutes again. I still get a small jolt of anxiety every time I see a long-running synchronous HTTP call in a pull request.

The biggest takeaway for me? Never trust the network to close a connection for you. If you are forced to build a synchronous integration that takes minutes to complete, you have to defend your sockets. A few bytes of TCP Keepalive stabilized the system, but it was a harsh reminder of how leaky our abstractions really are.