DEV Community

Cover image for Why Your Services Randomly Throw 502 Bad Gateway Behind AWS ALB (And the 3-Line Fix)
Abdul Rehman Khan
Abdul Rehman Khan

Posted on Edited on

Why Your Services Randomly Throw 502 Bad Gateway Behind AWS ALB (And the 3-Line Fix)

If your microservices behind an AWS Application Load Balancer (ALB) are intermittently throwing 502 Bad Gateway errorsβ€”especially during low-traffic valleys, background cron runs, or right after a quiet spellβ€”stop debugging your memory limits.

Your server is not crashing, your CPU is not spiking, and your security groups are completely fine.

You are hitting a microsecond TCP in-flight race condition.

By default, an AWS ALB maintains idle backend TCP connections for 60 seconds. If your backend web framework (Node.js, Python Uvicorn, or Go) has a default keep-alive timeout of 50 or 60 seconds, your backend process will close the idle socket with a TCP FIN packet at the exact microsecond the ALB decides to forward a new client request down that same socket.

The packets cross on the wire. The ALB receives a TCP RST (Connection Reset) on an in-flight request, gives up immediately, and dumps an HTTP 502 directly into your user's lap.

Here is the exact packet-level mechanism, the Athena SQL query to prove it from your access logs, and the 3-line configuration fix across every major web runtime.


The Anatomy of the Ghost 502

An AWS Application Load Balancer maintains two distinct connection pools:

  1. Front-End Connections: Between the client browser and the ALB listeners.
  2. Back-End Connections: Between the ALB and your target containers (ECS, EKS, EC2, or Lambda).

To minimize TLS handshakes and TCP connection overhead, the ALB keeps backend TCP sockets alive and reuses them across multiple subsequent HTTP requests.

By default, the ALB's Idle Timeout is configured to 60 seconds.

Here is how the microsecond collision unfolds:

The Step-by-Step Packet Collision

  1. At t = 0.000s: A user requests /api/v1/orders. The ALB opens or reuses a backend socket, proxies the request, receives a 200 OK, and keeps the socket sitting idle in its connection pool.
  2. At t = 50.000s: Fifty seconds elapse with no new traffic on this specific target. Node.js's default server.keepAliveTimeout (which defaults to 5 seconds in older Node, and 50–60 seconds in many custom configs) expires.
  3. At t = 50.001s: The backend runtime decides the socket is stale and sends a TCP FIN packet toward the ALB to initiate a graceful TCP teardown.
  4. At t = 50.002s: A new user clicks "Checkout". The ALB looks at its pool, sees an apparently open connection that has only been idle for 50 seconds (well below its 60-second limit), and forwards POST /api/v1/checkout down the socket.
  5. At t = 50.003s (The Collision): The FIN packet traveling leftward crosses the POST HTTP payload traveling rightward on the network wire.
  6. At t = 50.004s: The backend server receives HTTP data on a socket it has already half-closed. The Linux kernel immediately rejects the invalid packet and replies with a TCP RST (Connection Reset).
  7. At t = 50.005s: The ALB receives the RST while waiting for the HTTP response headers. Because the request was already sent to the target, the ALB refuses to automatically retry it on another target (to avoid idempotency violations on non-idempotent POST requests).
  8. At t = 50.006s: The ALB immediately generates an HTTP 502 Bad Gateway back to the client.

The entire failure takes less than 1 millisecond.


How to Prove It in AWS Athena (The Smoking Gun)

Before altering your production code, verify this exact failure mode in your raw ALB access logs.

When this race condition occurs, the ALB access log emits a very specific behavioral fingerprint:

  • elb_status_code: 502
  • target_status_code: - (hyphen, meaning the target never sent an HTTP status code)
  • request_processing_time: 0.000 or -1
  • target_processing_time: -1 (the socket broke before the target processed the headers)

Run this SQL query against your ALB access log table in Amazon Athena:

SELECT 
    time,
    client_ip,
    target_ip,
    request_verb,
    request_url,
    elb_status_code,
    target_status_code,
    request_processing_time,
    target_processing_time,
    response_processing_time
FROM "alb_access_logs"
WHERE elb_status_code = 502
  AND target_status_code = '-'
  AND target_processing_time = -1
  AND time > current_timestamp - interval '24' hour
ORDER BY time DESC
LIMIT 50;
Enter fullscreen mode Exit fullscreen mode

If you see rows matching 502 with target_status_code = '-' and target_processing_time = -1, you are 100% suffering from the keep-alive timeout race condition.


The Golden Fix Rule

To eliminate this race condition forever, you must enforce a strict asymmetry:

πŸ›‘οΈ The Architectural Rule:

Backend Keep-Alive Timeout (65s) > ALB Idle Timeout (60s)

The entity further upstream (the ALB) must always close idle connections before the downstream backend server does. If the ALB closes the connection first via a graceful FIN, it will never attempt to dispatch a client request on a dying socket.

Since AWS ALB defaults to 60 seconds, your backend keep-alive timeout must be set to at least 65 seconds (a 5-second safety buffer).


Production Configurations Across Frameworks

1. Node.js (Express & Fastify)

In Node.js, there is a dangerous secondary gotcha: headersTimeout must always be strictly greater than keepAliveTimeout. In Node 18+, if headersTimeout <= keepAliveTimeout, the Node HTTP runtime will throw silent ERR_HTTP_HEADERS_TIMEOUT exceptions.

Express.js Fix:

// server.js
const express = require('express');
const app = express();

app.get('/api/health', (req, res) => res.json({ status: 'ok' }));

const server = app.listen(3000, () => {
    console.log('Server running on port 3000');
});

// CRITICAL FIX FOR AWS ALB:
// 1. Set keepAliveTimeout higher than ALB's 60s idle timeout
server.keepAliveTimeout = 65000; // 65 seconds

// 2. Set headersTimeout strictly higher than keepAliveTimeout (Node 18+ requirement)
server.headersTimeout = 66000;   // 66 seconds
Enter fullscreen mode Exit fullscreen mode

Fastify Fix:

// server.js
const fastify = require('fastify')({
    logger: true,
    serverFactory: (handler) => {
        const server = require('http').createServer(handler);
        server.keepAliveTimeout = 65000;
        server.headersTimeout = 66000;
        return server;
    }
});

fastify.get('/api/health', async () => ({ status: 'ok' }));

fastify.listen({ port: 3000, host: '0.0.0.0' });
Enter fullscreen mode Exit fullscreen mode

2. Python (FastAPI / Uvicorn & Gunicorn)

Uvicorn defaults to a 5-second keep-alive timeout (--keep-alive 5), which causes catastrophic 502 storms under bursty or low-frequency ALB traffic.

Running Uvicorn via CLI:

uvicorn main:app --host 0.0.0.0 --port 8000 --timeout-keep-alive 65
Enter fullscreen mode Exit fullscreen mode

Running Gunicorn with Uvicorn Workers (gunicorn.conf.py):

# gunicorn.conf.py
bind = "0.0.0.0:8000"
workers = 4
worker_class = "uvicorn.workers.UvicornWorker"

# CRITICAL FIX FOR AWS ALB:
keepalive = 65  # Seconds to wait for requests on a Keep-Alive connection
Enter fullscreen mode Exit fullscreen mode

3. Go (net/http)

Go's http.Server struct provides an explicit IdleTimeout parameter. If left at 0, it inherits ReadTimeout, which can prematurely sever idle connections.

package main

import (
    "net/http"
    "time"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/api/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        w.Write([]byte(`{"status":"ok"}`))
    })

    server := &http.Server{
        Addr:         ":8080",
        Handler:      mux,
        ReadTimeout:  15 * time.Second,
        WriteTimeout: 15 * time.Second,
        // CRITICAL FIX FOR AWS ALB:
        // Must be greater than ALB's 60s idle timeout
        IdleTimeout:  65 * time.Second,
    }

    server.ListenAndServe()
}
Enter fullscreen mode Exit fullscreen mode

4. Nginx (Reverse Proxy / Ingress)

If you run an Nginx container as a sidecar or gateway behind the ALB:

http {
    # Keep idle connections to ALB alive for 65s
    keepalive_timeout 65s;

    # Ensure requests over a single connection are allowed to cycle
    keepalive_requests 1000;

    upstream backend_nodes {
        server 127.0.0.1:3000;
        # Enable connection pooling between Nginx and local app
        keepalive 32;
    }
}
Enter fullscreen mode Exit fullscreen mode

Alternative Approach: Lowering ALB Idle Timeout

If you run a legacy compiled third-party container where backend keep-alive timeouts cannot be patched, you can invert the formula by reducing the ALB Idle Timeout in AWS:

  1. Navigate to EC2 Console β†’ Load Balancers.
  2. Select your Application Load Balancer.
  3. In the Attributes tab, click Edit attributes.
  4. Change Idle timeout from 60 seconds down to 15 seconds (or whatever is safely below your backend's unconfigurable timeout).
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    ALB TIMEOUT SUMMARY                      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Layer                    β”‚ Recommended Production Setting   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ AWS ALB Idle Timeout     β”‚ 60 seconds (AWS Default)         β”‚
β”‚ Backend KeepAliveTimeout β”‚ 65 seconds (+5s buffer)          β”‚
β”‚ Node.js headersTimeout   β”‚ 66 seconds (Must exceed keepAlive)β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Enter fullscreen mode Exit fullscreen mode

Production Pre-Flight Checklist

Before closing out your ticket on this incident:

  • [ ] Run the Athena query on your ALB logs to verify sporadic 502 errors correspond to target_status_code = '-'.
  • [ ] Ensure backend keepAliveTimeout is set to 65 seconds (or at least 5s above the ALB idle timeout).
  • [ ] For Node.js targets, verify server.headersTimeout is explicitly set to 66 seconds to satisfy the Node 18+ invariant.
  • [ ] Deploy to your staging environment and run an idle-connection soak test (dispatch one request, wait 55 seconds, dispatch a second request).
  • [ ] Verify that CloudWatch Metric HTTPCode_Target_5XX_Count returns to zero.

Eliminating this single configuration discrepancy resolves one of the most frustrating, transient production bugs in modern cloud architecture.


Published on Dev.to. If you found this post helpful, give it a ❀️ or πŸ¦„ and follow for more battle-tested cloud and distributed systems debugging playbooks.

Top comments (0)