DEV Community

R (RYUSLOG DEV)
R (RYUSLOG DEV)

Posted on Originally published at ryuslog.com

nginx 502 Bad Gateway: reading the upstream error line correctly

nginx returns its own error page instead of the application's:

502 Bad Gateway
Enter fullscreen mode Exit fullscreen mode

and the error log names the upstream precisely:

[error] 31#31: *7 connect() failed (111: Connection refused) while connecting to upstream
[error] 31#31: *9 upstream prematurely closed connection while reading response header
Enter fullscreen mode Exit fullscreen mode

Those two lines are different bugs. A 502 always means the same thing at one level — nginx accepted the client connection and could not get a valid response from the upstream — but the log text tells you which phase failed: connect, send, or reading the response header. Reading the phase is faster than reading the status code.

Map the log line to a cause

Error log text Cause Fix
connect() failed (111: Connection refused) Nothing listening at that host and port Correct the host/port, or start the upstream
connect() failed (110: Connection timed out) Packets dropped: wrong network, firewall, security group Check routes and service exposure
upstream prematurely closed connection Upstream died or timed out while answering Read upstream logs; raise its timeout
no live upstreams while connecting Every backend failed its passive health checks Fix one backend or tune max_fails
upstream sent too big header Response header larger than the buffer Raise proxy_buffer_size
recv() failed (104: Connection reset by peer) Kernel reset, often a half-closed keepalive socket Check keepalive on both sides
SSL_do_handshake() failed TLS mismatch to the upstream Fix the scheme and SNI

The mistake that produces most of these: localhost across containers

Inside a container, 127.0.0.1 is the container itself. In Compose, the app is reachable by its service name over the network Docker creates. So this config inside an nginx container returns 502 with errno 111 every time, because nginx has nothing on port 3000:

location / {
    proxy_pass http://127.0.0.1:3000;   # wrong inside a container
}
Enter fullscreen mode Exit fullscreen mode

The correct target is the service name:

location / {
    proxy_pass http://app:3000;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Enter fullscreen mode Exit fullscreen mode

Test it from the right namespace. curl localhost:3000 on the host proves nothing about what nginx can reach:

docker compose exec proxy getent hosts app
docker compose exec proxy wget -qO- http://app:3000/
Enter fullscreen mode Exit fullscreen mode

If getent prints nothing, the two containers are not on the same network. Check the Compose file for a shared networks key before touching nginx.

Start-up order is not readiness

depends_on waits for the container to start, not for the process to listen. nginx starts first, fails to connect, and keeps returning 502 until you reload it. Use a health check and a condition:

services:
  app:
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
      interval: 5s
      retries: 10
  proxy:
    depends_on:
      app:
        condition: service_healthy
Enter fullscreen mode Exit fullscreen mode

For a target whose IP changes — Kubernetes, ECS, docker compose up --scale — a static proxy_pass keeps the first resolved address forever. Add a resolver so nginx re-resolves per request:

resolver 127.0.0.11 valid=10s ipv6=off;
set $upstream http://app:3000;
proxy_pass $upstream;
Enter fullscreen mode Exit fullscreen mode

Header buffers: the 502 that only hits logged-in users

A large Set-Cookie or a JWT in a header can exceed nginx's 4 KB default header buffer. The log says upstream sent too big header, and anonymous requests keep working, which makes it look like an application bug:

proxy_buffer_size 16k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
large_client_header_buffers 4 16k;
Enter fullscreen mode Exit fullscreen mode

The same symptom with a 400 instead of a 502 is usually the client header being too large, which large_client_header_buffers covers.

Timeouts that look like crashes

An application slower than proxy_read_timeout (60 seconds by default) is closed by nginx; the app logs a broken pipe and the client sees 504. Align the app's own timeout, proxy_read_timeout, and whatever load balancer sits in front of nginx. Raising only one moves the failure to the next hop.

PHP-FPM after a restart

With fastcgi_pass unix:/run/php/php8.3-fpm.sock, a 502 usually means the socket path in the pool config differs from the one nginx uses, or the pool ran out of children. Check the actual socket with ss -xl | grep php, and raise pm.max_children when the log shows server reached pm.max_children.

Things not to blame

  • nginx being broken. It is reporting an upstream failure; the upstream is the subject.
  • The nginx version. A changed upstream port or a new health endpoint is far more common than a regression after an upgrade.
  • Reloading after a DNS change without a resolver. A static proxy_pass host is resolved at start-up; a reload is needed for the new address, and a resolver for changes after that.
  • Editing /etc/nginx/nginx.conf in the official image. The bundled /etc/nginx/conf.d/default.conf has a server block on port 80 that shadows yours.

A 502 that disappears when you test from the host is a network-scope problem every time.

The full walkthrough, with a Compose setup that reproduces each of these log lines on purpose, is on my site: nginx 502 Bad Gateway.

References: ngx_http_proxy_module, ngx_http_upstream_module.

Top comments (0)