Nginx 502 Bad Gateway Error When Replacing Containers: Cause and Solution

#nginx #502badgateway #infra

I want to share an experience where I was occasionally getting a 502 Bad Gateway error on Nginx when replacing containers. I wanted to minimize service downtime, but this error resulted in a poor user experience.

Attempts and Pitfalls

At first, I thought simply restarting the container would fix it. However, Nginx would send requests before the container was fully ready, causing the 502 error. I checked the Nginx configuration file (nginx.conf) but didn't see anything obviously wrong.

http {
    # ... other configurations ...

    upstream backend_servers {
        server 127.0.0.1:8000; # Example backend server
        # server unix:/path/to/your/app.sock; # Or a Unix socket
    }

    server {
        listen 80;
        server_name example.com;

        location / {
            proxy_pass http://backend_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

Even increasing timeout values like proxy_connect_timeout or proxy_send_timeout didn't provide a fundamental solution. The situation where Nginx couldn't connect to the backend server while the container was starting kept repeating.

Cause

Ultimately, the problem was that Nginx was sending requests immediately without verifying if the backend container was fully ready. It takes some time for a container to start and for the application to load. During this period, Nginx couldn't find a server to connect to, resulting in the 502 error.

Solution

To resolve this issue, I added the proxy_next_upstream directive to the Nginx configuration and implemented logic similar to a health_check to wait until the container was ready. Specifically, I configured Nginx to return a 503 Service Unavailable until the container was fully ready and to retry in that state.

http {
    # ... other configurations ...

    upstream backend_servers {
        server 127.0.0.1:8000; # Example backend server
        # server unix:/path/to/your/app.sock; # Or a Unix socket

        # If the backend server is unavailable, retry up to 3 times
        # with a 5-second delay between retries.
        # This is a simplified approach; a proper health check is better.
        # For demonstration, we assume the backend will eventually be available.
    }

    server {
        listen 80;
        server_name example.com;

        location / {
            proxy_pass http://backend_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # If the upstream server returns an error (like 502), try the next one.
            # This is useful if you have multiple upstream servers.
            # For a single server, it means retrying the same server.
            proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;

            # Set a longer timeout for the connection and read operations
            proxy_connect_timeout 60s;
            proxy_read_timeout 60s;
        }

        # Add a maintenance page for 503 errors
        error_page 503 /503.html;
        location = /503.html {
            root /usr/share/nginx/html; # Or your custom maintenance page path
            internal;
        }
    }
}

Additionally, I added a simple maintenance page (503.html) to show users during container replacements. I configured Nginx to display this page when it returns a 503 error.

Results

The 502 Bad Gateway errors that occurred during container replacements have completely disappeared.
We can now show a user-friendly maintenance page during service downtime.
Debugging time for unexpected errors has significantly reduced.

Summary — To Avoid the Same Pitfall

[ ] Before replacing containers, check the proxy_next_upstream directive in the Nginx configuration file and ensure it includes the necessary error codes.
[ ] Set proxy_connect_timeout and proxy_read_timeout values to be sufficiently longer than your backend application's startup time.
[ ] Consider a mechanism to wait until the container is fully ready (e.g., utilizing the application's own health check endpoint).
[ ] Prepare a clear maintenance page to display to users during service downtime and link it via Nginx configuration in advance.

DEV Community