DEV Community: Praneeth Kodumagulla

Docker Swarm Said “Insufficient Resources.” The Running Service Was Using 480 KiB

Praneeth Kodumagulla — Thu, 27 Aug 2026 14:30:00 +0000

I expected this task to schedule.

The running service was barely doing anything:

MemoryUsage=480KiB / 5.772GiB
MemoryPercent=0.01%
CPU=0.00%

Then I created another Docker Swarm service.

Swarm responded with:

0/1
Pending

and:

no suitable node (insufficient resources on 1 node)

At first glance, those observations seem incompatible.

They aren't.

The missing piece is that runtime memory usage and scheduler reservations answer different questions.

I built a small single-node Swarm lab to make that difference reproducible.

The result: a service consuming roughly 480 KiB can coexist with a scheduler that refuses another task for insufficient memory — because Swarm is making the placement decision from configured reservations, not from that service's current memory consumption.

The lab

I used a disposable single-node Docker Swarm with:

Docker client: 29.4.3
Docker server: 29.2.1
CPUs:          4
Node memory:   5.772 GiB

There were no existing Swarm services.

The workload itself was deliberately boring: an Alpine container sleeping in a loop.

The interesting part was the reservation.

For portability, the reproduction script calculates reservation sizes from whatever memory the Swarm node reports:

Service A       60%
Service B       50%   initially
Service B       20%   after the update

So the first placement attempt asks Swarm to accommodate:

60% + 50% = 110%

The second asks for:

60% + 20% = 80%

That gives us a controlled way to flip the scheduler decision by changing one reservation.

Phase 1: reserve 60%, consume almost nothing

On my 5.772 GiB node, Service A received a reservation of:

3.463 GiB

Swarm scheduled it successfully:

swarm-reservation-a   1/1

Then I checked its actual runtime usage:

Service=swarm-reservation-a
MemoryUsage=480KiB / 5.772GiB
MemoryPercent=0.01%
CPU=0.00%

That distinction matters.

The 3.463 GiB reservation is not saying:

this container is currently consuming 3.463 GiB.

It is part of the resource requirement Swarm considers when deciding where the task may be placed.

Docker's documentation makes another important distinction: --reserve-memory is not a runtime memory limit. If you need to enforce a maximum amount of memory a task can consume, that is what --limit-memory is for.

Phase 2: add another 50%

Next I created Service B with a reservation equal to 50% of node memory.

On this node:

Node capacity:          5.772 GiB

Service A reservation:  3.463 GiB
Service B reservation:  2.886 GiB
                        ---------
Combined reservation:  ~6.349 GiB

The workload itself had not suddenly become memory-hungry.

But the declared reservations no longer fit.

Swarm showed:

NAME                    REPLICAS
swarm-reservation-a     1/1
swarm-reservation-b     0/1

And docker service ps gave the important line:

Pending
"no suitable node (insufficient resources on 1 node)"

Meanwhile, the running lab service was still around:

480 KiB
0.01%

That is the contradiction I wanted the lab to isolate.

It is possible to look at runtime usage and think:

There is hardly any workload here.

while the scheduler is effectively answering:

I cannot satisfy the resource commitments you declared for another task.

Both statements can be true.

Phase 3: change one number

I didn't add memory.

I didn't add another Swarm node.

I didn't restart Docker.

I didn't change the application.

I changed only Service B's reservation:

50% → 20%

Now the scheduler math became:

Service A:  60%
Service B:  20%
            ---
Total:      80%

On the tested node:

Service A reservation:  3.463 GiB
Service B reservation:  1.154 GiB
                        ---------
Combined reservation:  ~4.617 GiB

That fits inside 5.772 GiB.

The result immediately changed to:

swarm-reservation-a   1/1
swarm-reservation-b   1/1

Service B moved from Pending to Running.

And now both lab containers were still tiny at runtime:

swarm-reservation-a   ~480 KiB
swarm-reservation-b   ~480 KiB

Why this isn't a Docker bug

Swarm is doing exactly what it was configured to do.

A reservation is information given to the scheduler about resources a task requires.

Docker documents the behavior directly: if no node has enough memory available to satisfy a service's reservation, the task stays pending.

So this:

docker stats

and this:

docker service ps

are giving us information about different layers of the system.

One tells us something about runtime consumption.

The other tells us whether the orchestrator can currently satisfy the task's scheduling requirements.

That's why:

low runtime usage

does not imply:

enough schedulable reserved capacity

The autoscaling trap

This becomes particularly interesting when infrastructure scaling and workload scheduling use different signals.

Imagine an external autoscaler whose scale-out rule is based on things such as:

host CPU utilization
host runtime memory utilization

while the workload scheduler is deciding placement from:

configured CPU reservations
configured memory reservations
placement constraints
available nodes

You can end up with:

Scheduler:
"insufficient resources"

Autoscaler:
"utilization is below the scale-out threshold"

Those answers do not necessarily contradict each other.

They may simply be measuring different kinds of pressure.

Important caveat: the lab in this article tests Docker Swarm scheduling and memory reservations. It does not implement or benchmark a particular infrastructure autoscaler. The autoscaling scenario is the operational implication of the scheduling behavior, not another result claimed by this experiment.

Reproduce it yourself

I packaged the experiment into a small public lab.

The script:

requires a disposable single-node Swarm
refuses to run if existing Swarm services are present
calculates reservation values from node memory
verifies the expected Pending state
captures the insufficient resources error
lowers only Service B's reservation
verifies the resulting Running state
removes only the services created by the lab

Run the reproducible Docker Swarm lab

A successful run ends with:

60% + 50% => Pending: VERIFIED
Reason => insufficient resources: VERIFIED
60% + 20% => Running: VERIFIED

LAB_RESULT=PASS

The repository also contains sanitized captured output, SHA-256 manifests, and a separate safety test proving that the script refuses to run in a non-empty Swarm without deleting the existing service.

The debugging lesson

When an orchestrated task refuses to schedule, don't reduce the question to:

How much CPU or memory is the machine using right now?

Also ask:

What did the workload request?
What did the scheduler reserve?
What capacity does the scheduler think remains?
What other placement requirements must be satisfied?
What signal is my infrastructure autoscaler actually watching?

The mental model I now use is:

Scheduler pressure ≠ runtime utilization.

That distinction can save a lot of time when an application looks lightweight but the orchestrator still says there is nowhere to put it.

One question for people running orchestrators

Have you run into the equivalent mismatch with Docker Swarm, Kubernetes requests, Nomad, ECS, or an external autoscaler — where scheduler pressure said one thing while runtime utilization said another?

I'm especially interested in what signal you eventually chose for scale-out when unschedulable work and host utilization disagreed.

References

AI assistance disclosure: I used AI to help structure and edit this article. I ran the lab myself, reviewed the commands and outputs, and verified the technical claims against Docker's documentation. The reproducible script and captured evidence are linked above.

How to Find Which Proxy Returned a 502 or 504 — A Reproducible NGINX Debugging Lab

Praneeth Kodumagulla — Tue, 25 Aug 2026 17:25:45 +0000

A 502 Bad Gateway or 504 Gateway Timeout tells you that something failed around a gateway boundary.

It does not necessarily tell you which gateway failed.

That distinction matters when a request crosses multiple layers:

Client
  |
  v
Edge proxy
  |
  v
API gateway
  |
  v
Application

If the client receives:

504 Gateway Timeout

several different things may have happened:

the gateway could not establish a connection to its upstream
the gateway connected successfully but waited too long for a response
an outer proxy could have timed out while an inner gateway was still waiting
the application could have continued processing after an upstream proxy had already abandoned the request

I built a small Docker lab with two NGINX proxies and a Python application to make these failure modes deterministic.

The most interesting result was one request for which the different layers recorded:

External client:  504
Edge NGINX:       504
Gateway NGINX:    499
Application:      eventually attempts 200

Those results are not contradictory.

They describe the same request from different points in the request path.

This lab shows how to reconstruct that path.

What we're building

The topology is deliberately simple:

                     localhost:8080
Client  ───────────► Edge NGINX
                         |
                         v
                    Gateway NGINX
                         |
                         v
                     Python API

For debugging, each layer is also exposed directly:

Application    localhost:8000
Gateway        localhost:18081
Edge           localhost:8080

I ran the lab on macOS using Colima with Docker Engine 29.2.1 on ARM64.

The containers used:

NGINX     1.31.4
Python    3.13.15

Container IP addresses shown later are specific to my run and should not be treated as fixed values.

Project structure

Create:

proxy-debug-lab/
├── app/
│   ├── app.py
│   └── Dockerfile
├── compose.yaml
├── edge/
│   └── nginx.conf
└── gateway/
    └── nginx.conf

Build a small backend that can be intentionally slow

The backend uses only Python's standard library.

app/app.py:

import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        parsed = urlparse(self.path)
        trace_id = self.headers.get("X-Request-ID", "-")

        if parsed.path == "/health":
            self.respond(
                200,
                {
                    "status": "healthy",
                    "trace_id": trace_id,
                },
            )
            return

        if parsed.path == "/fast":
            self.respond(
                200,
                {
                    "status": "ok",
                    "endpoint": "fast",
                    "trace_id": trace_id,
                },
            )
            return

        if parsed.path == "/slow":
            params = parse_qs(parsed.query)

            try:
                seconds = float(params.get("seconds", ["5"])[0])
            except ValueError:
                seconds = 5

            seconds = max(0, min(seconds, 30))

            print(
                f"layer=app trace={trace_id} "
                f"path={parsed.path} sleeping={seconds}s",
                flush=True,
            )

            time.sleep(seconds)

            self.respond(
                200,
                {
                    "status": "ok",
                    "endpoint": "slow",
                    "slept_seconds": seconds,
                    "trace_id": trace_id,
                },
            )
            return

        self.respond(
            404,
            {
                "status": "not_found",
                "path": parsed.path,
                "trace_id": trace_id,
            },
        )

    def respond(self, status, payload):
        body = (json.dumps(payload) + "\n").encode()

        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()

        try:
            self.wfile.write(body)
        except BrokenPipeError:
            print("layer=app client_connection_closed=true", flush=True)

    def log_message(self, fmt, *args):
        trace_id = self.headers.get("X-Request-ID", "-")

        print(
            f'layer=app trace={trace_id} message="{fmt % args}"',
            flush=True,
        )


if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8000), Handler)

    print("layer=app listening=0.0.0.0:8000", flush=True)

    server.serve_forever()

The endpoint we care about most is:

/slow?seconds=5

It accepts the request, sleeps for five seconds without sending response headers, and then attempts to return 200.

That lets us deliberately place shorter proxy timeouts in front of it.

The Dockerfile is minimal:

FROM python:3.13.15-alpine3.24

WORKDIR /app

COPY app.py .

EXPOSE 8000

CMD ["python", "-u", "app.py"]

Instrument the gateway

Before introducing failures, make the proxies observable.

gateway/nginx.conf:

events {}

http {
    map $http_x_request_id $trace_id {
        default $http_x_request_id;
        ""      $request_id;
    }

    log_format trace
        '$time_iso8601 '
        'layer=gateway '
        'trace=$trace_id '
        'status=$status '
        'upstream_status=$upstream_status '
        'request_time=$request_time '
        'upstream_response_time=$upstream_response_time '
        'upstream_addr=$upstream_addr '
        'request="$request"';

    access_log /var/log/nginx/access.log trace;
    error_log  /var/log/nginx/error.log notice;

    server {
        listen 80;

        add_header X-Gateway-Observed "true" always;

        location = /health {
            proxy_pass http://app:8000/health;

            proxy_set_header X-Request-ID $trace_id;
            proxy_read_timeout 2s;
        }

        location = /fast {
            proxy_pass http://app:8000/fast;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        # Reachable host, but nothing listens on port 9999.
        location = /bad-gateway {
            proxy_pass http://app:9999;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 2s;
        }

        # App sleeps for 5s, but Gateway waits only 2s.
        location = /gateway-timeout {
            proxy_pass http://app:8000/slow?seconds=5;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 2s;
        }

        # App sleeps for 5s and Gateway is willing to wait 10s.
        location = /edge-timeout {
            proxy_pass http://app:8000/slow?seconds=5;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        error_page 502 = @gateway_502;
        error_page 504 = @gateway_504;

        location @gateway_502 {
            default_type text/plain;

            add_header X-Gateway-Observed "true" always;
            add_header X-Debug-Generated-By "gateway" always;

            return 502 "gateway generated 502\n";
        }

        location @gateway_504 {
            default_type text/plain;

            add_header X-Gateway-Observed "true" always;
            add_header X-Debug-Generated-By "gateway" always;

            return 504 "gateway generated 504\n";
        }
    }
}

The important log fields are:

$status
$upstream_status
$request_time
$upstream_response_time
$upstream_addr

NGINX also exposes fields such as $upstream_connect_time and $upstream_header_time, which are useful additions in a production logging format.

Configure the outer edge proxy

edge/nginx.conf:

events {}

http {
    map $http_x_request_id $trace_id {
        default $http_x_request_id;
        ""      $request_id;
    }

    log_format trace
        '$time_iso8601 '
        'layer=edge '
        'trace=$trace_id '
        'status=$status '
        'upstream_status=$upstream_status '
        'request_time=$request_time '
        'upstream_response_time=$upstream_response_time '
        'upstream_addr=$upstream_addr '
        'request="$request"';

    access_log /var/log/nginx/access.log trace;
    error_log  /var/log/nginx/error.log notice;

    server {
        listen 80;

        add_header X-Edge-Observed "true" always;
        add_header X-Request-ID $trace_id always;

        # If Gateway itself returns an error, pass that response through.
        proxy_intercept_errors off;

        location = /health {
            proxy_pass http://gateway/health;

            proxy_set_header X-Request-ID $trace_id;
            proxy_read_timeout 3s;
        }

        location = /fast {
            proxy_pass http://gateway/fast;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        location = /bad-gateway {
            proxy_pass http://gateway/bad-gateway;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 5s;
        }

        # Give Gateway enough time to generate its own 504.
        location = /gateway-timeout {
            proxy_pass http://gateway/gateway-timeout;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        # Edge gives up first.
        location = /edge-timeout {
            proxy_pass http://gateway/edge-timeout;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 2s;
        }

        error_page 502 = @edge_502;
        error_page 504 = @edge_504;

        location @edge_502 {
            default_type text/plain;

            add_header X-Edge-Observed "true" always;
            add_header X-Request-ID $trace_id always;
            add_header X-Debug-Generated-By "edge" always;

            return 502 "edge generated 502\n";
        }

        location @edge_504 {
            default_type text/plain;

            add_header X-Edge-Observed "true" always;
            add_header X-Request-ID $trace_id always;
            add_header X-Debug-Generated-By "edge" always;

            return 504 "edge generated 504\n";
        }
    }
}

The X-Debug-Generated-By headers are lab instrumentation, not standard NGINX headers.

I added them so that the experiments make the response origin visually obvious.

In a real environment, correlation IDs and logs are much more important.

Run everything with Docker Compose

compose.yaml:

services:
  app:
    build:
      context: ./app

    ports:
      - "8000:8000"

  gateway:
    image: nginx:1.31.4-alpine3.24

    volumes:
      - ./gateway/nginx.conf:/etc/nginx/nginx.conf:ro

    ports:
      - "18081:80"

    depends_on:
      - app

  edge:
    image: nginx:1.31.4-alpine3.24

    volumes:
      - ./edge/nginx.conf:/etc/nginx/nginx.conf:ro

    ports:
      - "8080:80"

    depends_on:
      - gateway

Start the environment:

docker compose up -d --build

Then check:

docker compose ps

Validate the loaded NGINX configurations:

docker compose exec gateway nginx -t
docker compose exec edge nginx -t

Both should end with:

syntax is ok
test is successful

Establish a healthy baseline first

Always prove the healthy path before deliberately breaking it.

Test the application:

curl -i http://localhost:8000/health

Then Gateway → Application:

curl -i http://localhost:18081/health

Then the complete path:

curl -i http://localhost:8080/health

The complete request returned:

HTTP/1.1 200 OK
X-Gateway-Observed: true
X-Edge-Observed: true
X-Request-ID: d66701d1ec8700e4bf111ab99b49372d

I then measured a normal /fast request:

curl -sS -D - -o /dev/null \
  -w '\nstatus=%{http_code}\nconnect=%{time_connect}s\nttfb=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  http://localhost:8080/fast

One saved run produced:

status=200
connect=0.000248s
ttfb=0.010474s
total=0.010642s

Both proxies recorded successful upstream responses.

Gateway:

layer=gateway
status=200
upstream_status=200

Edge:

layer=edge
status=200
upstream_status=200

Now we have a control case.

Failure 1: Immediate upstream connection refusal

The /bad-gateway route points Gateway at:

app:9999

The application container exists and is reachable, but nothing is listening on that port.

Request it through the full chain:

curl -sS -D - http://localhost:8080/bad-gateway

The response was:

HTTP/1.1 502 Bad Gateway
X-Gateway-Observed: true
X-Debug-Generated-By: gateway
X-Edge-Observed: true

gateway generated 502

The measured request:

status=502
connect=0.000342s
ttfb=0.001889s
total=0.002027s

Gateway's error log explained why:

connect() failed (111: Connection refused) while connecting to upstream

Its access log showed:

layer=gateway
status=502
upstream_status=502
upstream_addr=172.20.0.2:9999

Edge recorded:

layer=edge
status=502
upstream_status=502

The sequence is therefore:

Client
  |
  v
Edge              reached successfully
  |
  v
Gateway           reached successfully
  |
  X
app:9999          connection refused

Gateway generates 502
Edge forwards 502

The client status alone does not show that sequence.

The adjacent logs do.

A backend being "down" did not always produce a 502

I also tried a less controlled experiment:

docker compose stop app

I initially expected another 502.

That did not happen in this environment.

The gateway attempted to connect to the application's previous container address, but the connection attempt did not immediately fail.

Instead, it waited until:

proxy_connect_timeout 2s;

expired.

The client received:

HTTP/1.1 504 Gateway Time-out
X-Debug-Generated-By: gateway

with:

status=504
connect=0.000293s
ttfb=2.003997s
total=2.004799s

Gateway logged:

upstream timed out ... while connecting to upstream

Compare that with the closed-port experiment:

Immediate connection refusal
    -> 502 in this lab

Connection attempt times out
    -> 504 in this lab

I would not generalize this into:

stopped container = 504

The network behavior of a stopped workload can vary by environment.

The useful lesson is:

"The backend is down" is not enough information to predict the status code. You need to determine how the upstream connection failed.

After this experiment I restored the application:

docker compose start app

Failure 2: Gateway connects successfully but times out reading the response

Now we create a very different failure.

The application waits:

5 seconds

Gateway waits:

2 seconds

Edge waits:

10 seconds

So Gateway must be the first layer to give up.

The route is:

location = /gateway-timeout {
    proxy_pass http://app:8000/slow?seconds=5;

    proxy_set_header X-Request-ID $trace_id;
    proxy_connect_timeout 2s;
    proxy_read_timeout 2s;
}

Request:

curl -sS -D - \
  -o /dev/null \
  -w '\nstatus=%{http_code}\nttfb=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  http://localhost:8080/gateway-timeout

The result:

HTTP/1.1 504 Gateway Time-out
X-Gateway-Observed: true
X-Debug-Generated-By: gateway
X-Edge-Observed: true

status=504
ttfb=2.005843s
total=2.005997s

A saved run measured:

status=504
connect=0.000261s
ttfb=2.004355s
total=2.005004s

The application proves that the request successfully reached it:

layer=app
path=/slow
sleeping=5.0s

Gateway logged:

upstream timed out (110: Operation timed out)
while reading response header from upstream

This is different from the previous connect-timeout error:

while connecting to upstream

So we have now produced two client-visible 504 responses during different phases:

Connection phase
    -> timeout while connecting to upstream

Response phase
    -> timeout while reading response header

According to the NGINX proxy module documentation, proxy_connect_timeout controls the timeout for establishing the upstream connection, while proxy_read_timeout controls the interval between successive read operations from the upstream.

Those are different failure boundaries.

Why the application later logged 200

There was another interesting result.

The application continued sleeping after Gateway had already returned the 504.

Eventually it reached:

"GET /slow?seconds=5 HTTP/1.1" 200

and then:

layer=app client_connection_closed=true

That application log entry does not mean the original client received 200 OK.

Our Python handler logs the intended response status as it starts generating the response.

By that point, Gateway had already timed out and closed the upstream connection.

When Python later tried writing the body, the socket was gone and the handler caught BrokenPipeError.

The client had already received:

504 Gateway Time-out

This is a useful reminder that an application-side status log and a client-visible status are observations at different boundaries.

Failure 3: Same client-visible 504, different proxy

Now reverse the timeout relationship:

Application delay       5s
Gateway read timeout   10s
Edge read timeout       2s

Gateway is willing to wait long enough for the application.

Edge is not.

Gateway:

location = /edge-timeout {
    proxy_pass http://app:8000/slow?seconds=5;

    proxy_set_header X-Request-ID $trace_id;
    proxy_connect_timeout 2s;
    proxy_read_timeout 10s;
}

Edge:

location = /edge-timeout {
    proxy_pass http://gateway/edge-timeout;

    proxy_set_header X-Request-ID $trace_id;
    proxy_connect_timeout 2s;
    proxy_read_timeout 2s;
}

Run:

curl -sS -D - \
  -o /dev/null \
  -w '\nstatus=%{http_code}\nttfb=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  http://localhost:8080/edge-timeout

The client again saw:

HTTP/1.1 504 Gateway Time-out

But now the diagnostic header changed:

X-Debug-Generated-By: edge

The measured request:

status=504
connect=0.000274s
ttfb=2.005233s
total=2.005678s

From the client's point of view, this looked almost identical to the previous gateway timeout.

Internally, it was completely different.

The 499 that reveals the real sequence

This was the most useful part of the experiment.

The request ID was:

d9921f9d15f2df8ea396c7baf6c81d5a

I correlated it across the three services:

docker compose logs --no-color app gateway edge \
  | grep 'd9921f9d15f2df8ea396c7baf6c81d5a'

The application recorded:

layer=app
trace=d9921f9d15f2df8ea396c7baf6c81d5a
path=/slow
sleeping=5.0s

Edge recorded:

layer=edge
trace=d9921f9d15f2df8ea396c7baf6c81d5a
status=504
request_time=2.003
upstream_response_time=2.003
request="GET /edge-timeout HTTP/1.1"

Gateway recorded:

layer=gateway
trace=d9921f9d15f2df8ea396c7baf6c81d5a
status=499
upstream_status=-
request_time=2.003
upstream_response_time=2.003
request="GET /edge-timeout HTTP/1.1"

Edge's error log also said:

upstream timed out ... while reading response header from upstream

The sequence was:

                        app still processing
                               |
                               |
Client -> Edge -> Gateway -> Application
           |
           | 2 second timeout expires
           |
           +--> Edge returns 504 to Client
           |
           +--> Edge closes its request to Gateway

From Gateway's point of view, Edge is the client.

Gateway was willing to keep waiting for the application, but its client disconnected first.

NGINX therefore recorded 499.

NGINX defines 499 internally as NGX_HTTP_CLIENT_CLOSED_REQUEST. It is not a standard HTTP status sent by the external client; it is used by NGINX to represent the client closing the request before a response could be completed.

The application continued its five-second operation and later attempted to send its 200.

By then, the original request path was already gone.

So one request resulted in:

External client     504
Edge                504
Gateway             499
Application         eventually attempts 200

All four observations are valid at their respective boundaries.

Be careful interpreting `$upstream_status` alone

There was one more subtle result.

For the edge-generated timeout, Edge logged:

status=504
upstream_status=504

It would be tempting to conclude:

Gateway returned HTTP 504 to Edge.

But Gateway's own log for the same trace ID showed:

status=499
upstream_status=-

and Edge's error log explicitly said that Edge itself timed out while reading from its upstream.

So in this experiment, using Edge's $upstream_status alone to identify which process emitted the client-visible 504 would have produced the wrong conclusion.

Correlation across adjacent layers gave us the actual sequence.

This is one reason I prefer to combine:

local status
upstream status
request timing
upstream timing
error log
correlation ID

instead of treating any single field as the whole story.

The NGINX upstream module documentation describes the available upstream variables, including $upstream_status, $upstream_connect_time, $upstream_header_time, and $upstream_response_time.

A practical debugging sequence

When a request crosses several gateways or proxies, I would debug it in this order:

Propagate one correlation ID through every hop.
At every proxy, log the local response status and upstream information.
Record timing for different phases of the request. For NGINX, useful fields include:

   $request_time
   $upstream_connect_time
   $upstream_header_time
   $upstream_response_time

Read the error message carefully.

These are materially different:

   while connecting to upstream

and:

   while reading response header from upstream

Find the same request ID at the adjacent layer before changing timeout values.
Determine which component actually generated the client-visible error.
Only then investigate why the upstream was unavailable or slow.

A timeout setting is often where a problem becomes visible.

It is not necessarily where the underlying problem originated.

Results

The deterministic experiments produced:

Scenario	Client result	Response generated by	Observed time
Healthy request	`200`	Application response propagated	~10 ms
Reachable host, closed upstream port	`502`	Gateway	~2 ms
Slow application, Gateway waits 2s	`504`	Gateway	~2.005 s
Slow application, Gateway waits 10s, Edge waits 2s	`504`	Edge	~2.006 s

The additional stopped-container experiment produced another Gateway 504, but because the connection attempt timed out, not because Gateway timed out waiting for application response data.

So the model I would avoid is:

502 = backend down
504 = backend slow

A more useful set of questions is:

Which hop failed?

During which phase did it fail?

Which component generated the response?

What did the adjacent hop record for the same request?

Once those questions are answered, a multi-proxy 502 or 504 becomes much easier to localize.

References

Disclosure

AI assistance was used for technical research and editorial review while developing this article. I manually built and ran the lab, and the configurations, commands, response codes, timings, request traces, and log behavior described above were validated against the running environment before publication.

DEV Community: Praneeth Kodumagulla

Docker Swarm Said “Insufficient Resources.” The Running Service Was Using 480 KiB

The lab

Phase 1: reserve 60%, consume almost nothing

Phase 2: add another 50%

Phase 3: change one number

Why this isn't a Docker bug

The autoscaling trap

Reproduce it yourself

The debugging lesson

One question for people running orchestrators

References

How to Find Which Proxy Returned a 502 or 504 — A Reproducible NGINX Debugging Lab

What we're building

Project structure

Build a small backend that can be intentionally slow

Instrument the gateway

Configure the outer edge proxy

Run everything with Docker Compose

Establish a healthy baseline first

Failure 1: Immediate upstream connection refusal

A backend being "down" did not always produce a 502

Failure 2: Gateway connects successfully but times out reading the response

Why the application later logged 200

Failure 3: Same client-visible 504, different proxy

The 499 that reveals the real sequence

Be careful interpreting $upstream_status alone

A practical debugging sequence

Results

References

Disclosure

Be careful interpreting `$upstream_status` alone