DEV Community: itops

Docker container keeps restarting? Find out why the loop won't stop

itops — Sun, 18 Oct 2026 02:42:41 +0000

You deployed to a VPS and docker ps keeps flipping between Restarting (1) and a few seconds of Up. The restart policy keeps bringing the process back, but it never fixes whatever kills it. Before you touch config, add memory or blame the database, find out why the process stops. Everything below is read-only; run it only on servers you're authorized to access, and replace YOUR_CONTAINER with the real name or ID. Docker access is root-equivalent, so don't expose the API or hand it to unknown helpers while debugging.

If the container exited once and stayed down, or you just want to decode a specific number, start with Docker exit codes: read the number before you restart anything. This post is about loops.

1. Check it's really a loop and write down the state

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
docker inspect --format 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} finished={{.State.FinishedAt}} restarts={{.RestartCount}}' YOUR_CONTAINER
docker events --since 30m --until "$(date +%s)" --filter container=YOUR_CONTAINER --filter event=die --filter event=start

Make sure it's the image and service you meant to deploy, and note the incident time and the last deploy. A restarts count that grows between runs, or die/start pairs repeating in the events, means a loop. FinishedAt is UTC.

The number in Restarting (N) is the last exit code. It points at the next check rather than giving the answer:

Code	Usually means
0	Command finished; a service may have daemonized into the background
1	Application error, read the logs
125 / 126 / 127	Docker couldn't run it / not executable / not found
137	SIGKILL, not proof of OOM
139 / 143	Segfault / stopped with SIGTERM

2. Restart policy and Docker's backoff

docker inspect --format 'policy={{.HostConfig.RestartPolicy.Name}} max_retries={{.HostConfig.RestartPolicy.MaximumRetryCount}}' YOUR_CONTAINER

Policy	Behaviour
`no` (default)	Never restarts
`on-failure[:N]`	Restarts only on non-zero exit, optionally at most N times, then stays `Exited`
`always`	Restarts whenever it stops; after a manual stop, only when the daemon or container is restarted
`unless-stopped`	Like `always`, but a container you stopped stays stopped across daemon restarts

Docker backs off between attempts: 100 ms, doubling each time, capped at one minute. If the container stays up for at least 10 seconds, the delay resets. That's why a crashing container spends longer and longer in Restarting and only flashes Up. In Compose this is the service's restart: key.

Changing the policy doesn't fix a broken startup; it only changes how often you see it fail. Also, don't let both Docker and systemd restart the same container.

3. Read a bounded window of logs

docker logs --since 15m --tail 200 --timestamps YOUR_CONTAINER

Look for the first error before each stop, not the last line. In a loop it repeats. Empty output doesn't mean healthy: the app may log to files, or the logging driver may not expose stdout.

Connection refused to the database: go to step 4. Inside a container, localhost is not the other service.
Permission denied / missing file: check mount paths, ownership, the runtime user and startup args. Don't chmod 777 your way out.
Config or migration failure: diff recent changes and work out rollback and data compatibility before editing production config.

4. Dependencies that aren't ready yet

Classic loop: the app starts before its database, cache or API is ready, fails its first connection, exits, and gets restarted until the dependency happens to be up.

docker compose ps
docker compose logs --since 15m --timestamps YOUR_DEPENDENCY

Plain depends_on only waits for the dependency's container to start, not to be ready. condition: service_healthy waits for its healthcheck, so it only helps if that service has one.
Check that the app's address matches the dependency's service name and port on the same network.
If the app gives up after one failed connection, a short retry in the app usually beats relying on restarts.

5. Healthchecks: what they do and don't restart

docker inspect --format '{{json .State.Health}}' YOUR_CONTAINER

On plain Docker Engine, unhealthy is only a label. Docker does not restart a container because its healthcheck fails. If unhealthy containers keep getting restarted, something else is doing it (Swarm, an orchestrator, an autoheal-style helper), and that tool's logs are where to look.

A check that's too strict, or has no start period for a slow app, can keep dependents waiting on service_healthy or keep triggering that external tool. Read the check's own output before blaming the app.

6. Memory, disk and ports

docker inspect --format 'memory_limit_bytes={{.HostConfig.Memory}}' YOUR_CONTAINER
docker stats --no-stream YOUR_CONTAINER
df -h

Memory: OOMKilled=true is evidence Docker recorded an OOM kill; a 137 alone isn't. docker stats is a snapshot and won't show the earlier peak.
Disk: a full filesystem breaks databases and apps at startup. Find what filled it before deleting anything.
Ports: "port is already allocated" means something else holds it; sudo ss -ltnp shows what.

7. Turn evidence into a next step

Write down the symptom, the evidence, the likely cause and the smallest next check. For production, define impact, a maintenance window and rollback first. Deleting data or containers and widening permissions are not universal fixes.

Illustrative example

Illustrative, not a real incident: after a host reboot the app shows Restarting (1) and its restart count keeps climbing while the database is Up. The app's first log line after every start is a refused connection to the database, and the database log shows it was still initializing at those times. Compose uses depends_on without a health condition. That supports "dependency not ready". The fixes, a database healthcheck plus condition: service_healthy, or a connection retry in the app, are changes to plan, followed by checking that the restart count stops growing.

A restart policy hides the failure; the exit code, events and the first log error tell you what it is.

I work on OpsMate, which puts an SSH terminal and an AI assistant on one page, so you can run the commands above yourself or ask the AI to compare the logs, restart count, policy, health and dependency status for you. Low-risk actions such as service restarts and log rotation run automatically by default, dangerous commands are blocked, and every step is logged. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first. You get 500 free AI requests per month. Read the full guide and start free on itops.sh.

Drafted with AI assistance and reviewed before publishing.

Exit code 137: was it really out of memory? Find who sent the SIGKILL

itops — Sun, 11 Oct 2026 18:40:39 +0000

docker ps -a says Exited (137) or Restarting (137), the app's logs stop mid-line with no error, and the box is a small 1–2 GB VPS running several services. It looks like "out of memory", and often it is. But 137 only tells you the main process received SIGKILL. The real question is who sent it. Everything below is read-only; run it only on servers you're authorized to access, and replace YOUR_CONTAINER with the real name or ID. Kernel log checks need sudo.

Why 137, and why the logs are silent

137 is 128 + 9: the process was ended by signal 9. SIGKILL can't be caught or delayed, so the app never gets to log a goodbye. A log that just stops fits a 137. It doesn't clear the app.

Several senders produce the same number:

Who sent SIGKILL	Where the evidence usually is	`OOMKilled`
Kernel OOM killer, container's own memory limit reached	Kernel: `Memory cgroup out of memory: Killed process …`; an `oom` event in `docker events`	Usually `true`
Kernel OOM killer, whole host out of memory	Kernel: `Out of memory: Killed process …`	`true` or `false`, depending on cgroup version and Docker/containerd versions
Userspace OOM daemon (systemd-oomd, earlyoom)	That daemon's journal; nothing from the kernel	`false`
`docker kill` / `docker rm -f`	`kill` event with `signal=9`	`false`
`docker stop`, `compose down` or a redeploy, app ignored SIGTERM for the whole grace period (10 s default)	`kill` `signal=15`, then `signal=9` about 10 s later	`false`
A script, agent or person running `kill -9` on the host	Only a `die` event; maybe that tool's own logs	`false`

Memory is the first suspect, not the answer.

1. Capture the exit state before a restart wipes it

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
docker inspect --format 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} started={{.State.StartedAt}} finished={{.State.FinishedAt}} restarts={{.RestartCount}} policy={{.HostConfig.RestartPolicy.Name}}' YOUR_CONTAINER

A restart resets the evidence. When the container starts again, Docker sets ExitCode back to 0 and OOMKilled back to false. With a restart policy you'll often see status=running exit=0 oom=false restarts=5, which looks clean but isn't. The previous exit then lives only in docker events (step 2) and the kernel log (step 4).
exited / 137 / oom=true: the container's own limit is the lead hypothesis. Still confirm with steps 3 and 4.
exited / 137 / oom=false: memory isn't ruled out, but check Docker's events first.
oom=true on a running container, or with another exit code: in recent Docker versions the flag is set when the kernel kills any process in the container. A killed worker can leave the container up, or the main process may exit later with its own code.
Memory trouble doesn't always end in 137. If a runtime hits its own heap limit first, it logs an error and exits another way: Node.js prints JavaScript heap out of memory and often exits 134; a JVM logs java.lang.OutOfMemoryError.
finished is UTC (ends in Z). Convert before comparing with local host logs.

2. Ask Docker whether it sent the signal

docker events --since 24h --until "$(date +%s)" --filter container=YOUR_CONTAINER --filter event=oom --filter event=kill --filter event=die --filter event=start
docker inspect --format 'stop_signal={{.Config.StopSignal}} stop_timeout={{.Config.StopTimeout}}' YOUR_CONTAINER

Repeated event= filters are OR'd. Patterns to look for:

oom → die exitCode=137: kernel killed a process under the container's limit, main process died.
oom with no die: something inside was killed, main process survived. Look for a dead worker in the app logs.
kill signal=15 → ~10 s → kill signal=9 → die exitCode=137: a stop that ran out of grace period. Not memory.
kill signal=9 → die: someone ran docker kill or docker rm -f.
die exitCode=137 with no kill before it: the signal came from outside Docker, usually the kernel. Go to step 4.
A start after every die is the restart policy. Newer Docker adds execDuration to die; if the exits are memory kills, similar values each time suggest something that grows until it's killed.

Docker keeps only recent events in memory (the last 256), and they're gone after a daemon restart, so check soon.

For the stop pattern: empty stop_signal and stop_timeout=<nil> mean the defaults (SIGTERM, 10 s). If PID 1 is a shell wrapper that doesn't forward SIGTERM, or the app has no SIGTERM handler, every stop ends in SIGKILL.

3. Check the limit and who is using memory

docker inspect --format 'memory_limit_bytes={{.HostConfig.Memory}} memory_swap_bytes={{.HostConfig.MemorySwap}}' YOUR_CONTAINER
docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
free -h
ps -eo pid,user,rss,comm --sort=-rss | head -n 10

memory_limit_bytes=0: no limit, so the container can eat host memory until the whole host runs short. A host-wide OOM is then the likelier path.
memory_swap_bytes is memory + swap. Equal to the limit means no swap for this container; -1 means unlimited.
docker stats only covers running containers. Without a limit, the limit column shows host RAM. Several unlimited containers on a 2 GB host are fighting over the same memory.
In free -h, read available, not free, and check whether there's any swap.
In a host-wide OOM the kernel picks its victim mostly by size and oom_score_adj, so the process that got killed isn't always the one that caused the pressure. A neighbour can grow and get your container killed.

These are snapshots of now, not of the kill. To tell a spike from steady growth, sample after the container comes back:

for i in 1 2 3 4 5 6 7 8 9 10; do date '+%H:%M:%S'; docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' YOUR_CONTAINER; sleep 60; done

Steady climb that never levels off → leak, or an unbounded cache or queue. Jumps during specific requests or jobs → peak workload. Ten minutes is a hint; a real trend needs hours or days of monitoring.

4. Read the kernel log, and know when OOMKilled lies

sudo dmesg -T | grep -i -E 'oom-kill|out of memory|killed process' | tail -n 20
sudo journalctl -k --since "2026-09-29 14:00" --until "2026-09-29 14:30" --no-pager | grep -i -E 'oom-kill|out of memory|killed process'
docker inspect --format '{{.Id}}' YOUR_CONTAINER

Pick a window around finished (converted to local time) or a die time. journalctl timestamps are usually more reliable than dmesg -T's converted ones.
dmesg only covers the current boot. After a reboot, sudo journalctl -k -b -1 shows the previous one, if the journal is persistent.
Memory cgroup out of memory: Killed process 2314 (node) → a cgroup limit, normally the container's. Out of memory: Killed process … without "Memory cgroup" → the whole host.
Newer kernels add an oom-kill: line first: constraint=CONSTRAINT_MEMCG for a cgroup limit, CONSTRAINT_NONE plus global_oom for host-wide. task_memcg= usually contains the container ID (e.g. /system.slice/docker-<id>.scope); search for its first 12 characters.
The PID is the host PID. anon-rss is roughly what the process held when killed.

No matching kernel line? Check for a userspace OOM daemon, which kills without the kernel logging an OOM:

systemctl is-active systemd-oomd earlyoom
sudo journalctl -u systemd-oomd -u earlyoom --since "2026-09-29 14:00" --no-pager

Memory was the cause but OOMKilled is false. How Docker learns about an OOM kill depends on the kernel, cgroup version and containerd/Docker versions. Things to check, not rules:

The container restarted before you inspected it.
A userspace daemon did the killing, not the kernel.
The whole host ran out. On cgroup v1 the event Docker listens for is tied to the container's own limit, so host-wide kills often leave the flag false. On cgroup v2 the kernel counts OOM kills per cgroup, so the flag is more likely set, though older containerd/Docker versions had gaps.
On Docker Desktop, memory runs out inside Docker's VM, invisible to Mac/Windows tools.

docker info --format 'cgroup={{.CgroupVersion}} driver={{.CgroupDriver}}'
docker exec YOUR_CONTAINER cat /sys/fs/cgroup/memory.events   # cgroup v2, if the image has cat

A nonzero oom_kill means the kernel killed a process in this container since it last started.

5. Pick a next step that matches the evidence

Each of these is a change to plan, with a way back:

Container limit hit, workload really needs more. Raise mem_limit / deploy.resources.limits.memory / --memory, after checking the host has room for all limits plus the OS. docker update --memory changes a running container, but the next Compose recreate reverts to the file.
No limit, host ran out. Limits keep one container's growth inside that container, so it dies instead of the database next door. If the services together need more than the host has, it's a bigger instance or fewer services.
Memory climbs after every start. Looks like a leak or an unbounded cache/queue; a higher limit only delays the kill. Compare with the last deploy and use the runtime's profilers. Recycling workers is a stopgap.
Runtime heap settings. Keep the runtime's own limit below the container's so failure shows up as a logged error, not a silent 137. JVM (JDK 10+, 8u191+) reads the container limit and typically caps heap at a quarter of it; set -XX:MaxRAMPercentage or -Xmx, leaving headroom for non-heap memory. Node.js: --max-old-space-size=<MB>. Multi-worker servers multiply per-worker memory.
Swap, with caveats. Absorbs short peaks, but under sustained pressure everything slows instead of failing fast, and leaks hide longer.
It was a stop, not memory. Handle SIGTERM, use exec-form CMD/ENTRYPOINT or exec in entrypoint scripts so the app is PID 1, or add --init / init: true. Raise the grace period only if clean shutdown genuinely takes longer.
Something else on the host. Find the script, agent or person from the event timestamp. Memory settings won't fix this.

Before touching production, write down the evidence, the change, how you'll verify it and how you'll roll it back.

Illustrative example

Illustrative example (not a real case): a 2 GB VPS runs api (Node.js), postgres and nginx with Compose. The API drops out for a few seconds several times a day. docker ps -a shows api as Up 12 minutes; inspect says status=running exit=0 oom=false restarts=5, clean only because the last start reset it. docker events --since 24h shows five die exitCode=137, each followed by start, no kill before any of them, so not docker kill or docker stop. memory_limit_bytes=0. One die is at 06:05:41 UTC, 14:05:41 host time (UTC+8). journalctl -k at that minute has an oom-kill: line with global_oom and a task_memcg path containing the api container ID, then Out of memory: Killed process 2314 (node) … anon-rss:1398212kB. Ten docker stats samples after restart show api rising about 15 MB a minute while postgres stays flat.

Together: the host ran out, the kernel picked the Node process in api, and api grows steadily after each start, which fits a leak or unbounded cache better than a one-off peak. Not proven: which code path grows. Next steps are a person's call: give api a limit, set the Node heap below it so failures get logged, and hunt the growth starting from the last deploy.

137 is a clue and OOMKilled is evidence, not proof. Conclude from the exit state, Docker events and kernel lines with matching timestamps.

I work on OpsMate, which puts an SSH terminal and an AI assistant on one page, so you can run the commands above yourself or describe the exit in plain language and check against the raw output. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first. Every action is sorted into one of three risk tiers. Low-risk actions (such as a confident service restart or log rotation) run automatically. Medium-risk actions (such as changing config or recreating a container) are sent to you for approval first, and run automatically if no one responds within 30 minutes. High-risk actions (database schema, kernel parameter, network and firewall changes) only raise an alert and are never executed. When the system isn't sure, it treats the action as the next tier up; dangerous commands are always blocked. The full guide is on itops.sh.

Drafted with AI assistance and reviewed before publishing.

Docker exit codes: read the number before you restart anything

itops — Sun, 11 Oct 2026 07:36:08 +0000

CI says the deploy succeeded, then docker ps -a shows Exited (1) or Exited (127). Don't restart yet: get the exit code, check what it usually means (and does not prove), then read the first log error. Read-only; authorized servers only. Replace YOUR_CONTAINER with the real name or ID.

1. Get the exit code

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}' YOUR_CONTAINER

With Compose, docker compose ps -a includes stopped ones; name is usually <project>-<service>-1.

Exited (1) 8 seconds ago is a single exit with code 1. Restarting (1) 3 seconds ago means a restart policy is looping; the bracketed number is from the last exit.

Inspect prints exit code, OOMKilled (true/false), and State.Error. Keep them apart. State.Error is set when Docker couldn't start the process; empty for most app crashes. Created means it never ran — usually a run-command or image problem.

docker inspect --format 'started={{.State.StartedAt}} finished={{.State.FinishedAt}} restarts={{.RestartCount}} policy={{.HostConfig.RestartPolicy.Name}}' YOUR_CONTAINER

A few seconds between started and finished is the "exits right after deploy" pattern. Timestamps are UTC (Z); convert before comparing with local logs.

2. Exit codes at a glance

Code	Usual meaning
0	Main process finished successfully
1	Generic application error
2	Shell syntax error, wrong usage, or an app-defined error
125	Docker itself failed to run the container
126	Command found but not executable
127	Command not found
132	Illegal instruction, SIGILL (128 + 4)
137	Killed with SIGKILL (128 + 9)
139	Segmentation fault, SIGSEGV (128 + 11)
143	Stopped with SIGTERM (128 + 15)
255	Exit status −1, a failed `ssh`, or Docker lost the process

Above 128 usually means "killed by signal (code − 128)". kill -l 137 prints KILL.

3. What each code means

0 — finished successfully

Expected for a migration or backup. A web service exiting 0 usually backgrounded itself or ran a one-shot. Check entrypoint/cmd:

docker inspect --format 'entrypoint={{json .Config.Entrypoint}} cmd={{json .Config.Cmd}}' YOUR_CONTAINER

0 only means the main process ended cleanly.

1 — generic application error

Most runtimes exit 1 for any uncaught exception or failed startup check. Read the logs (section 4) for the first error, then compare with this deploy's changes (image, env, config, migrations). The number alone does not name the error.

2 — syntax, usage, or app-defined error

No Docker-specific meaning — whatever the main process returned. Common sources:

Shell script that doesn't parse. Bash/dash exit 2 on syntax errors. An entrypoint with Windows (CRLF) endings often shows syntax error: unexpected end of file (bash) or Syntax error: end of file unexpected (dash). Outside if/case, CRLF can instead yield not found / exit 127.
Wrong builtin usage. Bash builtins return 2 for an invalid option or missing argument (e.g. source with no filename).
Program rejecting arguments. Python argparse → unrecognized arguments; Go flag → flag provided but not defined. Compose command: typos show up here.
Missing file. python3 /app/main.py exits 2 with can't open file ... No such file or directory if the path is wrong.
Unrecovered Go panic. Prints panic: plus a stack trace, then exits 2.

docker logs --timestamps YOUR_CONTAINER 2>&1 | head -n 30
bash -n entrypoint.sh          # parse only
grep -c $'\r' entrypoint.sh    # 0 = LF line endings

CRLF fix: sed -i 's/\r$//' entrypoint.sh, add *.sh text eol=lf to .gitattributes, rebuild. The log line decides the cause.

125 — Docker could not run the container

docker run or the daemon rejected the request; the process usually never started. Read the CI/deploy error and State.Error. Typical: unknown flag, pull denied, port already allocated (sudo ss -ltnp). Not an app bug. When Docker can't start it, status is often Created; when a shell inside hits 126/127, look in docker logs.

126 — found but not executable

Check the command and permissions inside the image. A bind-mounted script keeps the host execute bit. Compare docker image inspect --format '{{.Os}}/{{.Architecture}}' YOUR_IMAGE with uname -m. Mismatch → usually exec format error. 126 = can't execute; 127 = can't find.

127 — command not found

The program never started. Confirm the binary exists in this image and on PATH. Slimmer bases and Compose command: typos are common. Does not prove a crash after startup.

132 — illegal instruction (SIGILL)

132 is 128 + 4 (kill -l 132 → ILL). The CPU hit an instruction it couldn't run:

Binary needs features this host lacks (AVX/AVX2/AVX-512, x86-64-v3); MongoDB 5.0+ on x86_64 documents AVX.
VM hides features behind a basic hypervisor CPU model.
QEMU emulating another arch hits unimplemented instructions.
Deliberate trap (ud2 / __builtin_trap()) — a bug, not a CPU mismatch.

docker logs --timestamps YOUR_CONTAINER 2>&1 | tail -n 20
sudo dmesg -T | grep -i 'invalid opcode' | tail -n 10
grep -m1 '^flags' /proc/cpuinfo | grep -o -w -E 'avx|avx2|avx512f' | sort -u

A restart policy won't help. Rule out the trap case before buying a new server.

137 — SIGKILL

137 is 128 + 9. Check OOMKilled: true → memory lead; false → docker kill, timed-out docker stop, or another host process. 137 alone is not proof of OOM. Check kernel log, memory limits, and docker events before raising limits.

139 — SIGSEGV

Invalid memory access; kernel sent signal 11. Read logs around the exit, then sudo dmesg -T | grep -i segfault | tail -n 20. Note native-dependency or runtime upgrades. Kernel lines only count if their time matches finished.

143 — SIGTERM

A normal stop. Find who asked: redeploy, docker stop, Compose recreate, host shutdown, orchestrator. Docker keeps only recent events (last 256):

docker events --since 1h --until "$(date +%s)" --filter container=YOUR_CONTAINER

Not a crash — who sent SIGTERM, and did the replacement come up?

255 — exit(−1), failed ssh, or Docker lost the process

Program returned −1. Only the lowest 8 bits reach the parent, so exit(-1) shows as 255 (bash/Python/Node/Go).
Failed ssh. ssh exits 255 when ssh itself fails (refused, auth, host key), not when the remote command fails. Backup/deploy jobs that end in ssh exit 255 on connection failure.
Docker lost the process. After a host crash, hard reboot or daemon crash, dockerd finds a container recorded as running with no saved exit status and marks it Exited (255) — Docker's bookkeeping, not the app's verdict.

docker inspect --format 'exit={{.State.ExitCode}} started={{.State.StartedAt}} finished={{.State.FinishedAt}} policy={{.HostConfig.RestartPolicy.Name}}' YOUR_CONTAINER
docker logs --timestamps YOUR_CONTAINER 2>&1 | tail -n 30
uptime -s
sudo journalctl -u docker.service --since "24 hours ago" --no-pager | grep -i -E 'starting up|completed initialization'

ssh: errors in the last lines → fix key / known_hosts / network. finished matching boot (uptime -s) or a dockerd Starting up line, with a clean log stop → why did the host/daemon restart? finished is UTC; convert first.

4. Find the first error in the logs

docker logs --since 30m --timestamps YOUR_CONTAINER 2>&1 | head -n 80
docker logs --since 30m --timestamps YOUR_CONTAINER 2>&1 | grep -n -i -E 'error|fatal|panic|exception|denied|not found|refused' | head -n 20

Works on a stopped container until removed (--rm takes logs with it). Use 2>&1 — most errors are on stderr. Read the first error after deploy, not only the last line.

Common first lines (hypotheses until checked):

connection refused: dependency down or wrong address; inside a container, localhost is itself.
permission denied: mount ownership or runtime user.
no such file or directory for a script that exists: often CRLF on the shebang, or a missing interpreter.
exec format error: wrong arch (see 126).
Missing env/config key: compare key names with the last good deploy; don't print secrets.

Redact passwords, tokens, connection strings and customer data before pasting logs.

Illustrative example

Illustrative example (not a real case): after a slimmer base image, docker ps -a shows Restarting (127) 4 seconds ago. Inspect: 127 false, empty error — process started, not OOM. started/finished ~1 s apart. First log line: /app/start.sh: 5: exec: gunicorn: not found. Entrypoint runs; new base lacks the app server. Restart policy or more memory won't help. Next: roll back the image tag, or rebuild with the dependency, then watch whether it stays up.

An exit code is a clue, not a diagnosis — combine the code, log lines and matching timestamps.

Drafted with AI assistance and reviewed before publishing.

Nginx 502 Bad Gateway: read one log line before you touch anything

itops — Fri, 09 Oct 2026 07:30:27 +0000

When a site starts returning 502 Bad Gateway, the instinct is to restart nginx, bump a timeout, or reboot the box. Most of the time none of that is needed, because nginx is the one part that's working. A 502 means nginx took the request, passed it to the upstream (your Node app, Gunicorn, PHP-FPM, a container), and got nothing usable back.

The fastest way out is to read the one log line nginx wrote at the moment of the failure. This post is the short version of how I do that on Ubuntu 22.04/24.04. Everything until the last section only reads state, and all of it is for servers you're authorized to access.

1. Make sure the 502 is really from your nginx

If you sit behind a CDN, its 502 page looks a lot like nginx's. Check from your laptop, then from the server itself:

curl -sS -o /dev/null -D - https://example.com/ | grep -i -E '^(HTTP|server|cf-ray)'
curl -sS -o /dev/null -w '%{http_code}\n' --resolve example.com:443:127.0.0.1 https://example.com/

A server: cloudflare or cf-ray header means the CDN answered. If the second command (run on the server, straight to local nginx) returns 200, the problem is between the CDN and your server, not behind nginx. If it's also 502, keep going.

Then confirm nginx itself is healthy:

systemctl status nginx --no-pager
sudo nginx -t

active (running) plus test is successful means stop suspecting nginx.

2. Find the line that matches the failing request

sudo tail -n 50 /var/log/nginx/error.log
sudo grep -E 'upstream|connect\(\)' /var/log/nginx/error.log | tail -n 20

If a server block has its own log, sudo nginx -T 2>/dev/null | grep -n -E 'error_log|proxy_pass|fastcgi_pass|upstream' shows where it goes and where each location proxies. (Don't paste nginx -T output anywhere public; it can contain credentials.)

A typical line:

[error] connect() failed (111: Connection refused) while connecting to upstream, client: 203.0.113.10, server: example.com, request: "GET /api/orders HTTP/1.1", upstream: "http://127.0.0.1:3000/api/orders"

The upstream: field is the exact address nginx tried. Comparing it with where the app actually listens solves a surprising number of 502s on the spot.

3. Map the message to a cause

error.log says	What it usually means
`(111: Connection refused)`	Nothing listening at that address: app stopped, crashed, still starting, or on another port
`unix:...sock failed (2: No such file or directory)`	Socket file doesn't exist: service stopped, or nginx points at an old path (classic after a PHP upgrade)
`unix:...sock failed (13: Permission denied)`	`www-data` can't open the socket
`unix:...sock failed (11: Resource temporarily unavailable)`	All workers busy, socket backlog overflowed
`upstream prematurely closed connection`	App accepted the request then died: crash, killed worker, app-server timeout
`(104: Connection reset by peer)`	Process restarted or worker killed, or a reused keepalive connection was already closed
`upstream sent too big header`	Response headers (often a growing `Set-Cookie`) don't fit the buffer
`SSL_do_handshake() failed ... wrong version number`	`proxy_pass https://` pointed at a plain-HTTP port
`no live upstreams`	Every server in the `upstream {}` group was marked failed earlier; scroll up for why
`upstream timed out (110: ...)`	That's a 504, not a 502: the app is slow, not gone

Two things that look related but aren't 502s: host not found in upstream is logged at [emerg] when nginx loads config, so nginx won't start or reload at all. And (13: Permission denied) on a TCP address is the SELinux case on RHEL-family systems; stock Ubuntu nginx isn't confined by AppArmor, so there it's a plain file-permission problem.

If you saw "The proxy server received an invalid response from an upstream server", that page is Apache mod_proxy, not nginx.

4. Confirm it on the upstream side

Is the service up, and what did it log?

systemctl status YOUR_APP --no-pager
sudo journalctl -u YOUR_APP --since "30 min ago" --no-pager | tail -n 50
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

Is something listening where nginx connects?

sudo ss -ltnp
sudo ss -lxp | grep -E 'php|gunicorn|uwsgi|\.sock'
ls -l /run/php/

Three mismatches I see a lot:

App moved to 3001 after a deploy, nginx still sends to 3000.
proxy_pass http://localhost:3000 while the app only listens on 127.0.0.1. nginx may resolve localhost to both 127.0.0.1 and [::1], and the IPv6 attempt is refused; the log shows upstream: "http://[::1]:3000/...". Writing 127.0.0.1 removes the ambiguity.
php8.1-fpm.sock in the site config, but only php8.3-fpm.sock exists after an Ubuntu 22.04 to 24.04 upgrade.

Does the app answer if you skip nginx?

curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --max-time 10 -H 'Host: example.com' http://127.0.0.1:3000/

Refused here too means the app isn't serving. 200 here but 502 through nginx means look at the address mismatch, socket permissions, or the response headers. (PHP-FPM speaks FastCGI, so curl can't test it directly.)

Was a worker killed? For prematurely closed or reset by peer, check the same minute:

sudo journalctl -k --since "1 hour ago" --no-pager | grep -i -E 'out of memory|killed process'
sudo journalctl -u YOUR_APP --since "1 hour ago" --no-pager | grep -i -E 'worker timeout|killed|signal'

Gunicorn's [CRITICAL] WORKER TIMEOUT means Gunicorn killed the worker after its own --timeout (30 s default). Raising nginx timeouts won't help, because the app server cut the request first.

The Docker gotcha

Inside an nginx container, proxy_pass http://127.0.0.1:3000; points at the nginx container itself. Proxy to the Compose service name (http://app:3000), make the app listen on 0.0.0.0 inside its container, or, with nginx on the host, publish the port to loopback (127.0.0.1:3000:3000). Also remember nginx resolves hostnames at config load: if the app container is recreated with a new IP, nginx can keep using the old one until reload.

When the log points at buffers or keepalive

upstream sent too big header needs bigger buffers, and you have to raise them together:

location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_buffer_size 16k;
    proxy_buffers 8 16k;
}

Raising only proxy_buffer_size makes nginx -t fail on proxy_busy_buffers_size. For PHP-FPM the equivalents are fastcgi_buffer_size / fastcgi_buffers.

Occasional 502s with an upstream {} block that uses keepalive: if the app closes idle connections sooner than nginx expects, a request can land on a just-closed connection. Node.js closes idle keep-alive connections after 5 s by default (server.keepAliveTimeout), so make the app's idle timeout longer than nginx's upstream keepalive_timeout (60 s default), or lower the latter.

Changing things safely

Only now do we change anything, and only the one thing the log pointed at:

Back up the file (sudo cp /etc/nginx/sites-available/example.com /root/example.com.bak).
Make the single change.
sudo nginx -t, then sudo systemctl reload nginx (reload keeps connections; restart drops them).
Repeat the failing request with sudo tail -f /var/log/nginx/error.log open.
No better? Restore the backup, test, reload.

Don't raise every timeout "just in case", and don't chmod 0666 a socket. Fix what the log names.

I work on OpsMate, which puts an SSH terminal and an AI assistant on one page, so you can run the commands above yourself or describe the 502 in plain language and check the answer against the raw output. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first. Every action is sorted into one of three risk tiers. Low-risk actions (such as a confident service restart or log rotation) run automatically. Medium-risk actions (such as changing config or recreating a container) are sent to you for approval first, and run automatically if no one responds within 30 minutes. High-risk actions (database schema, kernel parameter, network and firewall changes) only raise an alert and are never executed. When the system isn't sure, it treats the action as the next tier up; dangerous commands are always blocked. The full guide, with every error message and the PHP-FPM and socket-permission details, is on itops.sh.

Drafted with AI assistance and reviewed before publishing.