DEV Community

Xiantao Cai
Xiantao Cai

Posted on

A container restarted 39,352 times. Every exit code was 0.

#docker #devops #debugging #node

A container on one of my servers had restarted 39,352 times. Every single exit code was 0.

That number is why nobody noticed. Monitoring watches for crashes, and this was not a crash. The process started, found nothing to do, exited successfully — and restart: unless-stopped dutifully brought it back. About 39,000 times.

This turned into one of those debugging sessions where every layer is behaving "correctly" and the system as a whole is broken. Here's the walk-through, and the one metric I now trust more than exit codes.

The successful infinite loop

The service was an MCP server running in stdio mode. In stdio mode the process talks over standard input/output — it reads requests from stdin and writes responses to stdout.

Here's the compose block, simplified:

services:
  mcp:
    image: my/mcp-server
    restart: unless-stopped
    # note what's NOT here:
    # stdin_open: true
    # tty: true
Enter fullscreen mode Exit fullscreen mode

Without stdin_open (the compose equivalent of docker run -i), the container gets no open standard input. So the MCP server boots, reads from stdin, and immediately hits EOF. There's nothing to read and never will be. It does the correct thing on EOF: it shuts down cleanly and exits 0.

restart: unless-stopped sees a stopped container and restarts it. The new process boots, reads stdin, hits EOF, exits 0. Restart. Boot. EOF. Exit 0. Restart.

Every individual decision in that loop is right. The process should exit when its input stream closes. The restart policy should revive a service that stopped without being told to. Compose it together and you get a clean, successful, infinite loop that a crash-based alert will never fire on.

The fix is one line — give it a stdin to hold open:

    stdin_open: true
    tty: true
Enter fullscreen mode Exit fullscreen mode

But the fix isn't the interesting part. The interesting part is why it stayed invisible for 39,352 iterations.

The health signal was inverted in both directions

While I was in there, I looked at the other containers on the same box. Two of them had been marked unhealthy for four weeks. Both were completely fine the whole time.

  • Health check #1 probed a port that nothing in the container listens on. The service was up; the check was pointed at the wrong door.
  • Health check #2 connected to localhost from a Node 18 process. Since Node 17, Node no longer reorders DNS results, so localhost resolves to IPv6 ::1 first. The service bound IPv4 only. Result: ECONNREFUSED, forever — against a service that was answering fine on 127.0.0.1. 
// Node 18+: this can resolve to ::1 and miss an IPv4-only server
const res = await fetch("http://localhost:3000/health");

// Force IPv4 if that's what your service binds:
const res = await fetch("http://127.0.0.1:3000/health");
Enter fullscreen mode Exit fullscreen mode

So on one machine, at the same time:

  • Two containers screaming unhealthy that were serving traffic perfectly.
  • One container reporting exit 0 that was broken and had been for tens of thousands of restarts.

The health signal wasn't just wrong. It was inverted in both directions at once. If I'd trusted it, I'd have "fixed" the two healthy ones and never looked at the broken one.

What I actually changed

  1. stdin_open: true + tty: true for the stdio service.
  2. Health check #1 repointed at a port the service actually listens on.
  3. Health check #2 switched to 127.0.0.1 (and I now treat "works in docker run, fails in a health check" as an IPv4/IPv6 smell).
  4. An alert on restart count velocity — restarts per hour — independent of exit code.

That last one is the takeaway.

Exit code 0 is not a success signal

Exit 0 only tells you the process agreed to leave. It does not tell you it should have.

Restart count, by contrast, is honest. A number that climbs into the thousands means something is wrong regardless of how politely each instance exited. It's a better smoke alarm than exit code, and a far better one than a health check that can be pointed at the wrong port or the wrong IP stack.

Cheap checks worth stealing:

  • Alert on restart count / restart velocity, not just non-zero exits.
  • A health check that has never once reported unhealthy is probably not testing anything.
  • When something works under docker run but a health check says it's down, suspect localhost → IPv6 before you suspect your code.
  • A stdio service with no stdin_open will exit cleanly on boot. "Clean" is not the same as "correct."

What's the longest a silent restart loop has run in your infrastructure before anyone noticed?

Top comments (0)