A container restarted 39,352 times. Every exit code was 0.

A container on one of my servers had restarted 39,352 times. Every single exit code was 0.

That number is why nobody noticed. Monitoring watches for crashes, and this was not a crash. The process started, found nothing to do, exited successfully — and restart: unless-stopped dutifully brought it back. About 39,000 times.

This turned into one of those debugging sessions where every layer is behaving "correctly" and the system as a whole is broken. Here's the walk-through, and the one metric I now trust more than exit codes.

The successful infinite loop

The service was an MCP server running in stdio mode. In stdio mode the process talks over standard input/output — it reads requests from stdin and writes responses to stdout.

Here's the compose block, simplified:

services:
  mcp:
    image: my/mcp-server
    restart: unless-stopped
    # note what's NOT here:
    # stdin_open: true
    # tty: true

Without stdin_open (the compose equivalent of docker run -i), the container gets no open standard input. So the MCP server boots, reads from stdin, and immediately hits EOF. There's nothing to read and never will be. It does the correct thing on EOF: it shuts down cleanly and exits 0.

restart: unless-stopped sees a stopped container and restarts it. The new process boots, reads stdin, hits EOF, exits 0. Restart. Boot. EOF. Exit 0. Restart.

Every individual decision in that loop is right. The process should exit when its input stream closes. The restart policy should revive a service that stopped without being told to. Compose it together and you get a clean, successful, infinite loop that a crash-based alert will never fire on.

The fix is one line — give it a stdin to hold open:

    stdin_open: true
    tty: true

But the fix isn't the interesting part. The interesting part is why it stayed invisible for 39,352 iterations.

The health signal was inverted in both directions

While I was in there, I looked at the other containers on the same box. Two of them had been marked unhealthy for four weeks. Both were completely fine the whole time.

• Health check #1 probed a port that nothing in the container listens on. The service was up; the check was pointed at the wrong door.
• Health check #2 connected to localhost from a Node 18 process. Since Node 17, Node no longer reorders DNS results, so localhost resolves to IPv6 ::1 first. The service bound IPv4 only. Result: ECONNREFUSED, forever — against a service that was answering fine on 127.0.0.1.

// Node 18+: this can resolve to ::1 and miss an IPv4-only server
const res = await fetch("http://localhost:3000/health");

// Force IPv4 if that's what your service binds:
const res = await fetch("http://127.0.0.1:3000/health");

So on one machine, at the same time:

• Two containers screaming unhealthy that were serving traffic perfectly.
• One container reporting exit 0 that was broken and had been for tens of thousands of restarts.

The health signal wasn't just wrong. It was inverted in both directions at once. If I'd trusted it, I'd have "fixed" the two healthy ones and never looked at the broken one.

What I actually changed

1. stdin_open: true + tty: true for the stdio service.
2. Health check #1 repointed at a port the service actually listens on.
3. Health check #2 switched to 127.0.0.1 (and I now treat "works in docker run, fails in a health check" as an IPv4/IPv6 smell).
4. An alert on restart count velocity — restarts per hour — independent of exit code.

That last one is the takeaway.

Exit code 0 is not a success signal

Exit 0 only tells you the process agreed to leave. It does not tell you it should have.

Restart count, by contrast, is honest. A number that climbs into the thousands means something is wrong regardless of how politely each instance exited. It's a better smoke alarm than exit code, and a far better one than a health check that can be pointed at the wrong port or the wrong IP stack.

Cheap checks worth stealing:

• Alert on restart count / restart velocity, not just non-zero exits.
• A health check that has never once reported unhealthy is probably not testing anything.
• When something works under docker run but a health check says it's down, suspect localhost → IPv6 before you suspect your code.
• A stdio service with no stdin_open will exit cleanly on boot. "Clean" is not the same as "correct."

What's the longest a silent restart loop has run in your infrastructure before anyone noticed?

Comments

[Raknaos]

The double inversion is the most instructive part: exit 0 makes the restart counter look like a crash metric that never fires, and a badge stuck red for four weeks trains everyone to ignore the badge. The Node 18 detail is real too — I've watched a health check pass with curl and fail inside a fetch for exactly the ::1-vs-IPv4 reason you describe.

Did you end up alerting on restart rate itself rather than exit code? A container that exits cleanly in under a few seconds repeatedly is definitionally not doing its job, and it's the one signal most restart: unless-stopped setups never emit. Curious whether the audit turned up other silent loops that had been spinning since day one.

[Xiantao Cai]

Yes — the alert now fires on restart rate (restarts/hour), not exit code, precisely because exit 0 keeps the crash-based alerts silent. The refinement I'd add: pair restart-rate with process lifetime. A container that exits 0 but whose median lifetime is a few seconds is, as you say, definitionally not doing its job — "short-lived + restarting" catches the clean-exit loop that neither a crash alert nor a health check will.

On the audit: the two mis-pointed health checks were the other real findings — a port nothing listened on, and the ::1-vs-IPv4 one you also hit. The bigger takeaway wasn't "how many loops" but that I'd been auditing services when the thing actually lying to me was the signals. And glad the fetch/::1 case wasn't just me — it's nasty because it looks like your code, not your DNS.

[Raju Dandigam]

“Exit 0 only tells you the process agreed to leave” is the operational distinction. I’d define a domain-ready milestone and minimum useful uptime, then classify any clean exit before readiness as a failed lifecycle even when the process code is zero; restart velocity becomes the symptom, not the only signal. For a stdio MCP server, could “transport opened and first request handled” serve as that readiness contract?

[Xiantao Cai]

That readiness-contract framing is sharper than what I had. For a stdio MCP server "transport opened and first request handled" mostly works, with one caveat: an idle server legitimately sits waiting on stdin, so "first request handled" can be too strict — you'd flag a healthy-but-quiet process. What actually failed here was upstream of readiness: stdin was EOF at boot, so the transport could never open.

So I'd split it — liveness = "stdin attached and not at EOF" (the exact condition that was false here), readiness = "first JSON-RPC message answered", and keep restart-velocity as the backstop for whatever the contract doesn't anticipate. "Clean exit before readiness = failed lifecycle even at code 0" is a great way to put it — that's exactly the classification the exit code refuses to give you.