The Socket Was Open. The Site Was Dead for Fourteen Hours.

A site had been down since 19:54 the previous evening. Nobody reported it. The monitoring did not fire. When I finally looked, the cause was one process that had outlived its parent and was holding a socket nobody could use.

Every individual check passed. DNS resolved. Port 443 accepted connections. The TLS handshake completed against a valid certificate with two months left on it. The web server was running and serving other sites on the same machine. And every request to that one site returned 503.

The failure lived in the gap between "the process exists" and "the process works," which is a gap most health checks do not cover.

What the checks reported

I started from the outside, because the outside is cheap to measure:

Layer	Check	Result
DNS	resolves to the expected address	fine
TCP	ports 80 and 443 accept	fine
TLS	handshake, chain, expiry	fine, 68 days left
HTTP (port 80)	GET /	302 to the HTTPS URL
HTTPS	GET /	first a 20-second hang, then 503
Web server	running, other sites on the box served fine	fine

Two of those rows are the reason nobody noticed.

The port-80 row returns 302 because the HTTP-to-HTTPS redirect is a server-level rewrite. It never reaches PHP. A monitor watching http:// sees a chain that starts with a perfectly healthy 302, and if it treats a redirect as up, it reports up. Forever. The application behind it can be completely dead.

The TLS row is worse, because it looks like the most meaningful check in the list. A successful handshake proves the listener is alive and the certificate is valid. It proves nothing whatsoever about whether anything can generate a response body.

The first HTTPS request I made hung for twenty seconds and timed out. The second returned 503 in 143 milliseconds. That inconsistency is itself a clue: the server was trying to hand the request to a backend, waiting, giving up, and eventually not bothering to wait at all.

The log line that mattered

The site's own error log was useless. Its last entry was seventeen days old, because the log level was WARN and this condition never produced a warning at the virtual-host level.

The server-wide log had it, at INFO:

[INFO] [...#site] Connection idle time too long: 301 while in state: 6 ...
[INFO] [...#site] HttpExtConnector state: 0, request body sent: 0,
                  response body size: -2, response body sent: 0, attempts: 0.
[INFO] [...#site] External processor is not available.
[NOTICE] [...#site] oops! 503 Service Unavailable

External processor is not available is the whole diagnosis in five words. The web server had a request, knew which backend should handle it, and could not get a usable connection to that backend. In LSAPI terms the external processor is the PHP process pool. In PHP-FPM terms it is the pool behind the socket you point fastcgi_pass at. Same shape, same failure mode.

Note attempts: 0 and response body size: -2. It did not fail partway through a response. It never got one started.

PPID 1

The pool for this site is configured for ten children and ten concurrent connections. I expected either zero processes — pool dead, should have respawned — or ten busy ones.

There was exactly one:

  PID   PPID STAT     ELAPSED     TIME %CPU   RSS CMD
793380      1 SN      13:52:58 00:07:31  0.9  7036 lsphp

Read that line carefully, because it contains the entire bug.

PPID 1. The parent is gone. This process was adopted by init. A pool child whose master has died is an orphan, and an orphan is supervised by nothing. Nothing will restart it, nothing will kill it, nothing will notice it.

ELAPSED 13:52:58 against TIME 00:07:31. Fourteen hours of wall clock, seven and a half minutes of CPU. It was not spinning. No load-average alert was ever going to catch this.

STAT SN — interruptible sleep, low priority. The kernel confirms it:

$ grep State /proc/793380/status
State:  S (sleeping)

$ cat /proc/793380/wchan
hrtimer_nanosleep

hrtimer_nanosleep means the process is inside a timed sleep. Some PHP path called sleep() and the value it was handed put the process to bed for what may as well be forever. An API retry backoff, a lock-wait loop, a scraper delay — the specific call matters less than the shape: a wait nobody bounded.

Its open file descriptors closed the case:

$ ls -l /proc/793380/fd
0 -> socket:[598208598]
1 -> /usr/local/.../stderr.log (deleted)
2 -> /usr/local/.../stderr.log (deleted)
3 -> socket:[598208598]

Two descriptors on the listening socket. And stdout and stderr pointing at a log file marked (deleted) — something removed that file while the process held it open, so every byte it wrote for fourteen hours went to an inode with no name. If it printed a fatal error, a stack trace, a reason, that is gone. Never let cleanup delete a log a live process still holds open; rotate it and signal the writer instead.

Why nothing restarted it

Here is the part that generalizes past any one server.

A process supervisor decides whether to start a pool by asking two questions: does the socket file exist, and is the PID in the pid file alive?

srwxr-xr-x 1 ... /tmp/.../siteuser.sock
-rw-r--r-- 1 ... /tmp/.../siteuser.sock.pid

Both answers were yes. The socket file was on disk with the right permissions. The pid file named a process that was alive — sleeping, low priority, adopted by init, but very much running.

So the supervisor concluded the pool was up and never spawned a replacement. Meanwhile the orphan, asleep inside a timer, accepted nothing. Every incoming connection landed on a socket with no reader, waited out the connect timeout, and became a 503.

The pool was neither alive nor dead. It was occupying the identity of a working pool — the socket path and the pid — without doing any of the work. A liveness check that asks "is the PID alive" cannot see this, and neither can anything watching for a crash, because nothing crashed.

Fourteen hours

I went back to the access log and counted requests per hour:

19:00  306
20:00   19
21:00    9
22:00   13
23:00    4

That is the moment of death, and it matches the process start time to within a few minutes. From 20:00 onward the only requests left are bots, retries, and the handful of people who tried, got nothing, and left.

Nineteen requests an hour is not zero, and that is the trap. A "traffic dropped to zero" alert never fires. A daily-total alert compares a bad day against a normal one and shrugs at a 90% drop if the threshold is 95%. The site served errors to every visitor for fourteen hours while the graph showed a line that was low but not flat.

The fix, and it is boring

kill -TERM 793380
rm -f /tmp/.../siteuser.sock /tmp/.../siteuser.sock.pid

Kill the orphan, remove the stale socket and pid file so the supervisor stops believing a pool exists, then send one request. The next request rebuilt the pool by itself. Local response: 200 in 0.171 seconds. From outside: 200 in 0.29 seconds, full page, no error signatures in the HTML. Four healthy children within seconds.

Under ten seconds of repair after fourteen hours of downtime. That ratio is the actual story. The failure was trivial to fix and completely invisible.

What to check instead

Monitor the URL users actually load, and follow redirects to the end. A 302 from a server-level rewrite is not evidence that your application runs. If your monitor reports on the first response in the chain, it is monitoring your rewrite rules.

Assert on the body, not the status code. Match a string that only appears when the application really rendered — a title, a footer, a token in the markup. Byte length works too: a 503 page is a few hundred bytes, a real page is tens of kilobytes.

Alert on relative traffic collapse, not on zero. A drop from 306 to 19 requests in one hour is a far better signal than any absolute floor, and it fires within the hour instead of the next morning.

Treat long-lived pool children as suspect. Pool workers are meant to be recycled. Any child running for hours with almost no CPU deserves a look. ps -eo pid,ppid,etime,time,cmd plus /proc/<pid>/wchan costs nothing in a cron job, and a worker with PPID 1 is an orphan by definition.

Cap every sleep and every outbound call. The root cause under all of this is an unbounded wait. Timeouts on HTTP clients, bounded retry backoffs, lock waits that eventually give up. A worker allowed to sleep forever will eventually do exactly that.

Never delete a log file a process still holds open. Rotate and signal. Otherwise the one artifact that would have told you why is writing into an unlinked inode.

The short version

• DNS, TCP, TLS and the certificate can all be perfect while the site returns 503 to every visitor.
• External processor is not available means the web server could not get a working connection to the PHP pool. It is a backend problem, not a network or certificate problem.
• A pool child whose parent has died gets PPID 1 and is supervised by nothing.
• If that orphan keeps the socket and pid file, the supervisor sees a live pool and refuses to spawn a real one. Neither alive nor dead, just occupying the identity.
• /proc/<pid>/wchan naming a sleep function, with hours of elapsed time and seconds of CPU, is the confirmation.
• Health checks that follow redirects and assert on body content would have caught this in the first minute.

The site was down for fourteen hours. The fix was one signal and two deleted files. Everything expensive about this outage happened in the gap between a process existing and a process working.

Written from production work at Alesta WEB.