The Local LLM Lab — Part 12
One controlled experiment. One number. One verdict.
A home server is a stack of things that can fail. 59 containers. 94 active systemd units. A GPU with models on it. A network of ports, a registry, a CI runner, a media server, a photo library, an LLM observability stack. Any one of them can die, and the question is never "will it fail" — it's "will you know, and will it come back."
I instrumented the whole stack: every container has a health check, every critical service has an uptime probe, and the monitor watches both. Over the last 7 days the probes ran 10,000 checks. This is what they found — and it is not a clean 100%. One probe target has been refusing connections for the entire window, and the monitor flagged it on day one. That is the point.
The setup
- Machine: HP Z840 Workstation, 2× Xeon E5-2680 v4, 64 GB RAM, one RTX 3090
- Stack: 59 Docker containers, 94 active systemd units (93 running, 1 failed)
-
Monitoring:
- Container health: Docker's own healthchecks on the containers that define them (the rest report state only)
- Service uptime: two HTTP probes against the homelab monitor's own endpoints, every 60 seconds, 7-day window, fail threshold of 2 consecutive failures
- Event log: edge-triggered alerts for OOM kills, threshold crossings, disk I/O spikes
-
The two things that are not up:
nginx.service(the old reverse proxy, a leftover) and the 9802 probe target (a port that no longer listens). Both are flagged, both are explained, neither is hidden.
The two non-up states are the honest details. They're not mistakes I'm hiding — they're the things that are down, and they're down for known reasons: a retired reverse-proxy unit, a retired probe port. The monitor sees them, flags them, and I left them, because the roles they used to fill no longer exist behind them. The article is about the detection, not the failures — the failures are the proof that the detection works.
The number
Over 7 days, 10,000 uptime checks ran against two endpoints. One returned 100% — 5,000 checks, in the low-teens of milliseconds, zero failures. The other returned 0% — connection refused on every check — and the monitor has flagged it as a critical since day one.
| Check | Target | Interval | Checks (7d) | Uptime | Latency | Failures |
|---|---|---|---|---|---|---|
| HomeLab 9800 | http://ardi:9800 | 60 s | 5,000 | 100% | ~14 ms | 0 |
| HomeLab 9802 | http://ardi:9802 | 60 s | 5,000 | 0% | refused | 5,000 |
The 0% is the number that matters. Not "the stack is up" — that's a state. "One endpoint up, one endpoint refusing connections for a week, both named" is a measurement. The 9800 endpoint answered 200 in about 14 ms on every check. The 9802 endpoint refused the connection on all 5,000. The fail threshold is 2 consecutive failures, so the 9802 alarm fired at the 2-minute mark on day one and has not cleared — because the port genuinely stopped listening.
The event log over the same week adds the texture: zero OOM kills, eight disk I/O spikes (the photo server writing thumbnails, the CI runner pulling images), and the single critical — 9802 down. No memory events, no GPU events, no service crashes. The only other "crit" on the box is the known, explained nginx unit.
What the monitoring actually does
The stack has three layers of liveness, and they answer three different questions:
- Is the process alive? Docker's healthchecks and systemd's unit state. This catches a crash — a process that died and didn't restart. It's the coarsest layer, and it's the one that's always on.
- Is the service answering? The uptime probes. A process can be alive and not answering — hung, deadlocked, stuck in a retry loop. The probe is a 60-second HTTP GET that fails after 10 seconds. It catches the "alive but not working" state that the process check misses.
- Is the service working correctly? The event log. A service can answer 200 and still be wrong — returning an error body, a 500 behind a proxy, a timeout that's just under the threshold. The edge-triggered events (OOM, threshold crossings, I/O spikes) catch the "answering but degrading" state.
The three layers are not redundant. Each one catches a failure the others miss. The process check catches the crash. The probe catches the hang. The event log catches the degradation. A 94-unit stack needs all three, because a 94-unit stack has 94 different ways to be subtly wrong.
The counterargument: 100% uptime is boring — so here is 100% and 0%
The fair objection: an article about a stack that didn't fail is an article with no story. 100% uptime is the baseline, not the achievement.
The point is the opposite. The achievement is that the stack didn't fail and that I would have known if it did. A 100% uptime number is only meaningful if the monitoring is good enough to have caught a failure. The 10,000 checks, the 2-consecutive-failure threshold, the three-layer liveness — that's the system that makes the 100% a fact instead of a hope. If 9800 had dropped for 30 minutes last Tuesday, the probe would have flagged it at the 2-minute mark, the event log would have recorded it, and I would have had a named, timestamped, attributed failure to look at — not a "why is the photo app slow" mystery.
And this week the system didn't have to be hypothetical. The 9802 endpoint stopped answering — not for a few minutes, for the whole window — and the monitor carried it as a critical from day one. Nobody "discovered" it. It's in the dashboard, marked, explained, and left, because the role behind it no longer exists. Same with the nginx unit: down for weeks, flagged the entire time, explained, and left. That's the system working: the failures are known, named, and triaged, not hidden.
What I would do now
If I were building a homelab today, the monitoring would have exactly this shape:
- Every container: a healthcheck if the image defines one, a state check if it doesn't. The cost is nothing; the value is the "is the process alive" layer.
- Every critical service: a 60-second uptime probe with a 10-second timeout and a 2-consecutive-failure threshold. The threshold is the key — 1 failure is a blip, 2 is a problem. One failure a day, over 94 units, would be 94 false pages; the threshold kills that.
- The event log: edge-triggered alerts for the things that matter — OOM kills, threshold crossings, disk I/O spikes. Not a log of every metric, but a log of every event. The difference is the difference between a dashboard you scroll and a feed you read.
- The one rule: a failure that is known, named, and explained is not a failure. It's a state. The nginx unit is down, and the 9802 probe is down, and that's fine, because both are flagged, both are explained, and neither is hiding. The system's job is not to prevent every failure — it's to make every failure known.
The 94-unit stack stays alive not because nothing fails, but because when something fails, the failure has a name, a timestamp, and a place in the dashboard. The 100% on 9800 is the output of that system, not the input — and the 0% on 9802 is the proof that the output is real.
Verdict
94 active systemd units, 59 containers, 10,000 uptime checks over 7 days: one endpoint at 100% (~14 ms mean), one endpoint at 0% (refused, flagged from day one), zero OOM kills, eight disk I/O spikes, two known-and-explained non-up states. The stack stays alive because the monitoring catches the three layers of failure — the crash, the hang, the degradation — and names each one. The 100% is not luck, and the 0% is not a miss. It's 10,000 checks that would have told me, in two minutes, if anything had broken.
The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of uptime and event telemetry on the box, not estimated. The series continues with the cost of keeping all of this alive — the full electricity bill for the whole machine, not just the GPU.
Top comments (0)