DEV Community: chemy_pvl

"Is it alive?" is the wrong question. Ask "is it working?"

chemy_pvl — Tue, 14 Jul 2026 00:51:22 +0000

"Is it alive?" is the wrong question. The right one is "is it working?"

I run a home lab: a couple of always-on machines, a GPU node, some AI jobs that are supposed to chew through work while I'm asleep or at my day job. The first fear anyone has with unattended infrastructure is simple — if it breaks, will I even notice? So you add notifications. You put up a dashboard. You get an alert when a process dies. That's a real and necessary step.

But here's the thing I keep relearning: monitoring can be green while nothing is getting done.

Last season on the blog I wrote up the incidents from actually running this stuff unattended — the silent stops, the healthy-looking numbers that were lying, the resource contention nobody flagged. The one that reframed everything for me: a GPU node sat idle for 16 hours. The process was alive. The network was fine. Every check I had said "OK." The only signal that didn't lie was the one I wasn't watching — the output count wasn't going up.

This post is the design I landed on, and the checklist I now use before I trust any monitoring setup, mine included.

Split monitoring into three layers

Most home-lab monitoring collapses into one undifferentiated blob of "checks." It's much easier to reason about — and much easier to find the holes — if you split it into three layers and ask what each one can and cannot tell you.

Layer 1 — Liveness. Is the process alive? This tells you it crashed, or it restarted. What it misses: the process is running but the work has stalled.

Layer 2 — Connectivity. Does the port open? Does the API respond? This tells you the thing is reachable from outside. What it misses: it responds, but it isn't doing the job you think it's doing.

Layer 3 — Progress. Did the amount of finished work increase over the last unit of time? This tells you the job is actually moving forward. What it misses: your definition of "output" could be wrong, or the measurement itself could be broken. (That failure mode is real, and I'll come back to it.)

Liveness is the "notice when it dies" layer. It's the cheapest to add and the one most people already have. Connectivity is the next step up — proving the port is open and the API answers, not just that a PID exists. Both of those are worth having. Neither of them answers the question I actually care about at 2am.

Because for AI agent workloads and GPU nodes, the interesting failures aren't crashes. They're waits. Queues that stop draining. Work that blocks on something and never times out. Silent stops where every component is technically healthy and collectively they've achieved nothing.

Measuring "is it alive" does not tell you "is it working." That gap is the whole reason to bring observability — not just monitoring — into a home lab.

Green dashboards lie, and here's how

Two incidents changed how much I trust a displayed value.

Capacity showed 99% free while real errors were happening. The monitored value said there was plenty of room. Actual operations were failing. The number wasn't wrong when it was written — it was stale, and stale values get rendered as "normal" by default. Nothing in the display distinguishes "I checked this a second ago and it's fine" from "I last checked this hours ago and it was fine then."

The rule I took from that: when a monitored value and a real error contradict each other, believe the real error. And structurally, the fix isn't a better threshold — it's a freshness gate. A metric that hasn't been updated recently should not be allowed to count as healthy.

A 10G link that only delivered about 3G effective. Bandwidth alone looked survivable. What was actually going on showed up somewhere else: the outward-facing port count climbed to roughly 240. That's the pattern worth internalizing — resources usually have a leading indicator that saturates before the headline number does. If you only watch the headline metric, you find out late.

So the honest summary is: monitoring starts lying the moment you install it. The value might be old. The notification might not get delivered. The word "OK" on a dashboard might be actively concealing a stalled job.

Put progress on top of what you already have

I want to be clear that none of this means throwing out your liveness checks or your telemetry pipeline. Both are necessary. The mistake is letting their roles blur together.

Liveness monitoring tells you something went down.
The telemetry pipeline carries observed values from A to B.
Progress monitoring asks whether the workload is producing results.

If you already have the first two, the realistic move is to add one progress signal on top. Not to rebuild everything. Pick the workload that would hurt most if it silently stalled, define what "one unit of progress" means for it, and alert when that number stops increasing for longer than it should.

The same skepticism generalizes. Backups are the classic case: the existence of a backup tells you nothing about whether you can restore. Only an actual restore test does. I found broken restores and prune candidates precisely because I stopped trusting the "backup completed" line and went and checked. Monitoring is not a substitute for operating the system — it's an instrument for finding the holes in how you operate it.

Which layers should you actually have?

Depends on what you're running. Roughly how I'd advise choosing:

Setup	Good fit for	Bad fit for	Upside	Downside
Liveness only	Anyone with no down-alerts at all today	Anyone depending on unattended jobs finishing	Fast to start	Silent stalls slip through
Liveness + connectivity	Services consumed from outside	Anyone who needs to see output increase	Easy to isolate unreachability	Confuses "responds" with "produces"
Liveness + connectivity + progress	Unattended AI work, backups, scheduled jobs	Anyone who can't yet define "progress"	You can answer "is it working?"	Needs a per-workload definition of output
Add nothing yet	Still deciding what you run and how you'd respond	Anyone already trusting unattended work	Avoids alert noise	The blind spot stays

If you can't define progress for a workload yet, don't rush to add more metrics. Write down the answer to this instead: what is the moment at which I can say this job succeeded? Is it that a file appeared? That N items completed? That a restore verification passed? The answer is different for every workload, and metrics you add before you can answer it are decoration.

The checklist: count your own layers

Go count what you have right now.

Liveness — if a process or service dies, will I find out? Through a channel I've actually tested, not one I assume works?
Connectivity — can I confirm the entry points I depend on are reachable and responding?
Progress — if output or completion count stops increasing for a defined window, will anything tell me?
Freshness — can a stale value be displayed as "normal" in my setup? If yes, that's a hole. Gate it.
Contradiction rule — when a metric says fine and a real operation fails, do I have the discipline to believe the failure?
Leading indicators — for my constrained resources, am I only watching the headline number, or the thing that saturates first?
Verification, not existence — for backups and anything else where "it exists" ≠ "it works": have I actually exercised the recovery path?

A missing layer isn't a failure. It's just the next thing to build. No liveness alerting? Start there — it's the cheapest reduction in "didn't notice for 16 hours" risk you can buy. Telemetry that you suspect isn't flowing? Verify the pipeline before you build on it. Liveness and connectivity both green but work still stalling? That's exactly where progress monitoring earns its keep.

What I want to be left with — and what I'd want for anyone running a lab like mine — isn't having notifications. It's being able to confirm, at the moment it matters, that things are moving forward.

Full write-up

The full Japanese article, with the incident details, the layer-by-layer breakdown, and the reasoning behind the freshness gate and the connectivity leading indicators, is on the blog:

「生きているか」ではなく「働いているか」——自宅サーバー監視を実装する / Not "is it alive" but "is it working": implementing home-server monitoring

Home Lab AI Agents: Deploying Is Easy, Keeping Them Alive Is Not

chemy_pvl — Thu, 09 Jul 2026 14:29:11 +0000

The agents were alive. The work was not.

I have been growing a home lab for years the way most people do: a NAS, then a switch worth caring about, then a compute node, then a second one. Adding hardware has a satisfying property — you rack it, you connect it, you configure it, and it either works or it visibly does not.

Then I started running AI agents on that hardware around the clock, and the property stopped holding.

A file server responds when a human asks it to. An agent fleet runs while I sleep. It makes outbound requests, occupies GPUs, collects data on a schedule, and hands results to an analysis node. It also, occasionally, plays dead while remaining alive: the processes exist, the network responds, the dashboard is green, and no actual work has completed in hours. Last season I documented the physical build — network, storage, compute, power, remote recovery. That was the cage. This season is about what lives in it, and what breaks when you keep something alive rather than merely installed.

Three failures from the log, before I get to the lessons:

A GPU sat silent for 16 hours. Every monitor reported normal.
A collector reported "success" every day for a month. More than half its runs had actually failed.
An agent reported that it had configured something. The code it described did not exist anywhere.

None of these are dramatic. All of them are quiet.

"Deploying" and "keeping" are different problems

A server is mostly a deploy target. Power it on, install the OS, bring up the service, set it to start at boot. Faults happen, but the service boundary the administrator intended is reasonably crisp: the port answers or it doesn't.

An agent is messier. It holds a task, queries something external, reads the result, does something that resembles judgment, and moves on. So "is the process running" is not a sufficient question. The questions that matter are: is it working, is it lost, and does the thing it claims to have produced actually exist on disk?

The dangerous mode here is not the crash. A crash is loud — logs go red, alerts fire, monitoring notices. The dangerous mode is silent failure, where nothing is technically down and nothing is getting done. Sixteen hours of a green dashboard over a dead GPU is not really a monitoring misconfiguration. It's a category error about what was being monitored. Watching component liveness and watching work progress produce two different worlds, and I was living in the wrong one.

The night the fleet used up the house internet

The most instructive incident in the set was a network one, and it is instructive precisely because the obvious explanation was wrong.

One night, DNS died across the entire house. Both DNS filters — deliberately deployed as a redundant pair — timed out against their upstream resolvers at the same minute. Redundancy failing in lockstep is a useful clue: it means the shared path broke, not the redundant components.

Bandwidth was the first suspect and the wrong one. The line is 10G. Traffic was around 3 Gbps. There was headroom.

The exhausted resource was NAT ports.

Here is the qualifier that makes this story technically correct rather than merely alarming: my line is MAP-E — IPv4 over IPv6 — where the subscriber is allocated only a couple hundred outer ports (on the order of 240 to 1008), not the roughly 64k port space a plain NAT gets per external address. Under a conventional consumer NAT, a couple dozen parallel agents plus continuous collectors plus monitoring would not have come close to exhausting the table. Under MAP-E, they did, in bursts of one to two minutes, five times over. New sessions failed first, and UDP DNS is the most eager creator of new sessions in a household, so the symptom presented as "the internet is broken" rather than "we are out of ports."

So the generalization is narrower than "consumer networks can't host agents." It is: if you are on a shared-address scheme — MAP-E, CGNAT, anything where your outer port range is a few hundred rather than tens of thousands — then concurrent session count, not bandwidth, is the budget you have to manage. This applies to anything that opens many short-lived connections: agent fleets, scrapers, P2P. No speed test will warn you about it. The speed test will look excellent.

The fix was unglamorous. I halved parallelism, from 16 to 8. Daily throughput did not change, because throughput had been limited by a different quota all along — the parallelism above 8 had been buying nothing but NAT pressure. I also moved DNS to an IPv6 upstream so it stops traversing MAP-E entirely.

The line I keep coming back to: overusing AI bills you in tokens, and it also bills you in your household's NAT table.

Care decomposes into monitoring, power, and network

Three things demand attention, and each has a default misconception attached.

Monitoring. The question is not "is it alive" but "is it working." The misconception is that process supervision suffices. It does not — a process can be perfectly supervised and perfectly idle.

Power. The question is not only "does it stay up" but "can it come back." The misconception is that anything with autostart configured is therefore unattended-capable. Unattended means recovering without you, from a state you did not anticipate, at an hour you were not awake.

Network. The question is not throughput but session behavior — exhaustion, skew, concurrency. The misconception is that a fast speed test means the network is adequate. See above.

The four things I check now

Before scaling any of this, the starting checklist is short on purpose:

What did this agent last actually complete? Not "when did it last run" — what finished.
Does the success report correspond to an artifact? If it claims to have written a file, is the file there? If it claims to have written code, does the code exist?
If it fails, will a human notice? Is there a notification or a log a person will actually read, or does failure look identical to success from the outside?
After a power cut or a line drop, how far back does it come automatically? Know the boundary before you need it.

The framing that keeps me honest: unattended operation is not unsupervised operation. It is a design that reduces the number of places a human must look, so that the hours a human isn't looking can be increased. Get that backwards and you spend months keeping a silence that looks like health.

Build the cage first, by all means. But an agent is not an ornament you place inside it. Design the care before you bring the animal home.

Full write-up

The canonical version of this piece, in Japanese, opens a season of incident reports from the lab — the 16-hour silent GPU, the collector that lied for a month, the agent that reported code it never wrote, each with the numbers and the recovery path.

AIを自宅で飼うということ — ホームラボAI 24時間無人運用シーズン開幕