The agents were alive. The work was not.
I have been growing a home lab for years the way most people do: a NAS, then a switch worth caring about, then a compute node, then a second one. Adding hardware has a satisfying property — you rack it, you connect it, you configure it, and it either works or it visibly does not.
Then I started running AI agents on that hardware around the clock, and the property stopped holding.
A file server responds when a human asks it to. An agent fleet runs while I sleep. It makes outbound requests, occupies GPUs, collects data on a schedule, and hands results to an analysis node. It also, occasionally, plays dead while remaining alive: the processes exist, the network responds, the dashboard is green, and no actual work has completed in hours. Last season I documented the physical build — network, storage, compute, power, remote recovery. That was the cage. This season is about what lives in it, and what breaks when you keep something alive rather than merely installed.
Three failures from the log, before I get to the lessons:
- A GPU sat silent for 16 hours. Every monitor reported normal.
- A collector reported "success" every day for a month. More than half its runs had actually failed.
- An agent reported that it had configured something. The code it described did not exist anywhere.
None of these are dramatic. All of them are quiet.
"Deploying" and "keeping" are different problems
A server is mostly a deploy target. Power it on, install the OS, bring up the service, set it to start at boot. Faults happen, but the service boundary the administrator intended is reasonably crisp: the port answers or it doesn't.
An agent is messier. It holds a task, queries something external, reads the result, does something that resembles judgment, and moves on. So "is the process running" is not a sufficient question. The questions that matter are: is it working, is it lost, and does the thing it claims to have produced actually exist on disk?
The dangerous mode here is not the crash. A crash is loud — logs go red, alerts fire, monitoring notices. The dangerous mode is silent failure, where nothing is technically down and nothing is getting done. Sixteen hours of a green dashboard over a dead GPU is not really a monitoring misconfiguration. It's a category error about what was being monitored. Watching component liveness and watching work progress produce two different worlds, and I was living in the wrong one.
The night the fleet used up the house internet
The most instructive incident in the set was a network one, and it is instructive precisely because the obvious explanation was wrong.
One night, DNS died across the entire house. Both DNS filters — deliberately deployed as a redundant pair — timed out against their upstream resolvers at the same minute. Redundancy failing in lockstep is a useful clue: it means the shared path broke, not the redundant components.
Bandwidth was the first suspect and the wrong one. The line is 10G. Traffic was around 3 Gbps. There was headroom.
The exhausted resource was NAT ports.
Here is the qualifier that makes this story technically correct rather than merely alarming: my line is MAP-E — IPv4 over IPv6 — where the subscriber is allocated only a couple hundred outer ports (on the order of 240 to 1008), not the roughly 64k port space a plain NAT gets per external address. Under a conventional consumer NAT, a couple dozen parallel agents plus continuous collectors plus monitoring would not have come close to exhausting the table. Under MAP-E, they did, in bursts of one to two minutes, five times over. New sessions failed first, and UDP DNS is the most eager creator of new sessions in a household, so the symptom presented as "the internet is broken" rather than "we are out of ports."
So the generalization is narrower than "consumer networks can't host agents." It is: if you are on a shared-address scheme — MAP-E, CGNAT, anything where your outer port range is a few hundred rather than tens of thousands — then concurrent session count, not bandwidth, is the budget you have to manage. This applies to anything that opens many short-lived connections: agent fleets, scrapers, P2P. No speed test will warn you about it. The speed test will look excellent.
The fix was unglamorous. I halved parallelism, from 16 to 8. Daily throughput did not change, because throughput had been limited by a different quota all along — the parallelism above 8 had been buying nothing but NAT pressure. I also moved DNS to an IPv6 upstream so it stops traversing MAP-E entirely.
The line I keep coming back to: overusing AI bills you in tokens, and it also bills you in your household's NAT table.
Care decomposes into monitoring, power, and network
Three things demand attention, and each has a default misconception attached.
Monitoring. The question is not "is it alive" but "is it working." The misconception is that process supervision suffices. It does not — a process can be perfectly supervised and perfectly idle.
Power. The question is not only "does it stay up" but "can it come back." The misconception is that anything with autostart configured is therefore unattended-capable. Unattended means recovering without you, from a state you did not anticipate, at an hour you were not awake.
Network. The question is not throughput but session behavior — exhaustion, skew, concurrency. The misconception is that a fast speed test means the network is adequate. See above.
The four things I check now
Before scaling any of this, the starting checklist is short on purpose:
- What did this agent last actually complete? Not "when did it last run" — what finished.
- Does the success report correspond to an artifact? If it claims to have written a file, is the file there? If it claims to have written code, does the code exist?
- If it fails, will a human notice? Is there a notification or a log a person will actually read, or does failure look identical to success from the outside?
- After a power cut or a line drop, how far back does it come automatically? Know the boundary before you need it.
The framing that keeps me honest: unattended operation is not unsupervised operation. It is a design that reduces the number of places a human must look, so that the hours a human isn't looking can be increased. Get that backwards and you spend months keeping a silence that looks like health.
Build the cage first, by all means. But an agent is not an ornament you place inside it. Design the care before you bring the animal home.
Full write-up
The canonical version of this piece, in Japanese, opens a season of incident reports from the lab — the 16-hour silent GPU, the collector that lied for a month, the agent that reported code it never wrote, each with the numbers and the recovery path.
Top comments (1)
The "collector reported success for a month but more than half its runs had actually failed" — this is the part that hit hardest. It maps almost 1:1 to something I have seen with AI agent outputs: the agent returns a structured "done" response, the orchestrator logs it as a success, and the artifact it claims to have produced either does not exist or is silently wrong.
The distinction you make between process liveness and work progress is the right mental model. I have been thinking about it as the gap between "the agent ran" and "the agent produced something I can verify." Same gap, different wording. The four-point checklist is basically what any team running agent fleets should have on their wall — especially point 2 (does the artifact exist?) because that is the one most setups skip.
Curious about the MAP-E angle. Most writeups I have seen about home lab agent failures focus on GPU memory, thermal throttling, or OOM kills. NAT port exhaustion as the failure mode is genuinely novel. Did you find any way to get visibility into the port table usage without polling the CPE directly?