My agent had been running fine for eleven weeks when the Raspberry Pi stopped answering. Not crashed — answering, but slowly, in that way that makes you think the network is bad. SSH took forty seconds to give me a password prompt. Then it dumped me into a shell and nothing responded to anything.
At 3:14 AM I pulled the power, because that's what you do at 3:14 AM. When it came back, the database file was intact but the last six hours of writes were gone, and my queue had 412 rows stuck in running.
The agent process itself was still alive and had been the whole time. That was the part that took me a week to understand: the thing that failed was the innocent bystander, and the thing that caused the outage was the only thing that survived.
What I actually built
The Pi runs a small document-processing agent. It watches a table for pending jobs, picks one up, and shells out to a Python subprocess that renders a PDF, extracts text, and writes structured results back. Each subprocess uses a headless browser, so it's ~200-350 MB of RSS while it's working.
There was a concurrency limit. I want to be precise about this, because "concurrency limit" is the phrase I used for two months and it was a lie:
MAX_WORKERS = 4 # I set this
That constant was read in exactly one place: a log line. logger.info(f"starting, max_workers={MAX_WORKERS}"). Every code path that actually started a worker did so from a for row in pending: loop with no check at all, because the loop was written before the constant existed and I never went back.
So when a bulk import dropped 400 documents in at 9 PM, the agent did what it was told: it walked the list and spawned. Not 4 at a time. All of them, as fast as subprocess.Popen could return.
The kernel did exactly what it's designed to do
Here's the part I got wrong in my head for months. I assumed the OOM killer would target the memory hog — my agent — and I'd lose some jobs. Instead the Pi didn't cleanly OOM-kill anything for a long time. With a swapfile on the SD card, 400 browser processes pushed the machine into swap thrash: the kernel spent its time paging, not running, and every process slowed to a crawl, including the ones that were almost done.
When it finally did start killing, it picked by oom_score, which is roughly proportional to resident memory. My agent is a 60 MB Python process. Postgres was the largest single allocation on the box. So the kernel killed Postgres — repeatedly, as it got restarted by systemd — and left the 400 browsers alone, because killing one of them only freed 300 MB and the scoring is about the process, not the herd.
The lesson generalizes further than my Pi: the OOM killer is not a referee. It's a scorer. It has no idea which process is the problem. It knows which process is biggest.
The three things I got wrong
1. I put the limit in the wrong layer. Application-level limits only cover the code paths you remember. My agent had four ways to start work — the cron poller, an HTTP endpoint, a retry sweeper, and a manual CLI I used for debugging. I patched the poller. Three entry points still bypassed it. A limit that lives in one function isn't a limit; it's a habit.
2. My monitoring measured the wrong thing. My alert was a heartbeat: the agent wrote a timestamp to Redis every 30 seconds, and if the timestamp went stale I got a push notification. That check passed for six hours while the box was unusable, because the agent was never blocked — it was thriving. It had 400 children and was having the best night of its life. Liveness monitoring answers "is the process running," which is almost never the question you actually have.
3. My load test used three files. Three documents, 400 KB each, once, on a machine with nothing else happening. It passed. It could not have failed. A load test that doesn't approach the real input volume tests your code's syntax, not its behavior.
What fixed it, in order of how much it mattered
Hard limits at the OS level. The fix that actually holds is one the application can't bypass because it isn't in the application. A systemd drop-in:
[Service]
MemoryMax=2G
MemoryHigh=1.5G
TasksMax=64
OOMPolicy=stop
Restart=on-failure
RestartSec=10
TasksMax=64 alone would have prevented the outage. Sixty-four tasks is roughly fifteen browsers plus overhead, and when the cap is hit, Popen fails with EAGAIN instead of eating the machine. My agent now gets an exception it can log instead of a corpse to discover.
Protect the bystanders. Postgres gets a lower OOM score so the kernel reaches for something else first:
[Service]
OOMScoreAdjust=-800
This isn't a guarantee — nothing at the kernel level is — but it changes which process dies first, and "the database dies last" is the property I actually wanted.
A queue that can only be drained by N workers. Instead of a list I iterate over, jobs are claimed inside a transaction, and the number of claimers is fixed by systemd template units:
# agent-worker@.service
[Service]
ExecStart=/opt/agent/worker.py
systemctl enable --now agent-worker@{1..4} gives me four workers. The fifth doesn't exist. There's no variable to forget to check, because the number of loops in the system is the number of unit files.
Alert on symptoms, not heartbeats. The three numbers I actually watch now: queue depth (is it growing faster than it drains), per-worker RSS (is one job leaking), and time-in-running per job (is something wedged). All three would have fired by 9:30 PM.
Turn off swap on the SD card, or use zram. Thrash is worse than death. A clean OOM kill loses one job; thrashing loses six hours and possibly your SD card. vm.swappiness=10 plus zram gets you a machine that fails fast and loud.
The cost, honestly
Six hours of writes and 412 re-run jobs. Two customers got their documents twice because I hadn't made the pipeline idempotent — the output was written, then written again on retry, and the second pass appended instead of replacing. That's the bug that actually reached a human, and it had nothing to do with memory. It was waiting there the whole time for any retry to expose it.
Total cost: about $40 in my own time refunded, one corrupted afternoon, and a rule I now apply to every service I run: if the failure mode is "the machine dies slowly," you have to bound it outside the thing that's dying.
The eleven good weeks weren't evidence the design was sound. They were evidence the input volume stayed low. That's the trap with automation you run yourself — nothing pages you when the only thing that changed is that it got popular.
If you're running anything agent-shaped on a Pi, put TasksMax and MemoryMax on it tonight. It takes four minutes and it's the difference between an error in your logs and a machine you have to unplug.
I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.
Top comments (0)