A headless Raspberry Pi 4. A failed OS upgrade. No monitor, no keyboard, no network. One AI agent, one Jetson Nano, and a Tailscale connection.
The Situation
I run a headless Raspberry Pi 4 called homepi that handles critical home infrastructure: NextDNS, PiVPN/WireGuard, Tailscale, Docker, and Pi-hole. It sits in a closet with no monitor attached.
Last week, I attempted to upgrade from Raspbian 10 (Buster) to 11 (Bullseye). The apt full-upgrade ran for hours, asked me a few config file questions, then went silent. The Pi never came back to the network.
No DHCP lease. No SSH. No ping. The router showed nothing.
I pulled the 32GB SanDisk microSD card and plugged it into my Mac. Finder showed only the FAT32 /boot partition. The ext4 root partition — where all the configs and logs live — was invisible to macOS.
This is where most people would reach for a fresh SD card and start over. But I had an AI agent, and I wanted to see how far it could go.
Phase 1: Triage From macOS (Blind)
I shared a screenshot of the /boot directory with the agent. It immediately noticed something suspicious: cmdline.txt was dated December 31, 1979 — the Unix epoch. Could be corruption?
Agent: "Step 1: On your Mac terminal, run
cat /Volumes/boot/cmdline.txt"
The file was intact — timestamp corruption only. The kernel command line looked fine. But the agent couldn't go deeper without reading the ext4 root partition. macOS can't do that natively.
We tried installing macFUSE. Homebrew threw errors. The macOS version (26.5, a developer beta of what appears to be macOS 16) wasn't compatible with any available macFUSE build.
Mac was a dead end. We needed Linux.
Phase 2: The Tailscale Pivot
I have a Jetson Nano on my Tailscale network. It runs JetPack (Ubuntu-based) and has a spare microSD slot. The agent suggested:
Agent: "Plug the microSD into a USB card reader and connect it to the Jetson. Then we SSH in via Tailscale."
I inserted the Pi's SD card into the Jetson's internal slot, and the agent connected over Tailscale SSH. Within seconds:
$ lsblk
mmcblk0 29.7G
├─mmcblk0p1 256M vfat boot
└─mmcblk0p2 29.5G ext4 rootfs
Both partitions visible. Both mountable. We had full access to the patient.
Phase 3: The Forensic Investigation
The agent mounted both partitions and began a systematic investigation. Here's what it found — in order:
Finding #1: The Interface Name Heist
The Pi's dhcpcd.conf had a static IP configuration for eth0 at 192.168.1.100. But Bullseye introduces predictable network interface names — eth0 becomes something like enxxx:xx:xx:xx:xx:xx. The interface eth0 no longer existed.
Fix: Added net.ifnames=0 biosdevname=0 to cmdline.txt to preserve traditional naming.
But that wasn't enough. The agent dug into the kernel logs:
May 30 15:17:05 kernel: bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
...
May 30 15:17:14 kernel: eth0: renamed from vethace5160
Finding #2: Docker Was Stealing the Interface Name
The Broadcom Ethernet driver (bcmgenet) was loading and detecting the hardware correctly. But then Docker started first and its virtual Ethernet interface claimed the name eth0 before the physical NIC finished initializing. The real Ethernet had no name to grab.
Fix: Disabled Docker and containerd from auto-starting — removed the symlinks from multi-user.target.wants.
Finding #3: Energy Efficient Ethernet
A known Raspberry Pi 4 quirk: Energy Efficient Ethernet can cause link negotiation failures with some switches.
Fix: Added dtparam=eee=off to config.txt.
Finding #4: The Root Cause 🔴
Three fixes applied, but the agent wasn't satisfied. It kept digging through the systemd journal and found this in the syslog:
May 30 15:17:05 homepi systemd[416]: dhcpcd.service: Failed to locate executable
/usr/lib/dhcpcd5/dhcpcd: No such file or directory
May 30 15:17:05 homepi systemd[1]: dhcpcd.service: Failed with result 'exit-code'.
May 30 15:17:05 homepi systemd[1]: Failed to start DHCP Client Daemon.
This message repeated six times on every boot. dhcpcd was failing silently before it even started — and the Pi had no DHCP client running at all.
The culprit was in /etc/systemd/system/dhcpcd.service.d/wait.conf:
[Service]
ExecStart=
ExecStart=/usr/lib/dhcpcd5/dhcpcd -q -w
This was a DietPi-era override from Buster. In Bullseye, dhcpcd moved from /usr/lib/dhcpcd5/dhcpcd to /usr/sbin/dhcpcd. The override was pointing to a binary that no longer existed. Systemd tried to spawn it, got ENOENT, and gave up.
Fix: One sed command:
sed -i 's|/usr/lib/dhcpcd5/dhcpcd|/usr/sbin/dhcpcd|g' wait.conf
The Full Hit List
When the agent finished its audit, here's what had been fixed:
| # | Issue | Impact |
|---|---|---|
| 1 |
net.ifnames=0 in cmdline.txt |
Interface renamed to enx..., dhcpcd couldn't find it |
| 2 | Docker autostart disabled | Docker veth stole eth0 before NIC initialized |
| 3 |
dtparam=eee=off in config.txt |
EEE causing link negotiation failures |
| 4 | dhcpcd override pointing to dead Buster binary | dhcpcd never started — no IP on any interface |
Layers 1–3 were preventing the interface from working. Layer 4 meant even if the interface existed, dhcpcd couldn't assign an IP. The Pi was booting, the kernel was fine, the Ethernet hardware was detected — but the DHCP client was dead on arrival.
The Moment of Truth
I pulled the SD card from the Jetson, put it back in the Pi 4, and powered it on.
The router showed a new DHCP lease. SSH connected. homepi was back.
$ ssh pi@192.168.1.100
Linux homepi 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l
Last login: Fri May 30 18:45:22 2026
pi@homepi:~ $
The Architecture: How This Worked
The recovery chain worked like this:
macOS (Finder only sees FAT32)
↓ "I can see /boot but not the root partition"
Hermes Agent (running on cloud VPS)
↓ "Plug the SD card into the Jetson — it runs Linux natively"
Jetson Nano (Tailscale SSH, JetPack/Ubuntu)
↓ Mounts mmcblk0p2 (ext4 root) + mmcblk0p1 (vfat boot)
↓ Reads apt logs, dpkg status, systemd journal, kernel logs
↓ Identifies 4 layered issues through forensic analysis
↓ Edits cmdline.txt, config.txt, systemd overrides in-place
Pi 4 (headless, no network)
↓ Boots with fixes → eth0 gets IP → network is back
The agent never had a keyboard plugged into the Pi. It never saw the boot screen. It never pinged the machine. Everything was done through forensic analysis of cold storage, mounted on a different machine across a Tailscale mesh network.
What This Means
We're entering an era where AI agents can perform legitimate sysadmin work — not just generating commands for humans to copy-paste, but actually diagnosing, investigating, and fixing systems autonomously.
The agent didn't just suggest "try reinstalling." It:
- Read and interpreted kernel logs to understand driver initialization order
- Cross-referenced systemd service files with filesystem reality
- Identified that a DietPi-era config survived a distribution upgrade
- Traced the exact chain of failures:
systemd → override → missing binary → no dhcpcd → no IP - Edited configuration files on a mounted filesystem, not the running system
- Performed all of this over Tailscale SSH to a machine it had never accessed before
And it did this for a system that had literally no network access. The patient was in a coma, and the surgeon operated through a different body.
This recovery was performed by Hermes Agent — an open-source AI agent framework that learns from experience and stores reusable skills. The entire session was conducted over Telegram, with the agent accessing the Jetson via Tailscale SSH and mounting the Pi's SD card for forensic analysis.
All four fixes, the investigation logs, and the recovery workflow have been saved as reusable skills for future incidents.
Top comments (0)