My Raspberry Pi 5 Kept Vanishing From the Network — Diagnosing With journalctl and Fixing It With a Watchdog

The symptom

I run a headless Raspberry Pi 5 (over Wi-Fi) via SSH and Tailscale. Every so often it would stay powered on (LED lit) but completely vanish from the network:

• SSH over LAN → timeout
• SSH over Tailscale → timeout
• RDP → timeout

And it kept recurring — every time I "fixed" it by power-cycling. Here is how I actually diagnosed it and set up auto-recovery.

Step 1: tailscale status tells you device-side vs your-side

When SSH hangs, it's tempting to assume "the Pi died." But the problem can be on your side (your PC's Wi-Fi / VPN). Rule that out first.

If you use Tailscale, tailscale status is the fastest check:

100.x.y.z   raspberrypi   user@   linux   active; relay "..."; offline, last seen 1h ago, tx 124956 rx 0

The key is the peer line: offline, last seen 1h ago. This comes from Tailscale's coordination server — it's a third-party fact, independent of your own SSH attempt.

• peer offline → the Pi really did drop off the network
• peer active/online → it's your side that's broken (your VPN/network)

last seen also tells you when it dropped. tx ... rx 0 means "you're sending, but getting zero bytes back."

Confirm your own side is healthy too via tailscale status --json → BackendState=Running / Self.Online=true.

Step 2: After recovery, read journalctl -b -1 (the previous boot)

This is the part people miss. While the Pi is off the network you obviously can't pull its logs — you read them after it's back. But the moment you reboot, the "current boot" is brand new, so journalctl (default) and dmesg won't contain the moment it dropped.

The drop is in the previous boot:

# tail of the previous boot
journalctl -b -1 --no-pager | tail -30

# only wlan0 / network-related lines
journalctl -b -1 --no-pager | grep -iE "wlan0|cfg80211|brcmfmac|LinkChange|network is unreachable" | tail -40

What I found, in order:

19:57  avahi-daemon: Withdrawing address record ... on wlan0   # IPv6 address flapping
20:08  tailscaled: LinkChange: major, rebinding ... rebind-reason=[time-jumped(13m50s),ips-changed,protocols-changed]
22:06  tailscaled: ... connect: network is unreachable          # fully down

The important part: the logs kept flowing the whole time. So this was not a full OS freeze or OOM — only the wlan0 / network layer dropped while the OS stayed alive (no oom, no panic at the tail of journalctl -b -1).

I couldn't pin the root trigger (why wlan0 dropped) from the logs — there was no explicit driver crash line, and vcgencmd get_throttled read 0x0 after recovery (no undervoltage flag). So from here it's a symptomatic fix for wlan0 drops in general.

The fix: a NetworkManager watchdog

If "the OS is alive but only the network is down," then a cron job that watches for it and restarts networking can auto-recover. Recent Pi OS uses NetworkManager (resolv.conf says # Generated by NetworkManager), so restart that.

/home/pi/.local/bin/net-watchdog.sh:

#!/usr/bin/env bash
# If the gateway is unreachable, restart NetworkManager to bring wlan0 back.
GATEWAY=192.168.1.1   # replace with your router IP
LOG=/home/pi/logs/net-watchdog.log
if ping -c 3 -W 3 "$GATEWAY" >/dev/null 2>&1; then
  exit 0
fi
echo "$(date -Is) gateway $GATEWAY unreachable, restarting NetworkManager" >> "$LOG"
systemctl restart NetworkManager

/etc/cron.d/net-watchdog (every 3 minutes, as root):

*/3 * * * * root /home/pi/.local/bin/net-watchdog.sh

Install:

chmod +x /home/pi/.local/bin/net-watchdog.sh
mkdir -p /home/pi/logs
echo "*/3 * * * * root /home/pi/.local/bin/net-watchdog.sh" | sudo tee /etc/cron.d/net-watchdog

Verify it does NOT restart when the network is fine

You don't want spurious restarts. Run it once by hand while the network is healthy:

/home/pi/.local/bin/net-watchdog.sh
echo $?        # → 0
cat /home/pi/logs/net-watchdog.log   # → no such file (= it did not restart anything)

While ping succeeds, nothing is logged and systemctl restart never runs. It only acts when the network is actually down.

Takeaways

• "Powered on but gone from the network" is usually a network-layer drop — the OS is still alive.
• Fastest triage: tailscale status (third-party offline record) → after recovery, journalctl -b -1 (the previous boot).
• A gateway-ping → restart-NetworkManager watchdog in cron auto-recovers within minutes, as long as the OS keeps running.
• Caveat: a cron-driven watchdog can't help if the OS itself stops executing.

I still haven't pinned the root cause, but at least I'm out of the "power-cycle it by hand every time" loop.

Follow-up: a selective outage the watchdog couldn't see — and watchdog v2

About two weeks after deploying the watchdog above, the Pi died in a completely different way — and since the gateway ping still succeeded, the watchdog never fired.

The symptom (nothing like last time)

• SSH (over Tailscale) worked fine
• Ping to the gateway worked fine
• But DNS resolution was completely dead (getent hosts github.com failed)
• Behavior differed per destination:
  • Google properties returned HTTPS 200
  • GitHub / Cloudflare (1.1.1.1) / everything else → TCP connect timeout
  • IPv6 was entirely dead

With "only Google works," it looks like an ISP or router outage. Spoiler: it was the Pi itself again — a wedged wlan0 Wi-Fi client state.

Triage 1: hit the same destinations from another device on the same router (the decisive check)

From a PC on the same Wi-Fi, GitHub and everything else worked normally. → The router and the ISP are innocent; the problem is Pi-specific. Skipping this check would have sent me down the useless "reboot the router" path.

Triage 2: probe each DNS server directly over UDP

No dig or nslookup on the Pi? A few lines of python3 can send a raw DNS query:

import socket, struct

def q(server, timeout=3):
    # minimal DNS A query for github.com
    pkt = struct.pack('>HHHHHH', 0x1234, 0x0100, 1, 0, 0, 0)
    for part in b'github com'.split():
        pkt += bytes([len(part)]) + part
    pkt += b'\x00' + struct.pack('>HH', 1, 1)
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.settimeout(timeout)
    try:
        s.sendto(pkt, (server, 53))
        data, _ = s.recvfrom(512)
        return f'OK answers={struct.unpack(">H", data[6:8])[0]}'
    except Exception as e:
        return f'FAIL {type(e).__name__}'
    finally:
        s.close()

for srv in ['192.168.1.1', '100.100.100.100', '8.8.8.8', '1.1.1.1']:
    print(srv, '->', q(srv))

Result: router=FAIL / Tailscale MagicDNS=FAIL / 8.8.8.8=OK / 1.1.1.1=FAIL. So DNS wasn't "down" — traffic to specific destinations was down, and DNS failure was just one symptom. ip route get and the firewall were both normal.

The fix: a full wlan0 reconnect (reapply is not enough)

nmcli dev reapply wlan0 (re-applying the config) did not fix it. What fixed it was a full disconnect → connect. Since SSH itself rides on wlan0, detach the command so it survives your own disconnection:

sudo nohup bash -c 'nmcli dev disconnect wlan0; sleep 3; nmcli dev connect wlan0' >/dev/null 2>&1 &

About 20 seconds later everything was back: all destinations 200, and the router DNS that had "looked dead" was answering again. The DNS death, the selective timeouts, and the IPv6 blackout were all symptoms of one root cause: the wedged wlan0 client state.

watchdog v2: also probe external connectivity

Gateway ping alone misses this failure mode, so I added a second tier:

#!/bin/bash
# net-watchdog v2: two-tier self-healing
# tier1: gateway unreachable -> restart NetworkManager
# tier2: gateway OK but external connectivity mostly dead (<2 of 4 probes) -> full wlan0 reconnect
LOG=/home/pi/logs/net-watchdog.log
STATE=/home/pi/logs/net-watchdog.last-reconnect
GW=192.168.1.1   # replace with your router IP
mkdir -p /home/pi/logs

if ! ping -c3 -W3 $GW >/dev/null 2>&1; then
  echo "$(date -Is) gateway unreachable -> restart NetworkManager" >> $LOG
  systemctl restart NetworkManager
  exit 0
fi

alive=0
for url in https://www.google.com https://github.com https://1.1.1.1 https://www.yahoo.co.jp; do
  curl -4 -sS --max-time 5 -o /dev/null "$url" 2>/dev/null && alive=$((alive+1))
done

if [ "$alive" -lt 2 ]; then
  now=$(date +%s)
  last=$(cat "$STATE" 2>/dev/null || echo 0)
  if [ $((now - last)) -lt 1800 ]; then
    echo "$(date -Is) external dead (alive=$alive/4) but reconnected <30min ago, skip" >> $LOG
    exit 0
  fi
  echo "$now" > "$STATE"
  echo "$(date -Is) gateway OK but external dead (alive=$alive/4) -> wlan0 reconnect" >> $LOG
  nmcli dev disconnect wlan0
  sleep 3
  nmcli dev connect wlan0
fi
exit 0

Design notes:

• 2 of 4 independent probes alive = healthy — a single site being down never triggers a reconnect
• https://1.1.1.1 is IP-literal, so this probe still works when DNS is dead
• Reconnects have a 30-minute cooldown (no flapping)
• As with v1: run it by hand while healthy and confirm it does nothing before wiring it into cron

Follow-up takeaways

• A gateway-ping watchdog is blind to "selective outage caused by a wedged Wi-Fi client state"
• "Only some sites work" looks like an ISP problem but can be your own device. The fastest triage is hitting the same destinations from another device on the same LAN
• nmcli dev reapply and a full disconnect/connect are different things — escalate to the full reconnect when reapply doesn't cut it