I Built a Health Monitor for My AI Agents — Now They Tell Me When They're Dying

For two weeks, my AI content pipeline was silently broken.

The cron job was still running. The logs showed "SUCCESS." But no articles were actually posting. The Dev.to API was returning 429s (rate limited) and the script was swallowing the error because I forgot to check the response status.

I only noticed because I happened to check the blog and saw the gap. Two weeks of missing posts. Two weeks of thinking everything was fine while the system was quietly failing.

That was the moment I realized: my agents need a pulse.

The Problem: Silent Failures Are the Worst Failures

I run three AI agents on three machines. Celebi on a Mac Mini, ProgrammierMinna on a Windows PC, DocMinna on... also the Mac Mini. They're connected via a simple router, triggered by cron jobs, and they talk to me through Telegram.

When everything works, it's magic. When something breaks, it's archaeology.

Common failure modes I'd hit:

• Ollama crashed on the Windows PC after a Windows update. No one knew. Queries just timed out.
• Disk full on the Mac Mini because model files kept accumulating. Logs stopped rotating. The system ground to a halt.
• API rate limits on Dev.to (max 10 posts/day). The publish script didn't retry or notify.
• Router misconfiguration after a router restart. The Windows PC got a new IP. Queries fell into the void.
• Model pulled but not loaded. I'd update a model, restart Ollama, but forget to actually ollama run it. First query would error out.

Each of these took 20-60 minutes to diagnose. Not because they're hard problems, but because I didn't know where to look.

The Fix: A 30-Line Health Check Script

I didn't build Prometheus. I didn't install Grafana. I wrote a Python script that runs every 10 minutes, checks the things I care about, and sends me a Telegram message if something is wrong.

That's it. No dashboard. No metrics server. Just "tell me when it's broken."

# health_check.py
import requests
import shutil
import subprocess
import json

TELEGRAM_BOT_TOKEN = "your_token"
TELEGRAM_CHAT_ID = "your_chat_id"
MACHINES = {
    "mac_mini": "http://192.168.1.102:11434",
    "windows_pc": "http://192.168.1.106:11434",
    "ubuntu": "http://192.168.1.100:11434",
}
MODELS = ["qwen3.5:9b", "qwen3-coder:30b", "granite3.2:8b"]

def alert(message):
    url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
    requests.post(url, json={"chat_id": TELEGRAM_CHAT_ID, "text": f"🚨 {message}"})

def check_ollama(machine_name, url):
    try:
        r = requests.get(f"{url}/api/tags", timeout=5)
        if r.status_code != 200:
            alert(f"Ollama down on {machine_name}: HTTP {r.status_code}")
            return False

        models = [m["name"] for m in r.json().get("models", [])]
        for model in MODELS:
            if model not in models:
                alert(f"Model {model} missing on {machine_name}")
                return False
        return True
    except requests.exceptions.ConnectionError:
        alert(f"Ollama unreachable on {machine_name}")
        return False

def check_disk():
    disk = shutil.disk_usage("/")
    percent = disk.used / disk.total * 100
    if percent > 90:
        alert(f"Disk usage: {percent:.1f}% — clean up needed!")
        return False
    return True

def check_last_article():
    # Check if an article was posted in the last 3 days
    # This reads a simple timestamp file written by the publish script
    try:
        with open("/tmp/last_article_timestamp.txt") as f:
            last = float(f.read().strip())
        import time
        if time.time() - last > 3 * 24 * 3600:
            alert("No article published in 3 days — pipeline may be stuck")
            return False
        return True
    except FileNotFoundError:
        return True  # No articles yet, that's fine

if __name__ == "__main__":
    all_ok = True
    all_ok &= check_disk()

    for name, url in MACHINES.items():
        all_ok &= check_ollama(name, url)

    check_last_article()  # Non-critical, don't fail on this

    if all_ok:
        # Optional: send heartbeat once per day
        pass

Total dependencies: requests (probably already installed). Total lines: ~50. Total setup time: 10 minutes.

What I Actually Monitor

1. Ollama Heartbeat

The most critical check. Every machine gets pinged at /api/tags every 10 minutes. If it doesn't respond in 5 seconds, I get a Telegram alert with the machine name.

This caught a Windows update restart last week. The PC came back up, but Ollama didn't start (I hadn't added it to startup yet). I knew within 10 minutes instead of finding out when I actually needed it.

2. Model Presence

Even if Ollama is running, the model I need might not be loaded. I check that my three core models (qwen3.5:9b, qwen3-coder:30b, granite3.2:8b) are available on their respective machines.

This saved me once when I updated qwen3-coder and the new version had a different tag. The old tag was gone. The router was pointing to a ghost. The monitor caught it before the first failed query.

3. Disk Space

Model files are big. A 30B parameter model is ~20GB. Three machines times three models each — that's a lot of storage that grows quietly. I alert at 90% disk usage.

The Mac Mini has a 256GB SSD. I hit 95% once and the system started swapping aggressively. Response times went from 2 seconds to 30 seconds. Now I clean up before it hurts.

4. Pipeline Liveness

This is the subtle one. The cron job runs, but is it actually doing anything? I write a timestamp file every time an article successfully publishes. The health check compares that timestamp to "now." If it's been more than 3 days, I get an alert.

This is what would have caught the silent 429 errors. The script was running, but not publishing. The timestamp wouldn't update. I'd know.

5. Router Reachability (Bonus)

I added a check that pings the gateway router itself. If the whole network is down, I get a different alert. This happened once during a power outage — the router rebooted faster than the machines, but DHCP reassigned IPs. I knew the network was wonky before I even tried to query anything.

What I Don't Monitor (On Purpose)

I'm not running a datacenter. There are things I deliberately don't check:

• GPU temperature. My RTX 3060 has a perfectly fine stock cooler. If it throttles, I'll notice in response times. Adding temp monitoring means sensors, drivers, more code. Not worth it.
• Network bandwidth. I'm not streaming video. Local network latency is never the bottleneck.
• CPU load. The Mac Mini sits at 15% most of the time. If it spikes, I don't care unless it's sustained — and then Ollama response times will tell me.
• Log aggregation. I read logs when something breaks. I don't need them shipped to Elasticsearch.

The rule: if a failure mode would be annoying to debug, monitor it. If it's just "nice to know," skip it.

How I Run It

The script runs as a systemd timer on the Mac Mini. Every 10 minutes, checks fire. Alerts go to Telegram. If everything is fine, nothing happens. Silent success is the goal.

# /etc/systemd/system/ai-health-check.timer
[Unit]
Description=AI Agent Health Check

[Timer]
OnCalendar=*:0/10
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/ai-health-check.service
[Unit]
Description=Run AI health check

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /home/sam/health_check.py
User=sam

sudo systemctl enable ai-health-check.timer
sudo systemctl start ai-health-check.timer

Takes 30 seconds to set up. Runs forever.

What Changed (The Honest Version)

I fix things before they hurt. The Windows PC issue was fixed within 10 minutes of the alert. Before monitoring, it would have been "why is this query so slow today?" followed by 20 minutes of ssh and log reading.

I trust the system more. There's a difference between "I think it's working" and "I know it's working because the monitor is green." I can focus on building instead of worrying.

But I also get more alerts than I'd like. The Ubuntu box is on WiFi and occasionally drops for 30 seconds. I get a "unreachable" alert, then 10 minutes later it's fine. I haven't tuned the thresholds yet because... honestly, it's not annoying enough to fix.

False positives exist. Once, the model check failed because Ollama was still loading after a restart. The model existed, but /api/tags returned an empty list for 15 seconds. I added a 5-second retry and it went away.

When This Matters (And When It Doesn't)

Do this if:

• You have automated processes running unsupervised
• You've had a "wait, when did that break?" moment
• You use Telegram (or Slack, or email) anyway
• You value knowing about problems over perfect metrics

Don't do this if:

• You're still debugging your setup manually every day (fix the core issue first)
• You want beautiful dashboards (use Prometheus + Grafana instead)
• You enjoy the thrill of surprise failures

The Real Lesson

The best monitoring isn't the one that tells you everything. It's the one that tells you the thing you actually need to know, at the time you can still do something about it.

My 50-line script isn't impressive. It won't get me DevOps cred. But it caught three real issues in the first month, and it cost me nothing but an afternoon.

If you're running AI agents at home and you don't know when they break... you just don't know when they break. Fix that first. Everything else is optimization.

---

Sam Hartley is a solo dev running a monitored 3-machine AI home lab. Writes about the boring infrastructure that makes local AI actually reliable.

→ Custom automation setups on Fiverr
→ Follow CelebiBots on Telegram

ai #agents #monitoring #devops #selfhosted #homelab #buildinginpublic