DEV Community

Zido
Zido

Posted on

vigil: the watchdog that tells you NOT to wake anyone up

#devchallenge #hf26challenge #ai #devops

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission 💸

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

Tags: #devchallenge #hf26challenge

What I Built

vigil is an agent-powered watchdog for small VPS boxes: dumb checks, smart diagnosis.

Most watchdogs tell you something is down. Then you SSH in at 2 AM, dig through logs for 20 minutes, and discover the service is fine — your watchdog was wrong. vigil closes that gap: when a check fails, it wakes a local AI agent (running an open-weight model on the same box) which investigates with real tools — journalctl, systemctl, ss, df — reads your actual logs, and sends you a report with STATE / EVIDENCE / ACTION / CONFIDENCE.

Here is a real report from the box vigil was built on, sent to my Telegram at 1:46 AM:

VIGIL — http://127.0.0.1:8764/hub/status (HTTP Error 401: Unauthorized)

STATE: healthy — 401 is expected, service is up and auth working as designed
EVIDENCE: systemctl is-active zidorouter = active; ss shows python3 pid 21858
  listening on 127.0.0.1:8764; unauthenticated GET /hub/status = 401 while
  POST /login returns 303 (auth flow responds, bad password rejected) — no
  errors in journalctl other than harmless connection-reset log noise
ACTION: fix the watchdog check, not the service: POST /login first and keep
  the session cookie, or point the probe at an unauthenticated endpoint
CONFIDENCE: high
Enter fullscreen mode Exit fullscreen mode

A threshold watchdog would have paged "SERVICE DOWN" for that. vigil's agent read the journal, verified the auth flow was working, and told me the alert itself was wrong. That report arrived minutes after vigil first went live — and it diagnosed its own misconfiguration.

Who would pay for it

The customer is whoever is the entire ops department of their own server: freelancers running a $5–20/mo VPS for their projects, indie SaaS founders, homelab folks. Today they have two options:

  1. No watchdog — they find out from a client or a user, hours late.
  2. A cloud monitor — $10–30/mo per box, alerts them "CPU high" or "HTTP 5xx", and still leaves them to do the 2 AM log archaeology themselves.

vigil replaces the diagnosis step, which is the expensive one. The checks are free (plain stdlib), the model is free (open-weight, on the box), and every report tells you whether to act or go back to sleep. I'd price it as a one-time purchase (~$15) or donation-ware for the homelab crowd — the market is everyone who owns exactly one server and refuses to buy Datadog for it.

I'm user #1: vigil runs as a systemd service on my production VPS right now, watching my gateway, my LLM router, and my tunnels.

Demo

Live on my box since October 14, 01:26 (timestamps from systemctl --user status vigil):

  • Setup → first real alert in ~20 minutes. On its very first pass, vigil flagged two checks and the agent diagnosed both as watchdog bugs, not service failures (a systemd scope mismatch — checking system units with systemctl when the service is user-level — and a 401 that was actually auth working as designed).
  • Cooldown works. After each alert diagnoses once, the check goes quiet (per-check cooldown, 15 min default). Overnight, 8+ hours of continuous operation, zero false pages. A quiet night is the product working.
  • E2E diagnosis latency: ~30–90 s per failure on a 4-core VPS, measured on repeated runs (one-shot agent call with tool access).

The box itself: github.com/ZidoCode/vigil — the code, MIT licensed. Setup is three commands (clone, config, systemctl --user enable --now vigil).

How I Built It

  • Checks: ~230 lines of pure Python stdlib. No dependencies. systemd unit states, HTTP probes, TCP listeners, disk/memory thresholds. Edge-triggered alerts with per-check cooldown, so a failing check diagnoses once instead of storming.
  • Diagnosis: Hermes Agent one-shot call (hermes -z) with tool access, pointed at an open-weight model served by a self-hosted gateway on the same box (Qwen3.8-27B via my own router). The agent gets the failing check + raw failure detail, then investigates on its own: reads journals, checks listeners, verifies service state. I don't script the investigation — the model decides which evidence to gather.
  • Delivery: any command that reads stdin, e.g. hermes send --to telegram -f -. Delivery failure falls back to stdout so reports are never lost.
  • What didn't work (honest list): my first Telegram delivery attempt (hermes send -s) failed — no such flag, had to switch to stdin piping. The first systemd check fired a false alert because it probed system-scope units while the service is user-level — the agent caught this in its very first diagnosis, which is both a bug story and the best demo of the product. And the model once took ~90 s on a busy box: diagnosis latency is model-bound, vigil is a watchdog, not an autohealer.

Why open innovation matters for what I built

The diagnosis step is the whole product, and it only works because the model runs locally:

  • The business math only works with open weights. A watchdog that calls a paid API on every hiccup has a cost curve that fights its own value proposition — you'd hesitate to let it diagnose anything. With an open-weight model on the box, diagnosis is free, so vigil can investigate every transient blip instead of rationing them.
  • Privacy is structural, not a policy. Diagnosing a failure means reading journals and process tables. With a cloud API, that evidence ships to a third party on every alert. Here it never crosses the network — the model that reads your logs runs on the machine that wrote them.
  • No vendor lock on the brain. provider/model are config keys. Any open-weight model your gateway serves works — llama.cpp, vLLM, a self-hosted router. The most opinionated component in the loop is swappable in one line.

What's next

  • Fix the two watchdog bugs its own agent found (scope-aware systemd checks, auth-aware HTTP probes) — filed as #1 and #2, fixes in progress.
  • A cases.jsonl log of every diagnosis, so vigil can show you its own track record.
  • If there's demand: a simple status page generated from diagnosis history.

Repo: https://github.com/ZidoCode/vigil — MIT, ~230 lines of stdlib, works with any OpenAI-compatible endpoint.

Built in ~20 hours during Hacktoberfest Week 2, deployed before it was announced, running in production since minute one.

Top comments (0)