There's one command on one box that can reach, inspect, and operate every other
machine in my homelab. It's an AI-driven CLI sitting at the top of the rack — think
of it as the estate's command deck — and underneath it is the least glamorous
automation stack I could build: secrets in a single vault, a pile of plain Python
scripts, and cron. No visual workflow canvas, no drag-and-drop nodes, no clever
event bus. I had the fancy version. I tore it out. This is why, and what the boring
replacement looks like.
The command deck: one entry point for the whole estate
Every box — the hypervisor nodes, the NAS, the low-power model host, the
Pis — used to mean its own SSH session, its own quirks, its own way in. The thing
that changed my life was putting one AI CLI at the top that knows how to reach
all of them and speaks plain English on the way in.
"What's low on disk?" "Which container didn't come back after the reboot?" "Show
me what changed on the proxy last night." I ask the deck; it runs the real
commands against the real boxes and answers. It's not magic and it's not
autonomous — it's a single, uniform front door to an estate that used to be a
dozen different doors with a dozen different keys.
Two design rules make it safe enough to trust with that reach:
- It reads freely; it acts narrowly. Inspecting state — logs, disk, service status, config — is unrestricted. Changing things runs through a small set of defined, reviewable actions, not "the model decided to run this." The blast radius of a bad idea is bounded by what tools exist, not by what the model can dream up.
- It's a helper, never a dependency. If the deck is down, every box keeps running exactly as before. Nothing in the critical path calls it. It sits above the estate, not inside it.
Secrets: one vault, or the whole thing is a lie
You cannot give one CLI the keys to every box and also leave those keys
scattered across a dozen config files. The command deck only works because of the
thing underneath it: every credential in the estate lives in one secrets
manager, and nothing else holds a copy on disk.
A box authenticates as a scoped machine identity, pulls exactly the secrets it
needs at runtime, and holds nothing persistent. The payoff is the question I can
finally answer: "where does this credential live, and what uses it?" — one
place, one answer. Rotating anything is a single change. The CLI at the top doesn't
hoard a master keyring; it, too, resolves what it needs, when it needs it, from the
one vault.
The habit that comes with it: verify a secret by its fingerprint, never by
printing it. The instant you echo a key to check it, you've made a new copy in
a scrollback the whole point was to avoid. Compare hashes; never look at the value.
Centralised access needs centralised secrets. Do one without the other and you've
just built a very convenient way to leak everything at once.
Why I ripped out the no-code tool
Here's the part people push back on. I used to run a chunk of my automation in a
popular visual, no-code workflow tool — the drag-boxes-and-connect-them kind. It's
genuinely good software. I replaced almost all of it with plain Python scripts run
by cron, and the estate got more reliable, not less. Three reasons, each one
learned the hard way.
1. The no-code tool hid failures; Python and cron make them loud.
My worst automation disaster ran in that visual tool: a set of workflows that
reported success on Discord while doing nothing at all, for months, because a node
was set to "continue on error" and the happy-path message fired regardless. The
canvas looked healthy — green everywhere — while the actual work failed silently.
When I rebuilt the same jobs as plain Python, a failure is an exception, an
exception is a non-zero exit, and a non-zero exit is something cron and my
monitoring can see. The boring stack can't paint itself green while lying.
2. Text diffs; a canvas doesn't.
A Python script is a file. It lives in git, it diffs, it code-reviews, I can see
exactly what changed and roll it back. A visual workflow is a blob of JSON
describing boxes and wires — technically versionable, practically unreadable in a
diff. When something breaks after a change, "what changed?" has a real answer with
files and an unanswerable shrug with the canvas.
3. Determinism, and keeping the model out of the decision.
This is the big one, and it's the rule the whole estate runs on: scripts gather
facts; models never do. My monitoring jobs are deterministic Python under cron
— they query the real backup timestamp, the real disk figure, the real
service status, and write those numbers down. Only then, if a human-readable
summary is wanted, does the AI layer get handed those already-gathered facts to
phrase. The clever, non-deterministic thing is allowed to word a report; it is
never the thing that produces the number. A visual tool with an LLM node in the
middle of the flow blurs exactly that line — and I've watched a model in that seat
invent a fact and present it with total confidence. cron + Python keeps the
gathering dumb, reproducible, and honest, and keeps the model firmly in the writer's
chair.
The tools that actually keep it running
Under the command deck, the things that keep the estate alive are all boring on
purpose:
- Deterministic monitors on timers. Each critical job — backups, delivery, patch drift — has a plain script that checks the true state and alerts on absence, not just on errors. A run that doesn't happen pages me; silence is never assumed to be health. (I test that by deliberately breaking each one and watching the alert arrive. One had a 24-hour blind spot I'd never have found otherwise.)
- A self-healing pass for the known-boring failures. The stuff that breaks the same way every time — a container that didn't restart, a stale lock after a reboot — gets a small, deterministic remediation, not an LLM improvising on production.
- One dashboard as the family-facing front door, so the humans in the house have a page of green dots and I have an at-a-glance exception list.
None of it is impressive to look at. That's the whole point. The impressive part —
the AI CLI you can talk to — sits on top of a foundation deliberately built to be
dumb, loud, and reproducible.
The shape, if you want to copy it
- Put one AI CLI at the top as a uniform front door to every box — reading freely, acting only through defined, reviewable tools.
- Centralise every secret in one vault, pulled at runtime by scoped machine identities, fingerprinted never printed. Central access demands central secrets.
-
Prefer plain Python on
cron/timers over a no-code canvas for anything that matters: it fails loudly, it diffs in git, and it keeps the model out of the decision. - Gather with scripts, phrase with the model. The deterministic layer produces every fact; the AI only ever words it.
- Alert on absence and test the alarms by breaking them on purpose.
The counterintuitive lesson is that the more capable the thing at the top gets — a
CLI you can talk to that can reach your whole estate — the more boring and
deterministic everything underneath it needs to be. The cleverness is safe only
because the foundation is dumb, loud, and honest. I traded a pretty canvas for a
pile of Python and a crontab, and it's the most reliable my homelab has ever
been.
Notes from running a small estate from a single AI command deck sat on a boring
foundation of one vault, plain Python, and cron. The glamour is at the top; the
reliability is entirely at the bottom.
🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.
Top comments (2)
nice setup. i would still keep the secrets store reachable only over the tunnel and verify after reboot that nothing rebound to 0.0.0.0. a quick ss -lntp plus a curl from a vpn peer catches that before it becomes an exposed api.
The two design rules are the interesting part — 'reads freely, acts narrowly' is the correct threat model, and bounding blast radius by which tools exist rather than what the model generates is the only version of agent safety that has ever held up in practice for me too.
One tension I'd be careful about: 'nothing else holds a copy on disk' plus a machine identity that pulls secrets at runtime still means whatever fetches them has access to them in memory, and your AI CLI is a process that assembles commands from those values. We hit this with a deck-shaped setup: the useful boundary wasn't vault-vs-disk but a separate, lower-trust identity for anything the CLI composes on the fly, distinct from the identity your deploy scripts use. Curious whether you've landed on that distinction or treat every consumer as equally trusted.