I Didn't Need a DAG. I Needed Cron.

Hey, nerds! Stop me if you've heard this one before.

Every homelab eventually hits the same wall. You have a pile of guests, a pile of small chores those guests need done on a schedule and absolutely no desire to SSH into each one by hand. Disk usage, uptime checks, arbitrary commands, pending updates, failed systemd units, the list goes on. This is all boring stuff that quietly rots away if nobody looks at it.

So I went shopping for an automation platform. I came back with nothing and then built one out of SSH, bash, cron and other bits and bobs already running in the homelab. It's called NetRunner, because I'm physically incapable of naming anything in this network without a cyberpunk reference.

Before we begin, I should take a moment to define what a DAG actually is. A Directed Acyclic Graph is a fancy name for a to-do list with dependencies. Each node in the graph is a task and each arrow says "this one waits for that one". Directed means the arrows only point one way. Acyclic means you can never follow them back to where you started, so you can't accidentally shoot yourself in the foot with a loop of tasks all waiting on each other forever.

The order comes from the arrows, not the order you wrote things down in. Say you take a backup, then verify it and copy it offsite, then clean up. Verify and copy both wait on the backup but not on each other, so they run side by side. The clean-up waits for both.

Let's go shopping!

I gave the usual suspects a fair go. Kestra, Windmill and Semaphore are all genuinely good software built by people who clearly know what they're doing. They're also built for teams. Teams that need a web UI, a worker model, RBAC and the plugin ecosystems. I'm just a weird little guy with a Proxmox cluster and a mild addiction to YAML.

Dagu came the closest. It's a single binary written in my favourite language, has jobs defined as YAML and no database to babysit. I liked it a lot and I really tried to force myself to use it. Then I sat down to write my first workflow and realised that not a single one of my jobs was actually a graph. They were all "run this thing on those boxes over yonder and read me the riot act if something failed".

Then, as I usually do in these moments, I took a step away from the desk and had a think. What is the bare minimum required to get this job done well?

The obvious answer was every piece of a central control node was already sitting in the network. Terraform already provisions the guests. Ansible already configures them. Forgejo already holds the code. Proxmox already knows every guest, its tags, its address AND comes packaged with an API. Apprise was also running in the homelab as a notification pipeline to a dedicated set of Slack channels. The only thing missing was a little bit of glue and a box to sit in the middle to press the button.

Boring almost always beats clever.

— Evergreen.

Big brain time!

NetRunner is one unprivileged LXC container with one core and a gig of memory. It has no web UI, no database and, with the exception of SSH, no open ports. It doesn't need Python or Ansible to run because the whole runtime is Bash, OpenSSH, curl, jq, flock and timeout. If you've administered a Linux box at any point in the last thirty years, you already know how every part of it works.

Here's an obligatory Mermaid chart with the basic workflow. We'll get to what nr is in a bit.

flowchart LR
  forge[Forgejo: jobs as code] --> ansible[Ansible playbook]
  ansible --> cron
  subgraph controller[NetRunner container]
    cron[cron] --> job[nr job run]
    job --> exec[nr exec]
  end
  pve[Proxmox API] --> exec
  exec -- "SSH, script on stdin" --> guests[LXC containers and VMs]
  job -- "on failure" --> slack[apprise to Slack]

The self-imposed design constraints were as follows:

• A job is a script that runs on the controller and only on the controller.
• The controller sends commands to the guests over SSH, collects their output and makes its decisions locally.
• The guests run no scheduler, no agent and no daemon. They keep no copy of any job.

That last point matters because the homelab runs on an immutable infrastructure policy, so a guest should look the same after a job as it did before. NetRunner pipes each command or script into the guest on standard input and the guest runs it with bash -s. Nothing lands on the guest's disk, so nothing needs cleaning up afterwards.

Obviously, there could be exceptions to that rule, but if I aim to write something ephemeral to a guest's disk, it goes in /tmp. If I need to write something permanent, I codify it in Ansible or Terraform. It's more of a guideline than a hard rule.

One key to rule them all... with great care.

Each guest gets a dedicated netrunner user. The LXC containers get it from a Proxmox hookscript on every start, which Terraform manages with the proxmox provider. New VMs get it from cloud-init on first boot.

That user's authorized_keys holds exactly one line:

from="10.0.0.xx",no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAA... netrunner

The key is an ED25519 pair that Terraform generates. The private half lives on the controller and nowhere else. The from= option is doing the real work here. Steal that private key, copy it to your laptop and try it from anywhere other than the controller's static address and every guest will politely tell you to get fucked. The key is only useful from one IP on the network.

I'm not going to pretend this is bulletproof, because it isn't. The netrunner user has passwordless sudo, since most of the chores I care about need root. That means the controller is effectively root on every guest. Own the container and you own the lab. It's the same trade every configuration management tool makes and I've made it with my eyes open.

The container is small, hardened, exposes nothing but SSH and has one job.

Finding the guests.

Hard-coding an inventory is how you end up running jobs against a guest you decommissioned weeks ago. So, to save me the trouble, every five minutes, nr refresh asks the Proxmox API for the running guests and caches the name, VM ID, node, type, tags and address of each one on disk.

The fun bit is that when you execute a job it never names a host. It selects guests with filters:

nr exec -t music -- uptime
nr exec -T lxc -P 8 -- 'apt list --upgradable 2>/dev/null | tail -n +2 | wc -l'
nr exec -a web -a dns -e THRESHOLD=70 -s filesystems.sh

Filter by Proxmox tag, by guest type or by a regex on the name. The tags come from Terraform. So when I tag a guest in its Terraform stack, I'm also quietly subscribing it to every job that selects that tag. A new guest needs zero changes on the controller. The hookscript or cloud-init gives it the account, the next refresh picks it up and it starts getting nagged on schedule like everybody else. This is the type of set-and-forget project I love to build.

The nr helper.

Everything on the controller goes through one command: nr. It's a bash CLI built with Bashly, which takes a YAML description of your commands, flags and help text and spits out a single, surprisingly well-designed bash script. If you've been reading this blog for a while ( you haven't ) you'd know that I've been using this utility for years.

Yes, there's YAML in the bash. There's always YAML. I've made my peace with it and so should you.

Command	What it does
nr exec	Runs a command or a script on every matching guest and collects the results.
nr guests	Lists the cached guests, with the same filters.
nr tags	Lists the Proxmox tags and the guests behind each one.
nr ssh	Opens a session to one guest by name.
nr notify	Sends a notification through apprise.
nr job run	Runs one job, exactly as cron would.
nr refresh	Rebuilds the guest cache and prunes old runs.

nr exec is the workhorse. It fans out to four guests at a time by default and writes <guest>.out, <guest>.err and <guest>.rc for each one, plus a summary.json that ties them together. A job script is then just ordinary bash and jq over a directory of files. No SDK, no bespoke expression language and definitely no plugin API I have to learn and quickly forget.

Secrets are covered as best I can for this kind of setup. nr exec --pass SECRET_SQUIRREL_TOKEN writes the variable into the stdin stream as an export line. This means a passed secret never shows up on a command line on the guest or its shell history.

It's also just a nice way to poke at the lab.

The scheduled jobs are the reason NetRunner exists, but the thing I didn't expect is how often I now reach for nr from a shell with no job in sight. The controller already knows every guest, so it's become the place I go when I have a question about the fleet and can't be bothered remembering an IP.

Some things I've typed within minutes of finally getting this running, more or less verbatim:

# Which guests are running and what are they tagged with?
nr tags

# Kernel versions across every LXC processing eight at a time.
nr exec -T lxc -P 8 -- uname -r

# Which VMs have a reboot pending?
nr exec -T qemu -- 'test -f /var/run/reboot-required && echo yes || echo no'

# Is docker happy on the media guests? Skip the music server.
nr exec -t media -x navidrome -- systemctl is-active docker

# Just get me a shell on that thing.
nr ssh koito

Obviously, I can easily eyeball a few of these from the Proxmox web UI, but we're not in a browser are we?

Every output line is prefixed with the guest name and each one is displayed as a single block, so a fleet-wide df -h is readable instead of a wall of text. At the end you get a list of anything that failed and the path to the results directory in case you want to go digging.

All results are stored locally on the controller as plain'ole files. This means composing stuff with the rest of the toolbox is fairly trivial. --json writes the summary.json to standard output and --quiet suppresses the guest output when you only care about the result:

# Name every VM guest where the command failed.
nr exec -j -T qemu -- hostname | jq -r '.[] | select(.rc != 0) | .name'

# Collect every os-release into a known directory, then read one.
nr exec -q -o /tmp/os -- 'cat /etc/os-release'
cat /tmp/os/koito.out

And because nr notify reads standard input, a one-off report is a pipe away:

nr exec -t music -- uptime | nr notify -T "Uptime of the music guests"

Here it is in the #netrunner channel moments later:

A preview of the resulting Slack notification.

Those ad-hoc runs land in /var/lib/netrunner/runs/adhoc/ and get pruned after a fortnight, same as everything else. This is also exactly how I test a new job before it gets a schedule. Run the commands by hand until they do the right thing, then move them into a script and add the YAML. There's no "run locally" mode to emulate because the shell is the local mode.

Jobs are code. Obviously.

Every job lives in the forge next to everything else. A job is two things: a script in jobs/bin/ and an entry in jobs/jobs.yaml.

- name: disk-usage
  description: Fail when a filesystem on a guest is at or above 85%.
  schedule: "0 * * * *"
  script: disk-usage
  timeout: 600
  exclude:
    - truenas

Info: You might notice I'm excluding TrueNAS here. It's not a standard guest, but a full-blown dedicated distribution that manages itself well enough. No need for my grubby scripts to taint all my legally-acquired ISOs.

The job script for something like "tell me which guests have failed systemd units" is about twenty lines:

#!/usr/bin/env bash
set -euo pipefail

results="${NR_RUN_DIR}/units"

# nr exec exits 1 when a guest fails. The script reads each result itself.
nr exec -q -P 8 -o "${results}" -- 'systemctl --failed --no-legend --plain | wc -l' || true

status=0
while IFS=$'\t' read -r name rc out; do
  if [ "${rc}" != "0" ]; then
    echo "UNREACHABLE ${name}"
    status=1
  elif [ "$(cat "${out}")" -gt 0 ]; then
    echo "FAILED UNITS ${name}: $(cat "${out}")"
    status=1
  fi
done < <(jq -r '.[] | [.name, (.rc // "skip" | tostring), .out] | @tsv' "${results}/summary.json")

exit "${status}"

Don't judge me, all Bash scripting looks ugly. Anyway, as is standard in Linux, you print whatever you want and exit non-zero when something's wrong. That's effectively the API in its entirety.

Ansible does the boring part.

I never touch the controller by hand. The playbook builds nr with Bashly, checks the result with bash -n before it swaps the old binary out and then reads jobs.yaml. It validates every job before it changes anything through an assertion and refuses to continue if any aspect of a job fails:

 ... more yaml up here ...
  tasks:
    - name: assert each job is valid
      ansible.builtin.assert:
        that:
          - item.name is defined and item.name is match('^[a-z0-9][a-z0-9-]*$')
          - item.schedule is defined and (item.schedule | split | length) == 5
          - item.script is defined
          - (netrunner_jobs_src ~ '/bin/' ~ item.script) is file
          - (item.env | default({}) | dict2items | map(attribute='key') | reject('match', '^[A-Za-z_][A-Za-z0-9_]*$') | list | length) == 0
          - item.notify | default('failure') in ['failure', 'always', 'never']
          - item.exclude | default([]) is sequence and item.exclude | default([]) is not string
          - (item.exclude | default([]) | reject('match', '^[A-Za-z0-9][A-Za-z0-9.-]*$') | list | length) == 0
        fail_msg: "Job {{ item.name | default('(no name)') }} in jobs/jobs.yaml is not valid. See the field list at the top of that file."
        quiet: true
      loop: "{{ jobs }}"
      loop_control:
        label: "{{ item.name | default('(no name)') }}"
  ... more yaml down here ...

You'll notice {{ jobs }} near the bottom. That's a variable defined by yet another YAML file containing a list of jobs formatted like the example at the start of prior section.

If the jobs pass, it writes two files on the controller. The first is jobs.json, with secrets resolved from my password manager at deploy time. The second is /etc/cron.d/netrunner, with one line per enabled job. Each line runs nr job run <name> as the netrunner user.

It's a stupid-simple template using that same {{ jobs }} variable passed through. This is what we use to create the resulting cron table:

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Keep the guest cache fresh for the helpers and the jobs.
*/5 * * * * {{ netrunner_user }} systemd-cat -t netrunner-refresh /usr/local/bin/nr refresh --quiet
{% for job in jobs if job.enabled | default(true) | bool %}

# {{ job.name }}: {{ job.description | default('no description') }}
{{ job.schedule }} {{ netrunner_user }} systemd-cat -t netrunner-{{ job.name }} /usr/local/bin/nr job run {{ job.name }}
{% endfor %}

So the entire workflow for a new automation is:

1. Write a script.
2. Add a few lines of YAML.
3. Push to the forge.
4. Run the playbook.

Want to pause a job? Set enabled: false and run the playbook. Want to drop a job? Delete it and run the playbook, which also removes the script from the container. If I want to see who fat-fingered the wrong thing, I have the git log to stare at. git blame tells me exactly who broke the thing aaaaand, oh wait, that's right, it was me.

Cron, but with a seatbelt.

Raw cron has two classic ways to ruin your weekend. Jobs pile up on top of each other when one runs long and jobs hang forever when a guest goes quiet. nr job run handles both with tools that ship with every Linux box on the planet.

flock makes sure a second run of a job won't start while the first one is still going. timeout kills a run once it goes over its budget, which defaults to an hour. Each run gets its own directory under /var/lib/netrunner/runs/ with its log, its exit code and the raw results from every guest. Output goes to the journal tagged netrunner-<name>, so reading a job's history is just journalctl:

journalctl -t netrunner-disk-usage --since today

Again, all tools that have existed forever, with the addition of things that already do real work in the homelab.

When a job fails, nr grabs the last forty lines of output and posts them to apprise, which drops them into a #netrunner Slack channel. That's the entirety of the alerting stack and it's currently more than adequate for my needs.

There actually are some ugly bits to this.

Here's what NetRunner doesn't do, in the interest of not pretending otherwise and I'm almost certain it's the part that most everyone reading this has been looking forward to:

• There's no failover. A single LXC means a single controller. If it dies, the jobs stop and the guests carry on regardless. The container has no backup either, because it doesn't need one. The jobs live in the forge, the key lives in Terraform state and one terraform apply plus one playbook run rebuilds the whole thing. If it goes down for whatever reason, I have a reachability check from Uptime Kuma that'll alert me in Slack as well.
• It doesn't verify host keys. Guests get recreated and DHCP addresses tend to wander, so strict host key checking would mostly produce noise. The cost is that a hostile box on the LAN that claims a guest's address could receive a job, secrets and all. It doesn't even need the key. The controller is the one doing the authenticating, not the guest. I accept that on my own network. I wouldn't anywhere else. I sure as shit wouldn't at work.
• The cache can be five minutes stale. If an address moves between guests inside that window, a job can land on the wrong box. It's a small window and it's documented. I'd get an alert for this as well, so no biggie. Well, it could potentially be a very BIG biggie depending on what I'm running...
• There's no UI. I consider this a feature, but I appreciate not everyone will.

None of these would survive a production review at work. That's fine. This isn't work, and every one of these trade-offs is written down in the homelab's wiki next to the reason I made it.

In closing ...

The platforms I looked at are not bad software. They solve problems that I simply don't have: many users, complex dependency graphs, audit requirements and a web UI for people who don't live in a terminal. I have a handful of guests and a list of chores.

NetRunner is some well structured bash, one SSH key, one cron file and a playbook. There's nothing to upgrade except bash and OpenSSH, and those were never going anywhere. Every part of it is something I already knew how to debug at two in the morning, which turns out to be the only feature that really matters.

I guess the whole point of this post is you don't always need to resort to using a super shiny off-the-shelf solution when you'll only use 5% of its functionality. Especially, when you already have everything you need, but the glue. The rule of YAGNI ( You Aren't Gonna Need It ) applies and prevails. At any rate, this was a fun weekend project!

The nr code is unique to my homelab and I don't plan on publishing it as it literally will not run anywhere else, but I'm flexible. If there's genuine interest, I'll post the source as well as the playbook so you can tinker with it youself.

Wake up, samurai. We have guests to patch.