I Gave My Agent Ten Ops Skills, and a Docker Test for Each One

#devops #ai #homelab #testing

Ten skills, ten Docker tests, nine green and one skipped off Ubuntu. That count is the only claim in this article worth checking, and tests/run.sh in the repo prints it.

The Problem With a How-To an Agent Reads

A runbook written for a person leaves gaps a person fills from context. An agent fills the same gaps by guessing, and a guess about ufw --force enable locks you out of the box.

So each procedure here ships as a folder the agent loads whole:

skills/vps-basics/
  SKILL.md          YAML front matter, then numbered steps
  references/
    jail.local      the file step 3 copies

Front matter carries name, description, version, license and platforms. Claude Code reads that layout natively. Any AgentSkills-spec loader reads the same folders, though only Claude Code and my own runner have been tested against these.

The Test Runs the Skill's Own Steps

A skill is a document, and a document rots. The harness closes that gap by executing the document:

docker run --rm --privileged \
  -v "$skill_dir/references:/refs:ro" \
  ubuntu:22.04 bash -c '
    apt-get update -qq && apt-get install -y -qq ufw fail2ban
    ufw default deny incoming
    ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp
    ufw --force enable
    cp /refs/jail.local /etc/fail2ban/jail.local
    service fail2ban start
    ufw status | grep -q "Status: active"
    ufw status | grep -q "22/tcp"
    fail2ban-client status sshd | grep -q "Status for the jail: sshd"
  '

Those commands are the steps of skills/vps-basics/SKILL.md, copied out. Edit the SKILL.md without editing the test and the two disagree; edit the test without editing the SKILL.md and the assertions stop matching. Either way the harness goes red before a reader gets the wrong version.

tests/run.sh walks every tests/<name>.sh, skips a skill whose front matter says host-specific: true, and exits non-zero on the first failure. One skill is host-specific: wan-failover-uplink-guard needs two physical uplinks, and no container gives it those.

What the Ten Cover

Free and MIT, in the public repo:

vps-basics: ufw default-deny plus a fail2ban sshd jail on a fresh box.
backup-restore-drill: every backup step paired with a restore step, so a backup counts only after it has been read back.
service-health-check: process, port and HTTP checked in that order.
systemd-unit-authoring: a script turned into a unit that restarts on crash and starts on boot.

Paid once the checkout opens, which it has yet to do: caddy-reverse-proxy, tailscale-funnel, pi-hardening, telegram-bot-deployment, wan-failover-uplink-guard, wol-wake-and-remote-shutdown.

Why Three Layers in a Health Check

service-health-check runs process, then port, then HTTP, and stops at the first failure. Each layer catches what the layer above misses. A process check alone passes a hung process that still holds its port. A port check alone passes an app that accepts the connection and returns 500 to every request. Only the HTTP check reads what a user would read.

What This Does Not Prove

A container test proves the steps run on a clean Ubuntu 22.04. Running on your box, with your kernel and your existing firewall rules, is a separate claim, and no skill in the pack makes it.

Ten skills is also a small sample for a format claim. The layout may need changing once a third loader reads it.

Repo: https://github.com/automate-it-all/agent-skills-pack
Paid six: https://automate-it-all.win/skills/
Questions: skills@automate-it-all.win
Who builds this: https://fidel-perez.github.io/portfolio/

DEV Community