I Gave My Agent Ten Ops Skills, and a Docker Test for Each One

Ten skills, ten Docker tests, nine green and one skipped off Ubuntu. That count is the only claim in this article worth checking, and tests/run.sh in the repo prints it.

The Problem With a How-To an Agent Reads

A runbook written for a person leaves gaps a person fills from context. An agent fills the same gaps by guessing, and a guess about ufw --force enable locks you out of the box.

So each procedure here ships as a folder the agent loads whole:

skills/vps-basics/
  SKILL.md          YAML front matter, then numbered steps
  references/
    jail.local      the file step 3 copies

Front matter carries name, description, version, license and platforms. Claude Code reads that layout natively. Any AgentSkills-spec loader reads the same folders, though only Claude Code and my own runner have been tested against these.

The Test Runs the Skill's Own Steps

A skill is a document, and a document rots. The harness closes that gap by executing the document:

docker run --rm --privileged \
  -v "$skill_dir/references:/refs:ro" \
  ubuntu:22.04 bash -c '
    apt-get update -qq && apt-get install -y -qq ufw fail2ban
    ufw default deny incoming
    ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp
    ufw --force enable
    cp /refs/jail.local /etc/fail2ban/jail.local
    service fail2ban start
    ufw status | grep -q "Status: active"
    ufw status | grep -q "22/tcp"
    fail2ban-client status sshd | grep -q "Status for the jail: sshd"
  '

Those commands are the steps of skills/vps-basics/SKILL.md, copied out. Edit the SKILL.md without editing the test and the two disagree; edit the test without editing the SKILL.md and the assertions stop matching. Either way the harness goes red before a reader gets the wrong version.

tests/run.sh walks every tests/<name>.sh, skips a skill whose front matter says host-specific: true, and exits non-zero on the first failure. One skill is host-specific: wan-failover-uplink-guard needs two physical uplinks, and no container gives it those.

What the Ten Cover

Free and MIT, in the public repo:

• vps-basics: ufw default-deny plus a fail2ban sshd jail on a fresh box.
• backup-restore-drill: every backup step paired with a restore step, so a backup counts only after it has been read back.
• service-health-check: process, port and HTTP checked in that order.
• systemd-unit-authoring: a script turned into a unit that restarts on crash and starts on boot.

Paid once the checkout opens, which it has yet to do: caddy-reverse-proxy, tailscale-funnel, pi-hardening, telegram-bot-deployment, wan-failover-uplink-guard, wol-wake-and-remote-shutdown.

Why Three Layers in a Health Check

service-health-check runs process, then port, then HTTP, and stops at the first failure. Each layer catches what the layer above misses. A process check alone passes a hung process that still holds its port. A port check alone passes an app that accepts the connection and returns 500 to every request. Only the HTTP check reads what a user would read.

What This Does Not Prove

A container test proves the steps run on a clean Ubuntu 22.04. Running on your box, with your kernel and your existing firewall rules, is a separate claim, and no skill in the pack makes it.

Ten skills is also a small sample for a format claim. The layout may need changing once a third loader reads it.

Repo: https://github.com/automate-it-all/agent-skills-pack
Paid six: https://automate-it-all.win/skills/
Questions: [e-mail]
Who builds this: https://fidel-perez.github.io/portfolio/

Comments

[Arsen Apostolov]

The "the test runs the skill's own steps" part is the whole trick, and it's the part most runbook authors skip. A doc and its test are two copies of the same truth, and the second one always rots first. I run my own agent on skill sheets the same way, and the only thing that keeps them honest is a check that executes the sheet rather than re-reading it — otherwise you're trusting a document to describe itself. The three-layer health check (process → port → HTTP) is worth stealing too: a process check passes a hung process that still holds its port, and that's the failure I keep hitting.

[Automate It All]

The hung-process case is why service-health-check runs the three layers in order rather than together: process, then port, then HTTP. A hung process passes the process check and still holds its port, so only the HTTP layer catches it. Checking in order makes the first failure name the layer.

Making the test run the sheet cost one rewrite per skill. tests/vps-basics.sh boots ubuntu:22.04, follows SKILL.md line by line, then asserts that ufw status reads active with 22, 80 and 443 open and the fail2ban sshd jail is up. Anywhere SKILL.md said "configure the firewall" without naming the command, the test could not follow it, so the sentence had to name it.

Nine of ten pass on a full checkout. The tenth needs two physical uplinks, so it skips off a single-NIC box. If you run your own sheets this way, I would like to hear what your check asserts: [e-mail]