Build and Test an AI Agent Skill with SKILL.md and Python

AI agents are good at interpreting intent, but they are not a reliable place to hide every rule in a workflow. If an agent must count characters, parse a file, or refuse to overwrite an existing artifact, prose instructions alone make the result harder to verify.

An Agent Skill gives that workflow a reusable home. The skill describes when it should activate and how the agent should reason. Small scripts handle repeatable checks. Tests protect the behavior when the skill changes.

This tutorial builds a small commit-crafter skill from the public how-to-create-a-skill-tutorial repository. The project is MIT licensed and documents the open Agent Skills specification. The repository's stable documentation describes the same folder structure and examples used here.

TL;DR

Create a folder with a SKILL.md, put deterministic validation in scripts/, keep long references in references/, and test the script independently. The model chooses and explains; the script verifies and computes.

Prerequisites

You need:

• Python 3.9 or newer
• An agent that discovers skills from a supported skills directory
• A Git repository with staged changes if you want to try the commit workflow

The example repository tests its skills with Python's standard library. You do not need an API key, a hosted model, or a third-party Python package for the minimal path.

1. Create the skill structure

The specification requires SKILL.md. The other directories are conventions that keep a skill maintainable:

commit-crafter/
|-- SKILL.md
|-- scripts/
|   `-- check_message.py
`-- references/
    `-- conventional-commits.md

Create a project-local skill directory on macOS or Linux:

mkdir -p .agents/skills/commit-crafter/scripts
mkdir -p .agents/skills/commit-crafter/references

On Windows PowerShell, use the equivalent commands:

New-Item -ItemType Directory -Force .agents\skills\commit-crafter\scripts
New-Item -ItemType Directory -Force .agents\skills\commit-crafter\references

The project-local location is useful when the skill belongs to one repository. A user-level location such as ~/.agents/skills/commit-crafter is better when you want the same skill available across projects. Check your agent's discovery rules before installing it globally.

2. Write a triggerable SKILL.md

The front matter is not decoration. The agent may read only the description while deciding whether to activate the skill, so describe both the job and the phrases that should trigger it.

---
name: commit-crafter
description: Write git commit messages from staged changes following Conventional Commits. Use when the user asks to commit, asks for a commit message, wants to fix or improve a commit message, or mentions conventional commits.
license: MIT
compatibility: Requires git and Python 3.9+
---

# Commit Crafter

Write a Conventional Commit message from the staged diff. The user reviews
and approves the message before anything is committed.

## Workflow

1. Run `git status` and `git diff --staged`.
2. Classify the change and whether it is breaking.
3. Draft the message in the format `type(scope): imperative summary`.
4. Validate it with `python scripts/check_message.py --file message.txt`.
5. Show the result. Commit only after explicit approval.

## Rules

- Never stage files yourself.
- Never commit secrets or unrelated files.
- Never add an AI attribution footer unless requested.

Notice the boundary around git commit. The skill can prepare and validate a message without silently changing repository history. That is a better default for an agent workflow that may be invoked from an ambiguous prompt.

The name must match the parent directory and use lowercase letters, numbers, and single hyphens. Keep the body short enough to load comfortably. Move detailed rules and examples into references/ and link to them one level deep.

3. Move stable rules into a deterministic script

The tutorial's central design rule is simple: let the model decide, let the script verify, and let the script crunch. A model can choose whether a change is a feature or a fix. A Python script can enforce the subject format every time.

The repository's check_message.py validates the Conventional Commits shape, allowed types, subject length, imperative mood warnings, blank-line separation, and breaking-change footers. It uses exit codes as a protocol:

• 0 means the input is valid
• 1 means validation found a rule violation
• 2 means there was no input

Run it against a candidate message:

python scripts/check_message.py --file message.txt

The script prints actionable violations to stderr. For example, it can report that a subject is too long or that a breaking marker lacks a BREAKING CHANGE: footer. The agent can read that output, revise the draft, and run the same check again.

That feedback loop is more robust than adding another paragraph of instructions. It also makes the rule testable without an agent in the loop.

4. Test the script before installing the skill

The example skill includes tests beside the script. Run them from the repository root:

python examples/commit-crafter/tests/test_check_message.py

The public repository's current test run completed successfully for the example suites. The commit-crafter suite covers valid messages, unknown types, subject limits, breaking changes, body formatting, and warning behavior.

Add a regression test whenever a real prompt exposes a failure. A useful test is not a copy of one successful output. It is a small input that would have caused the agent to make the wrong decision before the fix.

5. Try the skill with realistic prompts

Trigger quality is part of the implementation. Test both positive and negative cases:

"commit this"                      -> should trigger
"write a good commit message"      -> should trigger
"reword my last commit"            -> should trigger
"what does the staged diff do?"     -> should not trigger
"create a git branch"               -> should not trigger

If the skill fails to activate, improve the description with the user's vocabulary. If it activates for unrelated requests, narrow the wording. This is a practical evaluation of the trigger, not a promise that every agent implements discovery identically.

When the skill does activate, inspect whether it follows the important boundaries: it should inspect staged changes, never stage files on its own, run the validator, and wait for approval before committing.

Why this structure works

SKILL.md is the workflow contract. scripts/ is the deterministic execution layer. references/ is progressive disclosure for material that would make the main instructions noisy. assets/ can hold templates or fixtures when a skill needs them.

This separation also helps with token use. The agent does not need to reconstruct a parser from prose or load every example before starting. It can read the short workflow, run a local script, and consult a reference only when the situation requires it.

The repository extends the same pattern with changelog-forge, lessons-keeper, and skill-starter. Those examples demonstrate deterministic Git parsing, idempotent memory writes, and a copy-paste scaffold.

Failure modes and security boundaries

A skill is executable code with the permissions of the agent. Read scripts before installing a third-party skill. Treat network calls, credential access, shell pipelines, and automatic writes as explicit capabilities that need a reason and a documented boundary.

Common problems include:

• A vague description that never triggers, or triggers for unrelated requests.
• Instructions that claim a validation rule but have no executable check.
• A script that appends repeatedly when retried instead of being idempotent.
• Unix-only path assumptions in a skill advertised for Windows.
• A skill that commits, publishes, or reads secrets without an explicit user decision.

The tutorial's examples prefer Python's standard library, bounded output, explicit exit codes, and local execution. These choices reduce dependencies, but they do not make an arbitrary skill automatically safe. Review the code and its compatibility requirements before installation.

FAQ

Is a skill the same as an MCP server?

No. A skill is an instruction and resource bundle discovered by an agent. MCP is a protocol for exposing tools and data. A skill can tell an agent when to use an MCP tool, but the two solve different problems.

Should every skill include a script?

No. Use a script when a stable rule benefits from exact, repeatable behavior. A small explanatory workflow may only need SKILL.md and a reference file.

Can the skill guarantee the agent will follow it?

No. It can improve activation, make important rules explicit, and provide executable checks. The host agent still controls discovery and execution.

Where should I install it?

Use a project-local .agents/skills/ path for repository-specific behavior. Use a user-level path only after reviewing the skill and confirming how your agent discovers global skills.

Takeaway

An effective Agent Skill is a small, reviewable software project: a precise trigger, a short workflow, deterministic scripts, references loaded on demand, and tests that reproduce failures. Start with one narrow task such as commit-message drafting, then add automation only where the boundary and verification are clear.

What is one repetitive agent workflow in your project that would benefit from a deterministic check before the agent is allowed to continue?

AI assistance disclosure: This tutorial was researched and drafted with AI assistance. Repository behavior, commands, version information, license, and test claims were checked against the public project sources before publication.