DEV Community

Cover image for Stop Vibe-Checking Your AI Agents: The Complete Guide to the SKILL.md Lifecycle
zerocool
zerocool

Posted on Originally published at Medium

Stop Vibe-Checking Your AI Agents: The Complete Guide to the SKILL.md Lifecycle

#ai #softwareengineering #development #githubcopilot

Let’s be honest about how most of us build AI skills today.

We sit down, write a few hundred lines of system prompts or instructions, run them once or twice in our local terminal, see that it works, and hit merge. We call it a day. There are no automated tests, no regression suites, and no validation gates.

We pull changes based entirely on a vibe check.

But here is the catch: AI skills are no longer just toys generating poetic text. In modern agentic workflows, your skills are triggering production deployments, drafting legal contracts, bootstrapping infrastructure, and communicating directly with internal databases via your APIs.

When a traditional software function breaks, you get a noisy stack trace. When an AI skill breaks—and it will break the second the underlying model receives a silent upstream update—it fails in absolute silence. It doesn’t crash. It simply hallucinates an answer with unearned confidence, directly to a user who trusts it.

If your AI skills have the power of code, you need to start treating them like code. That means moving from vibes to engineering: design, validation, evaluation, and strict retirement. Let’s take a journey through what it actually takes to run a professional AI skill pipeline.

What an AI Skill Actually Is Under the Hood

Stripped of the marketing hype, an AI skill is remarkably simple: it’s a folder.

At the root of this folder sits a single file: SKILL.md. This file contains a YAML header (known as the frontmatter) followed by a body written in standard Markdown.

  • The frontmatter tells the agent when to use the skill.
  • The body tells the agent how to do the work. 
---
name: setup_nextjs_app
description: Use this skill when the user wants to bootstrap a new Next.js project with Tailwind CSS.
allowed_tools: [bash, file_writer]
---
# Setup NextJS App
Follow these steps to safely initialize the repository...
Enter fullscreen mode Exit fullscreen mode

SKILLS.md lifecycle
This format is a rapidly emerging open standard. Born out of research at Anthropic and open-sourced at agentskills.io, it has been rapidly adopted across the ecosystem by tools like Claude Code, GitHub Copilot, Cursor, Mistral, and Gemini CLI. Think of a SKILL.md file as the package.json of the agentic era.

The magic that keeps this system efficient is an architectural pattern called progressive disclosure. If an agent had to read the full body of every single skill you wrote at the start of every user prompt, your context window would explode. Instead, agents load skills in three distinct waves:

  • Discovery: The agent scans your repository but only reads the name and description from the YAML frontmatter. This costs a tiny fraction of tokens (around 100 tokens per skill).
  • Activation: Only when a user’s prompt matches a specific description does the agent open the file and inject the full Markdown body into its context.
  • Execution: Scripts, templates, and complex environment references are fetched purely on-demand as the steps run.

SKILLS.md progressive disclosure

This architectural reality reveals the number one reason why custom skills fail to trigger: it’s almost never a code problem; it’s a description problem.

Because the agent never looks at the body of the skill during the discovery phase, it relies entirely on the description to route the request. If your description reads like a documentation comment rather than an explicit behavioral trigger, your agent will walk right past it.

The Two Families of Skills

Not all skills are created equal. Understanding how they age dictates exactly how you need to test them. Anthropic separates them into two distinct categories:

1. Capability Uplift Skills

  • What it does: Teaches the model to do something it cannot do well natively (e.g., parsing a highly specific PDF format or generating a perfectly styled Word document).
  • Lifespan: Short. Obsolescence comes fast in this category. What a model fails at today might be handled flawlessly and natively by an upstream foundational model update a year from now.
  • Why it needs evals: To detect obsolescence. Your evaluations will warn you the moment a newer foundation model renders your custom code redundant.

2. Encoded Preference Skills

  • What it does: It doesn’t teach the model anything fundamentally new—the model already understands every individual step. Instead, it sequences those steps according to your company’s exact business logic or architectural workflow.
  • Lifespan: Long. These are highly durable. They live as long as your internal business process lives.
  • Why it needs evals: To verify absolute fidelity to your workflow. You need to ensure the model doesn’t skip a mandatory compliance, security, or logging step over time.

Skip managing this lifecycle, and you will fall victim to skill rot—a slow accumulation of dead folders and conflicting descriptions that pollute your discovery phase and confuse your router.

SKILLS.md life cycle

Phase 1: Designing with Intent

The most common trap is starting with the Markdown body. It feels natural to write out the beautiful, styled steps first, but expert engineering requires the exact inverse.

  • Write the description first: Treat it like a user prompt. If a developer wanted to invoke this skill explicitly, what would they type? Put those exact keywords into the description.
  • Expose hidden assumptions: Bugs love the spaces between what the model thinks it knows and what your environment actually expects. Before writing execution code, explicitly state three categories of hypotheses:
  • Trigger assumptions: What should match, and critically, what should not match?
  • Environment assumptions: Does this skill assume an empty directory? A specific package manager? Active network access? (Pro tip: always write a step that checks these prerequisites before executing real work).

  • Execution assumptions: What dependencies must be present? What is the strict sequence of events?

  • Write the body as explicit directives: Use numbered steps and authoritative vocabulary. “Always use method()” performs exponentially better than “It is recommended to use method()”. LLMs obey directives far better than they synthesize suggestions.

Phase 2: The Human Quality Gate (Validation)

Validation sits right between design and automated evaluation. Think of it as a manual code review checkpoint to see if a skill is clean enough to earn a spot in your test runner.

Before committing, run through this 7-point validation checklist:

  • Is the frontmatter syntactically complete and accurate?
  • Does the description read like a routing trigger, not a comment?
  • Does the body include explicit “when to apply” and “when to ignore” sections?
  • Is the methodology structurally sound with numbered steps and zero TODO items?
  • Is the allowed_tools block minimized? (If a skill only reads data, block it from running bash scripts).
  • Is the exact output format stringently documented?
  • Does it follow the Single Responsibility Principle? (One skill, one domain).

Phase 3: Writing Targeted Evals

An AI skill “eval” is not a massive, generalized academic benchmark. It is a highly targeted, reproducible unit test. It consists of four elements: a prompt, an agent run execution trace, a suite of checks, and a historical score to track regression over time.

When defining what a “successful run” looks like, you must look across four distinct dimensions:

  • The Outcome: Did the agent finish the task? Does the generated application actually boot up?
  • The Process: Did it invoke the correct skill? Did it use the right tools, or did it go rogue?
  • The Style: Does the code respect your company’s formatting rules, nomenclature, and exact SDK imports?
  • The Efficiency: Did it achieve the goal optimally?

Efficiency is where invisible cost leaks hide. Two different skills can produce the exact same working code output. But if Skill A takes 3,000 tokens and Skill B takes 12,000 tokens because it repeatedly retried failing bash commands, Skill B represents an invisible architectural regression. You can only catch this by tracking the trace.

Building Your Automated Test Suite

You don’t need thousands of test cases to start. A lean set of 10 to 20 highly diverse prompts is more than enough for a single skill. As users encounter real-world failures, harvest those edge cases and feed them directly into your test suite.

Ensure your test set breaks down into these four core buckets:

  1. Explicit Invocation: The prompt names the skill directly.
  2. Implicit Invocation: The prompt describes a business scenario without mentioning the skill’s name.
  3. Contextual Noise: The prompt wraps the intent in realistic, chaotic developer conversations to test routing durability.
  4. Negative Controls: Prompts designed specifically to see if the skill falsely triggers when it shouldn’t.

Phase 4: The 3-Layer Grading Matrix

Once your tests run, how do you grade the output programmatically? You build a tiered grading structure.

Layer 1: Deterministic Graders

These are your unit tests. They are fast, completely reproducible, and dirt cheap. They read the JSON execution trace or the disk artifacts and evaluate binary realities:

  • Did the agent write a package.json to the disk?
  • Did the code import the correct SDK version?
  • Did it call the new method instead of the deprecated one?

This layer is usually written in a few dozen lines of Python or TypeScript using simple regex, file checking, or AST parsing. If a check can be written deterministically, it must be written deterministically.

Layer 2: LLM-as-a-Judge

Deterministic code can tell you if a file exists, but it cannot tell you if the architectural trade-offs inside that file make sense. For qualitative assessment, we use a secondary LLM as an impartial judge, scoring the output against a highly granular grading rubric.

To ensure your system can parse the evaluations automatically, enforce a strict JSON schema layout for your rubrics:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "JSON Rubric for Deterministic Checks",
  "type": "object",
  "properties": {
    "overall_pass": { "type": "boolean" },
    "score": { "type": "integer", "minimum": 0, "maximum": 100 },
    "checks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "pass": { "type": "boolean" },
          "notes": { "type": "string" }
        },
        "required": ["id", "pass", "notes"],
        "additionalProperties": false
      }
    }
  },
  "required": ["overall_pass", "score", "checks"],
  "additionalProperties": false
}
Enter fullscreen mode Exit fullscreen mode

When building an LLM judge, you must enforce two non-negotiable rules:

  1. Enforce a strict JSON schema output so you can programmatically track scores over time in your CI pipeline.
  2. Run the judge 3 to 5 times per test and evaluate the score distribution. Because the judge itself is an LLM, its grading has a variance curve. Tighten your rubrics to minimize this drift.

Layer 3: Advanced Environment Runtime Checks

The ultimate layer of confidence mimics a real production environment:

  • Run a static analysis build check (npm run build or mvn compile) to catch broken imports.
  • Run git status --porcelain to ensure the agent left a clean workspace.
  • Spin up the containerized app and ping it with an automated curl request to verify a 200 OK response.

Phase 5: Spotting “Outgrowth” and Retiring Skills

Most teams are fantastic at creating skills, but terrible at deleting them. Over time, your codebase gets bloated with legacy instructions. Automated evals give you the exact instrument you need to clean house via two specific patterns:

  • Catch Regressions: Your skill worked perfectly a month ago. The model provider ships a stealth update, and suddenly the skill begins looping or failing. Scheduled CI evals will instantly sound the alarm before your users find out.
  • Spot Outgrowth: You wrote a complex Capability Uplift skill in 2025 to help a model parse intricate markdown tables. By 2026, the native foundational model has improved so much that it can do this out of the box.

How do you test for outgrowth? Run your eval suite with the skill completely turned off. If the agent passes your rigorous tests without the extra instructions, the model has absorbed the capability natively. Your skill is officially dead weight. Move it to an archive/ folder, log the retirement date, and delete it.

SKILLS.md deprecation strategy

Shift from How to What: The “Skills as Code” Matrix

If we peel back the AI hype and look at the structural design of a SKILL.md setup, it becomes undeniable: AI skills are code.

Let’s look at this direct architectural mapping as a clean side-by-side alignment:

  • The YAML Description ➔ Your Public API Contract (It handles discovery and routing).
  • The Markdown Body ➔ Your Implementation Details (The sequential instructions).
  • The Allowed Tools ➔ Your Core Dependencies (Filesystem, bash, or API permissions).
  • Automated Evals ➔ Your Unit & Integration Tests (The absolute guardrails of success).

The Evolution of the Body

Right now, a SKILL.md file is an imperative implementation plan. It explicitly holds the model’s hand, telling it what to do step-by-step.

But look closely at what a robust evaluation suite actually contains. It holds target prompts, expected file artifacts, explicit formatting conventions, and token efficiency budgets. Your evals already describe the what—while remaining entirely silent on the how.

In traditional software history, every time an industry transitions from telling the machine how to do something to simply telling it what we want achieved, engineering capabilities skyrocket and friction drops. We saw it when manual imperative scripts gave way to declarative infrastructure (like Terraform or Kubernetes), and we saw it when hand-crafted loop indexing gave way to SQL query planners.

As foundational LLMs continue to become more capable, the imperative Markdown body of your custom skills is going to shrink. Models will already inherently know how to safely execute standard engineering workflows. Legacy capability uplift instructions will vanish entirely. Your business-specific preference skills will reduce down to a single paragraph of pure intent, a few gold-standard examples, and a rigorous test set asserting your company’s constraints.

The code inside the skill body is temporary. The evaluation contract is permanent.

Today, your evals verify your skills. Tomorrow, the eval could be the skill. The description of success will become so precise that a highly capable model will simply deduce the implementation path dynamically at runtime.

Writing strict, automated e-vals today isn’t just a defensive quality-assurance practice to keep your current agents from breaking next week—it is a long-term architectural investment.

Stop guessing if your AI agents actually work. Build your contracts, establish your baselines, and start measuring.

Sources & Further Reading

To dive deeper into the AI agent skill lifecycle, engineering workflows, and evaluation strategies, explore these official open standards and research papers:

Top comments (0)