I started with a weird question:
What if I cannot train the model, but I can train the instructions around the model?
Not fine-tuning. Not LoRA. Not changing weights. I mean taking a closed model, wrapping it with a SKILL.md file, running benchmarks against it, editing the skill, measuring again, and repeating the loop until the benchmark improves.
Basically:
- Sentences become parameters.
- Edits become optimization steps.
- Benchmark score becomes the loss signal.
- The model stays frozen.
- The skill gets trained.
That idea is not as strange as it sounds. It sits at the intersection of Agent Skills, automatic prompt optimization, textual gradients, eval-driven development, and black-box optimization.
I would call this idea Skill Opt.
The model is closed, but the behavior is not
A closed model does not give you access to weights, gradients, optimizer state, or training data. You cannot backpropagate through it in the classical machine-learning sense.
But modern AI systems are not only the model. They are the model plus instructions, tools, retrieval, examples, policies, workflows, and context.
Agent Skills make this explicit. The Agent Skills specification defines a skill as a directory that contains, at minimum, a SKILL.md file. That file contains YAML frontmatter plus Markdown instructions. The required frontmatter fields are name and description, and the Markdown body contains the operational instructions the agent should follow. Skills can also include scripts, references, assets, and other supporting files.
So SKILL.md is not just documentation. It is executable behavior in text form. The agent reads it, decides when it applies, and uses it to guide actions.
That means the model is frozen, but the system behavior is still tunable.
That is the core of Skill Opt.
SKILL.md as a parameter surface
In normal training, parameters are numbers. You compute a loss, estimate a gradient, update the numbers, and run again.
In Skill Opt, the parameters are not numbers. They are pieces of text:
| Machine learning concept | Skill Opt equivalent |
|---|---|
| Model weights | Sentences, examples, constraints, workflows, and scripts in SKILL.md
|
| Forward pass | Run the agent against an eval prompt |
| Prediction | The artifact, answer, code, file, or action produced by the agent |
| Loss | Failed assertions, low benchmark score, wrong tool usage, wasted tokens, wrong skill triggering |
| Gradient | Natural-language critique explaining what went wrong |
| Optimizer step | Edit the skill |
| Epoch | One full benchmark pass |
| Validation set | Prompts not used while editing the skill |
| Overfitting | A skill that passes your exact tests but fails on real prompts |
The Agent Skills docs already recommend a version of this loop for descriptions. They suggest building labeled trigger queries, running them multiple times because model behavior is nondeterministic, measuring trigger rate, splitting train and validation sets, revising only from train failures, and selecting the best version by validation pass rate.
For output quality, they recommend structured eval cases with realistic prompts, expected outputs, optional files, baseline comparisons, assertions, grading evidence, aggregate pass rates, token counts, timing, human review, and iteration.
That is already optimization. Skill Opt just makes the analogy explicit and extends it from the description field to the whole skill.
The research already points in this direction
The closest research family is automatic prompt optimization.
Automatic Prompt Engineer, or APE, treats instructions as programs. It asks an LLM to generate candidate instructions, evaluates them with a score function, and keeps the best candidates. The paper reports that automatically generated instructions outperform a prior LLM baseline and are better than or comparable to human-written instructions on 19 of 24 tasks.
Automatic Prompt Optimization with “Gradient Descent” and Beam Search gets even closer to the Skill Opt metaphor. It uses minibatches of examples to produce natural-language “gradients” that criticize the current prompt, then edits the prompt in the opposite semantic direction of that critique. The method is guided by beam search and bandit selection, and the paper reports improvements of up to 31 percent.
OPRO, or Optimization by PROmpting, treats the LLM itself as the optimizer. It gives the model previous candidate solutions and their scores, then asks it to generate better candidates. In prompt optimization experiments, OPRO found prompts that outperformed human-designed prompts on GSM8K and Big-Bench Hard tasks.
TextGrad takes the analogy even further. Instead of numerical gradients, it uses natural-language feedback from LLMs to improve components of compound AI systems. The authors frame this as automatic “differentiation” via text and report improvements across question answering, coding, reasoning prompts, molecule optimization, and radiotherapy treatment planning.
DSPy abstracts language-model pipelines as text transformation graphs and uses a compiler to optimize them against a metric. This is important because Skill Opt is not only prompt editing. It is closer to optimizing a small language-model program made of instructions, examples, scripts, and resources.
MIPRO optimizes instructions and demonstrations for multi-stage language-model programs without access to module-level labels or gradients. That maps well to skills because a good skill often includes multiple steps where only the final artifact is scored.
GEPA uses natural-language reflection and Pareto-style evolution to improve prompts in compound AI systems. This is highly relevant because skill quality is multi-objective: correctness, cost, latency, safety, and maintainability can all matter at once.
PromptBreeder evolves task prompts and also evolves the mutation prompts that generate new task prompts. That is very close to the idea of an optimizer improving not only the skill, but also the way skill edits are proposed.
Self-Refine and Reflexion show that LLM systems can improve outputs or future behavior through iterative feedback without changing model weights. Self-Refine uses the same model to generate feedback and revise its own output, while Reflexion stores verbal lessons learned in memory to improve future trials.
Together, these papers point to the same conclusion:
Even when model weights are fixed, text around the model can be searched, scored, criticized, mutated, selected, and improved.
Skill Opt applies that pattern to SKILL.md.
The Skill Opt loop
A practical Skill Opt loop would look like this:
skill_version = SKILL.md v1
repeat:
run benchmark prompts with skill_version
collect outputs, tool calls, token usage, duration, and failures
grade with assertions, scripts, LLM judges, and human review
summarize failures as natural-language gradients
propose candidate edits to SKILL.md
run candidates on train evals
promote the best candidate by validation score
save the winning version
until validation score stops improving
The critical part is that the benchmark must be external to the optimizer. If you edit the skill using all test cases, you are not training. You are memorizing.
The Agent Skills docs explicitly warn about this for description optimization: split the query set into train and validation, use only train failures to guide changes, and select the best version by validation pass rate.
The same rule should apply to the entire skill.
What exactly gets optimized?
A skill has several trainable surfaces.
1. The description
The description field is the trigger. If it is too vague, the skill does not activate. If it is too broad, it activates when it should not.
The Agent Skills docs say the description is the primary mechanism agents use to decide whether to load a skill for a task.
So description optimization is like optimizing the router.
A weak description:
description: Helps with reports.
A trained description:
description: >
Use this skill when the user asks to turn research notes, links,
benchmark findings, or technical analysis into a structured engineering
article. Apply it for DEV-style posts, architecture writeups,
implementation retrospectives, and source-backed technical explainers.
That is not just better writing. It changes when the skill enters the context.
2. The body instructions
The body of SKILL.md is the behavioral contract. It can define:
- Steps to follow.
- Order of operations.
- Quality bars.
- Edge cases.
- Formatting rules.
- Forbidden shortcuts.
- Validation steps.
- Examples.
The Agent Skills specification recommends step-by-step instructions, examples of inputs and outputs, and common edge cases in the Markdown body.
Optimizing this section is like optimizing the policy the agent follows after activation.
3. Examples
Examples are high leverage. They show the model what good output looks like.
In Skill Opt, examples are not decoration. They are training-like demonstrations embedded in the runtime context.
You can add:
- Positive examples.
- Negative examples.
- Edge cases.
- Before and after transformations.
- “Do not do this” examples.
- Good and bad final artifacts.
Then you measure whether those examples improve benchmark performance.
4. References
References are long-form knowledge. They should not all live inside SKILL.md, because loading too much context can waste tokens and confuse the agent.
The Agent Skills specification recommends using focused reference files and loading them on demand. It also recommends keeping the main SKILL.md under 500 lines and moving detailed reference material to separate files.
Skill Opt should measure whether moving content into references improves quality, cost, or reliability.
5. Scripts
Scripts turn fragile natural-language instructions into deterministic operations.
The Agent Skills specification allows optional scripts/ directories for executable code, and the evaluation guide recommends verification scripts for checks like valid JSON, correct row counts, file existence, and expected dimensions.
So Skill Opt should not only ask:
Can better wording fix this?
It should also ask:
Should this instruction be code instead of text?
The benchmark is the loss
A skill without evals is just a belief.
To optimize a skill, you need a benchmark suite. At minimum, each eval should include:
- A realistic user prompt.
- Expected output.
- Optional input files.
- Assertions.
- A baseline run without the skill or with the previous skill.
- A score.
The Agent Skills evaluation guide recommends this exact style of eval-driven workflow: realistic prompts, expected outputs, optional files, with-skill and without-skill comparisons, assertions, grading evidence, aggregate pass rates, token counts, timing, and human review.
That gives you several possible loss signals:
- Failed assertions.
- Lower human preference score.
- Worse LLM judge score.
- More tokens for the same quality.
- Longer runtime.
- Wrong files created.
- Wrong tool used.
- Skill failed to trigger.
- Skill triggered when it should not.
- Unsafe or over-permissive behavior.
A good optimizer does not chase only one number. This is why GEPA’s Pareto-style approach is interesting: it treats optimization as a search across multiple objectives rather than a single scalar score.
The “gradient” is critique
The gradient in Skill Opt is not a vector. It is a diagnosis.
For example:
Failure pattern:
The agent produced a correct article structure, but it repeatedly forgot to include sources after technical claims.
Suggested semantic update:
Add a mandatory citation rule under the "Source handling" section. Require citations at the paragraph level for all factual claims derived from research, and require a final source list.
That critique becomes an edit.
This is the spirit of textual-gradient methods, but there is an important caveat: the gradient metaphor is useful, not literal. A 2025 paper argues that textual gradients often improve prompts, but the gradient analogy does not fully explain their behavior.
That warning matters.
Skill Opt should not pretend text optimization is mathematically equivalent to gradient descent. It should borrow the discipline of optimization without pretending the math is the same.
Skills are becoming software artifacts
The newest research makes Skill Opt feel less like a hack and more like an emerging engineering discipline.
A 2026 survey describes agent skills as composable packages of instructions, code, and resources that agents load on demand to extend capability without retraining.
A 2026 data-driven analysis of 40,285 public skills describes skills as reusable, program-like modules that define triggering conditions, procedural logic, and tool interactions. It also identifies safety risks, including skills that enable state-changing or system-level actions.
SkillsBench evaluates whether skills actually help. It reports that curated skills raise average pass rate by 16.2 percentage points across tested tasks, but effects vary by domain and some tasks show negative deltas. That is an important finding: skills can help, but they can also hurt.
A July 2026 paper studies how AI agent skills are written, adapted, and maintained. It treats skills as engineered artifacts whose content and evolution shape agent behavior. That framing is exactly what Skill Opt needs.
EvoSkills proposes self-evolving skill packages using a generator and verifier loop. It reports higher pass rates than several baselines on SkillsBench across Claude Code, Codex, and other LLMs.
AgentSkillOS looks at selection, orchestration, and benchmarking of skills at ecosystem scale, using capability trees, retrieval, DAG-based orchestration, and pairwise evaluation.
SkCC proposes compiling skills into a strongly typed intermediate representation to improve portability and security across agent frameworks. It reports pass-rate improvements, security-trigger results, and token savings.
All of this points in the same direction:
Skills are not just notes for the model. They are software artifacts that deserve tests, versioning, benchmarks, security review, and optimization.
The security problem: optimizing text can optimize bad behavior too
There is a dark side.
If SKILL.md is operational text, then malicious or careless text can change agent behavior.
A 2026 paper on semantic supply-chain attacks argues that SKILL.md is not passive documentation. It can influence discovery, selection, and governance. The paper reports that short textual triggers can improve adversarial skill visibility, description framing can bias agents toward adversarial variants, and semantic evasion can help malicious skills avoid blocking.
Another paper argues that Agent Skills enable realistic and simple prompt-injection attacks, including malicious instructions hidden inside long skill files or referenced scripts.
A 2026 paper on dynamic malicious skills shows that malicious instructions embedded in natural-language documentation, including SKILL.md, can induce an agent to dynamically inject malicious logic during execution.
This matters for Skill Opt because optimization pressure can accidentally reward unsafe behavior. If the only benchmark is “complete the task,” the skill may learn to be pushier, skip confirmation, overuse tools, or hide uncertainty.
So the loss function must include safety:
- Do not optimize only for pass rate.
- Include permission boundaries.
- Include negative tests.
- Include prompt-injection tests.
- Include tool-use constraints.
- Include human review for high-risk actions.
- Version-control every skill edit.
- Treat third-party skills like code, not like harmless documentation.
A skill optimizer without safety evals is just a bug generator with good intentions.
What Skill Opt is, and what it is not
Skill Opt is not model training.
It does not make the base model smarter in the weight-level sense. The model has not learned permanently. If you remove the skill, the behavior disappears.
But Skill Opt can make the system smarter in the engineering sense. The agent behaves better because the context, instructions, examples, scripts, activation rules, and validation loops are better.
That distinction is important.
We are not training the model.
We are training the interface between the model and the work.
Why this matters
Most teams cannot train frontier models. They cannot see the weights, modify the architecture, or run gradient descent over the model itself.
But they can write skills.
They can benchmark skills.
They can edit skills.
They can version skills.
They can compare skills.
They can build evals around skills.
That makes SKILL.md one of the most practical optimization surfaces available to people using closed AI systems.
The big idea is simple:
A closed model does not mean a closed system.
If the model is frozen, train the skill.
References
Agent Skills Overview
URL: https://agentskills.io/Agent Skills Specification
URL: https://agentskills.io/specificationAgent Skills: Optimizing Skill Descriptions
URL: https://agentskills.io/skill-creation/optimizing-descriptionsAgent Skills: Evaluating Skill Output Quality
URL: https://agentskills.io/skill-creation/evaluating-skillsZhou et al., “Large Language Models Are Human-Level Prompt Engineers”
URL: https://arxiv.org/abs/2211.01910Pryzant et al., “Automatic Prompt Optimization with ‘Gradient Descent’ and Beam Search”
URL: https://arxiv.org/abs/2305.03495Yang et al., “Large Language Models as Optimizers”
URL: https://arxiv.org/abs/2309.03409Yuksekgonul et al., “TextGrad: Automatic ‘Differentiation’ via Text”
URL: https://arxiv.org/abs/2406.07496Khattab et al., “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines”
URL: https://arxiv.org/abs/2310.03714Opsahl-Ong et al., “Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs”
URL: https://arxiv.org/abs/2406.11695Agrawal et al., “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning”
URL: https://arxiv.org/abs/2507.19457Fernando et al., “Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution”
URL: https://arxiv.org/abs/2309.16797Madaan et al., “Self-Refine: Iterative Refinement with Self-Feedback”
URL: https://arxiv.org/abs/2303.17651Shinn et al., “Reflexion: Language Agents with Verbal Reinforcement Learning”
URL: https://arxiv.org/abs/2303.11366Wang et al., “Voyager: An Open-Ended Embodied Agent with Large Language Models”
URL: https://arxiv.org/abs/2305.16291Melcer et al., “Textual Gradients are a Flawed Metaphor for Automatic Prompt Optimization”
URL: https://arxiv.org/abs/2512.13598Ling et al., “Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality”
URL: https://arxiv.org/abs/2602.08004Li et al., “SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks”
URL: https://arxiv.org/abs/2602.12670Xu and Yan, “Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward”
URL: https://arxiv.org/abs/2602.12430Gao et al., “From Registry to Repository: How AI Agent Skills Are Written, Adapted, and Maintained”
URL: https://arxiv.org/abs/2607.00911Zhang et al., “CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification”
URL: https://arxiv.org/abs/2604.01687Li et al., “Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale”
URL: https://arxiv.org/abs/2603.02176Ouyang et al., “SkCC: Portable and Secure Skill Compilation for Cross-Framework LLM Agents”
URL: https://arxiv.org/abs/2605.03353Saha et al., “Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry”
URL: https://arxiv.org/abs/2605.11418Schmotz et al., “Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt Injections”
URL: https://arxiv.org/abs/2510.26328Chen et al., “Dynamic Malicious Skills in Agentic AI”
URL: https://arxiv.org/abs/2606.16287
Top comments (0)