DEV Community

Solon Framework
Solon Framework

Posted on

Treat SKILL.md Like Code: Reviewable Workflows in SolonCode

#productivity

A prompt can be useful once. A Skill is different: it is an instruction package that will be discovered, selected, and reused by an agent later.

That makes a Skill closer to a small software artifact than to a long prompt. It has an activation contract, an execution path, supporting resources, and failure modes. If it is vague, it can be selected at the wrong time. If it is too large, it consumes context before the actual work begins. If it changes without review, the agent's behavior changes without an obvious code diff.

SolonCode's own skill-creator package provides a useful structure for treating Skills as reviewable assets. This article turns that structure into a practical workflow.

Start with the contract, not the prose

The smallest Skill package has a required SKILL.md file. Its frontmatter must define at least:

---
name: repository-release-notes
description: "This skill should be used when preparing release notes from a repository's merged changes and changelog entries."
license: Complete terms in LICENSE.txt
---
Enter fullscreen mode Exit fullscreen mode

The important part is not the name alone. The description is the discovery contract. It tells the agent when this package is relevant.

A weak description says:

description: Helps with release notes.
Enter fullscreen mode Exit fullscreen mode

A stronger description names the work and the evidence it expects:

description: This skill should be used when converting merged pull requests, changelog entries, and release labels into a concise repository release-notes draft.
Enter fullscreen mode Exit fullscreen mode

The second version is still short, but it gives the selector more useful signals. It also avoids claiming that the Skill handles every kind of writing.

The skill-creator guidance explicitly recommends specific descriptions and a third-person form such as “This skill should be used when...”. That is a small metadata rule with a large operational effect: if the package is not selected reliably, the quality of the instructions inside it does not matter.

Keep the top-level instructions small

A Skill package can contain three kinds of supporting resources:

repository-release-notes/
├── SKILL.md
├── scripts/
│   └── collect_changes.py
├── references/
│   └── release-style.md
└── assets/
    └── release-template.md
Enter fullscreen mode Exit fullscreen mode

Each directory has a different job.

  • scripts/ is for executable work that should be deterministic or is repeatedly rewritten.
  • references/ is for detailed material that should be loaded when the task needs it.
  • assets/ is for files used in the output, such as templates, icons, or starter files.

This is more than tidy packaging. It is a context-management strategy.

Put the essential procedure in SKILL.md: how to recognize the task, which steps must happen, and which supporting file to open. Put long schemas, examples, and policies in references/. Put reusable output material in assets/. The result follows progressive disclosure: metadata is always available, the Skill body is loaded when selected, and deeper resources are loaded only when needed.

A compact SKILL.md might look like this:

---
name: repository-release-notes
description: This skill should be used when converting merged pull requests, changelog entries, and release labels into a concise repository release-notes draft.
---

# Repository Release Notes

1. Inspect the repository's changelog and merged changes.
2. Load `references/release-style.md` before drafting.
3. Use `scripts/collect_changes.py` when a deterministic change list is needed.
4. Preserve links to source changes and separate verified facts from suggestions.
5. Render the result with `assets/release-template.md`.
6. Check the final draft against the repository's release style before returning it.
Enter fullscreen mode Exit fullscreen mode

The file is not trying to contain the whole domain. It is routing the work.

Make examples executable or at least checkable

A Skill is easier to review when its instructions imply a fixture that can be checked.

For a release-notes Skill, a fixture could contain:

fixtures/release-case/
├── CHANGELOG.md
├── merged-changes.json
└── expected-draft.md
Enter fullscreen mode Exit fullscreen mode

The fixture does not need to be part of the distributed Skill package. It can live in the repository that owns the Skill. The purpose is to answer concrete questions:

  1. Does the description cause the Skill to be selected for the intended request?
  2. Does the procedure tell the agent where to find evidence?
  3. Does the output preserve links and uncertainty?
  4. Does the template produce a usable result rather than a generic essay?

This is the same mindset used for code tests: make the expected behavior visible, then make regressions cheap to detect.

There is one important distinction. The documented skill-creator package describes the package shape and packaging validation; it does not claim that SolonCode automatically executes semantic tests for every Skill. Semantic checks remain the responsibility of the Skill author and the project that consumes it.

That boundary is healthy. Syntax and structure can be validated mechanically. Whether a workflow gives good answers requires task-specific fixtures and human review.

Use the documented creation path

When creating a new Skill from scratch, the source documents this initializer:

scripts/init_skill.py repository-release-notes --path ./skills
Enter fullscreen mode Exit fullscreen mode

The initializer creates the Skill directory, a SKILL.md template, and example resource directories. After that, remove unused examples and replace the placeholders with the smallest useful package.

The workflow is deliberately staged:

understand examples
        ↓
plan scripts, references, and assets
        ↓
initialize the package
        ↓
write the contract and procedure
        ↓
run a real fixture
        ↓
package and review the result
Enter fullscreen mode Exit fullscreen mode

This prevents a common failure mode: writing a large instruction file first and only later discovering that half of it belongs in a reference document or a deterministic script.

Package validation is a release gate

After editing, the documented packaging command is:

scripts/package_skill.py ./skills/repository-release-notes ./dist
Enter fullscreen mode Exit fullscreen mode

The package step validates the YAML frontmatter, naming conventions, directory structure, description quality, and resource references before creating a zip archive.

That validation is useful, but it is not a substitute for behavioral review. A package can have valid frontmatter and still have an ambiguous trigger. It can have a correct directory structure and still instruct the agent to invent facts. Treat the package command as a release gate for structure, then run the fixture as a release gate for behavior.

A practical review checklist looks like this:

[ ] name is stable and descriptive
[ ] description says when the Skill should be used
[ ] SKILL.md contains the short procedure, not the whole encyclopedia
[ ] every referenced file exists
[ ] scripts have a deterministic reason to exist
[ ] references are loaded only when needed
[ ] assets are output resources, not hidden instructions
[ ] examples cover the important path and at least one failure path
[ ] package validation passes
[ ] a reviewer can explain how to roll back the change
Enter fullscreen mode Exit fullscreen mode

The last item is easy to overlook. A Skill changes agent behavior. It should be possible to revert it by reverting a directory change or selecting the previous package, just as a code change can be rolled back to a known commit.

Review Skills in the SolonCode Web workflow

SolonCode exposes Skills as a managed part of the workspace rather than treating them as invisible prompt text. The Web settings code renders Skill names, descriptions, and paths for mounted sources, and provides management actions.

That makes a useful authoring loop possible:

  1. edit the Skill in a workspace;
  2. inspect its metadata and source location in the Web settings surface;
  3. run a concrete task against a fixture;
  4. revise the package;
  5. refresh the runtime's Skill discovery when needed;
  6. rerun the same fixture;
  7. keep or revert the package based on the result.

The runtime reference used by SolonCode documents engine.refreshSkills() as the refresh operation. The exact refresh timing should still be verified in the running product and version being used; a source-level method name is not a promise that every client refreshes at the same moment.

This is where a Skill starts to feel like code review. The author can show the package, the fixture, and the observed result instead of saying only “the prompt seems to work.”

Use task queueing to run the review loop without losing your train of thought

Skill authoring often produces a sequence of related checks:

  1. inspect the package;
  2. run the fixture;
  3. compare the output with the expected draft;
  4. inspect the failure case;
  5. update the description;
  6. package the result again.

SolonCode's Web interface supports queued follow-up messages for this kind of loop. While a task is running, a follow-up can wait in the queue and run after the current task. The queue is persisted through /web/chat/queue and the session's queue-tasks.json; after a refresh, queued text is restored for review but is not automatically sent.

That last behavior matters. Restoring a queue should not unexpectedly start a chain of model calls. Cold restore hydrates the UI; the user explicitly resumes it.

Steering is a different tool. An Enter submission during a running task is sent to /web/chat/steer. The message first appears as pending, then becomes visible when the backend reports system.steer_applied. If the run changes or the steering mailbox is full, the message is demoted to the normal queue rather than silently discarded.

The distinction is useful when reviewing a Skill:

Steer = change the current investigation
Queue = run the next check after the current investigation
Enter fullscreen mode Exit fullscreen mode

For example, if the agent is testing the wrong fixture, steer it: “Use the failure-case fixture instead.” If the current test is correct and you simply want the next comparison, queue: “Now compare the output with expected-draft.md.”

The implementation also gives steering a deliberate boundary. SteerInterceptor stores pending text in a transient session mailbox, injects it at a reasoning boundary, and does not interrupt an in-flight model stream or tool call. It guards the first turn and avoids injecting while the last assistant message still has open tool calls. If the task ends before the message is consumed, the backend emits a dropped event and the frontend moves it to the regular queue.

This is a good model for safe workflow control: immediate feedback does not have to mean unsafe interruption.

A small operating procedure for Skill changes

Here is a repeatable review sequence for a real Skill repository.

1. Read the current package

Record the name, description, instruction steps, and every referenced resource. Do not review only the prose; the trigger metadata is part of the behavior.

2. Identify one intended request and one near miss

Write a request that should select the Skill and another that should not. This tests whether the description is specific or merely enthusiastic.

3. Run the happy-path fixture

Use a small, known input and record the expected output shape. Preserve source links and mark any inference as an inference.

4. Queue the next check

While the first check is running, queue the failure-case comparison. Keep the current task focused rather than stuffing every future instruction into the active prompt.

5. Steer only when the current direction is wrong

If the agent starts inspecting the wrong file or uses the wrong fixture, steer it. Do not use steering as a replacement for a clearly ordered test plan.

6. Package-validate the change

Run the documented packaging command. Fix missing references, invalid metadata, or structural problems before asking anyone else to consume the package.

7. Review the diff and keep a rollback point

A Skill is behavior. Review it like behavior: what activates it, what it can cause the agent to do, and how to return to the previous version.

The larger lesson

The most useful Skills are not the longest ones. They are the ones with a precise activation contract, a short procedure, well-separated supporting material, and a checkable result.

SolonCode's package structure makes those boundaries visible:

  • SKILL.md explains what to do and when;
  • scripts/ makes repeatable work deterministic;
  • references/ keeps detailed knowledge available without bloating every run;
  • assets/ supplies reusable output material;
  • package validation catches structural mistakes;
  • fixtures and review catch behavioral mistakes;
  • steering and queueing let the author control an active review without losing the next planned check.

Once Skills are treated as code-like assets, “prompt tweaking” becomes an engineering loop: define, test, inspect, package, observe, and roll back when necessary.

Source reviewed: opensolon/soloncode

Key source paths:

  • soloncode-cli/release/skills/skill-creator/SKILL.md
  • soloncode-cli/src/main/java/org/noear/solon/codecli/portal/web/SteerInterceptor.java
  • soloncode-cli/src/main/resources/static/js/app-streaming.js
  • soloncode-cli/src/main/resources/static/js/app-settings-mounts.js
  • soloncode-cli/src/main/java/org/noear/solon/codecli/portal/web/WebController.java

Top comments (0)