A skill that rewrites itself is a deployment. I want it to go through the same door as every other deployment.
Note to self before publishing: every block marked [[ALI: ...]] needs real numbers from my own AutoHarness run. Do not publish with any of them still in place, and do not round them up.
I have a folder of Claude Code skills in a .NET repository that I wrote by hand over the last few months. Some of them are good. Some of them are a pile of “and also, never do X” lines that I added the night something went wrong. A couple of them I am fairly sure nobody has triggered since July.
So when two things landed in my feed in the same week, one paper-backed optimizer that “trains” skills like parameters and one Claude Code plugin that writes, merges and prunes skills from your own sessions, I paid attention. The pitch of both is the same: stop hand-maintaining this folder.
This post is what I think after reading both carefully and running one of them on a real project. Short version: I like the direction, I like some of the engineering a lot, and I am not letting either of them write into a shared repository without a gate I control. The rest of this post is that gate.
The framing I agree with
Chandi Bhandari’s write-up on SkillOpt (Medium, September 2026) puts it in a way I keep coming back to: a prompt says what you want right now, a skill says how an agent should approach a whole class of work, again and again. Once you accept that, the skill stops being text and starts being software. It needs a version, an owner, tests, a release path and a rollback.
I agree with that completely. Where I get cautious is the next step, where the thing that writes the skill is also, in practice, the thing that decides it is good enough.
What SkillOpt actually claims (and who is claiming it)
SkillOpt is a paper from a Microsoft Research team (Yang et al., arXiv:2605.23904, May 2026) plus an open-source implementation at microsoft/SkillOpt. The core idea from the abstract:
- The target model stays frozen. The skill document is the thing being trained.
- A separate optimizer model reads scored rollouts and proposes bounded add, delete or replace edits on a single skill file.
- An edit is accepted only if it strictly improves a held-out validation score.
- A textual “learning rate” budget caps how much text can change per step.
- A rejected-edit buffer keeps failed edits around as negative evidence, so the optimizer stops re-proposing the same bad idea.
The headline numbers are the authors’ own benchmark results, not something I or anyone I know has reproduced: best or tied-best on all 52 (model, benchmark, harness) cells they evaluated, and on GPT-5.5 an average lift over the no-skill baseline of +23.5 points in direct chat, +24.8 inside Codex and +19.1 inside Claude Code. They also report that optimized skills transfer across model sizes and between Codex and Claude Code. Again: their setup, their benchmarks, their numbers.
One thing I think gets blurred in the secondary coverage. Chandi’s article goes further than the paper, and says so. The skill registry (owner, version, compatible harnesses, scorecard, approval), the canary promotion (5% traffic, a 24-hour window, a minimum task count, rollback on regression) and the “ship the evals with the skill” folder layout are his production design built on top of the paper. They are not results from the paper. The Python in his post is illustrative, not lifted from the SkillOpt codebase. I think his additions are the most useful part, honestly, but I want to keep the two apart.
I asked him in the comments how he decided the size of a bounded edit, whether it was tuned or just “keep it small”. No answer yet at the time of writing. That question matters to me more than the headline numbers, because the edit budget is the knob that decides how far a skill can drift in one step.
What AutoHarness actually does
AutoHarness (tigerless-labs/autoharness, MIT, Python 3.11+, Linux and macOS) takes the opposite bet. Its README says it plainly: no daemon, no benchmark.
Instead of validating against a held-out set, it validates by use. Roughly:
- A hook counts tool calls in your session. Every 50 tool calls (default), a background reflection runs on the recent window.
- A reflector subagent reads that episode, compares it against existing skills, and proposes add, merge, patch or delete. It has no write tools.
- A promoter is the only writer. It lints the proposal (structure, safety, size, “self-authored only”) and, if it passes, does an atomic rename straight into .claude/skills/.
- Every 250 tool calls a curator looks at the whole library and folds near-duplicates under an umbrella skill, after snapshotting the skill tree.
- A per-skill ledger (.ledger.jsonl) records why each skill was created or changed, with a pointer to a redacted slice of the session as evidence.
- A lifecycle pass tracks how often each skill is actually loaded relative to how many requests came in. Unused skills get archived , not deleted: they move to .claude/skills/.archive/ and can be moved back.
- It only touches skills it wrote. Anything without its ledger marker, including the skills I wrote by hand, is invisible to it.
Two corrections to how I first described it to myself. It does not delete anything; it archives. And “unused” is slow at the defaults: a new project skill sits in probation for 100 requests before it can even be reviewed for archiving, and the README says the defaults are placeholders pending calibration.
The README also opens with a 42% to 78% jump on CORE-Bench from changing the harness around the same model. That number comes from the HAL paper and is about harnesses in general. It is the vendor’s motivation, not a measurement of AutoHarness.
There is a lot here I genuinely like. The proposer has no write access. Merges are snapshotted. Retirement is reversible. Every change has a reason and evidence on disk. That is more discipline than most hand-maintained skill folders have, mine included.
The two approaches side by side
+----------------------+-------------------------------+-------------------------------+
| | SkillOpt (paper + repo) | AutoHarness (plugin) |
+----------------------+-------------------------------+-------------------------------+
| What gets trained | One skill document | A library of skills |
| Signal | Held-out validation score | Usage: loads per request |
| When it learns | Offline training runs | During your normal sessions |
| Edit size control | Textual learning-rate budget | 25-line body cap, lint |
| Bad ideas memory | Rejected-edit buffer | Rejected list in run records |
| Who accepts a change | Validation gate | Promoter lint, then live |
| Needs a benchmark | Yes | No (by design) |
| Retirement | Not its job | Archive on low usage |
| Human in the loop | Optional, your pipeline | Not by default |
| Evidence source | Authors' benchmarks | Vendor README |
+----------------------+-------------------------------+-------------------------------+
The row that matters to me is “who accepts a change”. SkillOpt’s gate is a score on data the optimizer did not use. AutoHarness’s gate is a linter, and after that the skill is live. Usage then decides whether it survives. But a skill that gets loaded a lot is not the same as a skill that is right. A wrong skill with a very good trigger description will look extremely healthy by that metric.
That is not a bug in AutoHarness. It is the stated trade-off: cheap, no benchmark, works on open-ended work. I just do not think “survives because it gets used” is a promotion criterion for anything that leaves my laptop.
Running AutoHarness on a real .NET project
Setup
The project: [[ALI: one-line description of the repo, e.g. “an ASP.NET Core API with EF Core, about N projects in the solution, xUnit tests”]]. I ran it on my Mac for [[ALI: N]] days of normal work, [[ALI: rough number]] sessions.
Install, typed inside Claude Code:
/plugin marketplace add tigerless-labs/autoharness
/plugin install autoharness@autoharness
/reload-plugins
The one setup trap the README warns about, and the first thing I checked on macOS: the hooks call bare python3, and Xcode ships 3.9 at /usr/bin/python3. If that one wins on your PATH, every hook silently turns off for the session.
which -a python3
python3 --version # must be 3.11 or newer
[[ALI: did this bite you? If yes, say what you saw and how you fixed it. If no, cut this sentence.]]
I left the defaults alone for the first [[ALI: N]] days, then lowered the reflection cadence because nothing was happening fast enough to write about:
{
"env": {
"AUTOHARNESS_REFLECT_EVERY_N": "20",
"AUTOHARNESS_NOTIFY": "desktop"
}
}
That goes in .claude/settings.json. The desktop notification is worth turning on; otherwise you only hear about a run one session later.
Collecting the evidence
Everything AutoHarness does lands as plain files, so I wrote a small script to turn the ledgers into a table instead of eyeballing folders. Run it from the repo root:
#!/usr/bin/env bash
# autoharness-report.sh: summarise what AutoHarness created, changed and archived
set -euo pipefail
report_dir() {
local base="$1" state="$2"
[-d "$base"] || return 0
for d in "$base"/*/; do
[-f "$d/.ledger.jsonl"] || continue # only skills AutoHarness wrote
name=$(basename "$d")
creates=$(grep -c '"action": *"create"' "$d/.ledger.jsonl" || true)
patches=$(grep -cE '"action": *"(patch|update)"' "$d/.ledger.jsonl" || true)
lines=$(grep -cv '^\s*$' "$d/SKILL.md" 2>/dev/null || echo 0)
printf "%-36s %-9s %7s %7s %6s\n" "$name" "$state" "$creates" "$patches" "$lines"
done
}
printf "%-36s %-9s %7s %7s %6s\n" "SKILL" "STATE" "CREATE" "PATCH" "LINES"
report_dir ".claude/skills" "live"
report_dir ".claude/skills/.archive" "archived"
echo
echo "Run records: $(ls .claude/autoharness/runs 2>/dev/null | wc -l | tr -d ' ')"
echo "Last run summary:"
cat .claude/autoharness/last_run.json 2>/dev/null || echo "(none yet)"
What it generated
[[ALI: paste the real output of autoharness-report.sh here, inside this code block.
Keep it as plain text. Example of the shape only, not data:
SKILL STATE CREATE PATCH LINES
<skill-name> live 1 0 12
]]
[[ALI: for each skill, two or three sentences. What scenario was it distilled from? Was the rule actually correct for this codebase? Would you have written it yourself? Be specific: name the .NET thing it was about (EF migrations, nullable warnings, test project layout, DI registration, whatever it really was).]]
What it merged or archived
[[ALI: did the curator fold anything? Check the ledgers for a fold that names an umbrella skill, and .claude/autoharness/snapshots/ for a tarball. If nothing was merged or archived, say so plainly.]]
At the default thresholds I did not expect much here in a few days. Probation is 100 requests per skill in the project layer, and the curator only runs every 250 tool calls. If you want to see merging and archiving inside a week, the README’s walkthrough values (AUTOHARNESS_MATURITY_PROJECT=5, AUTOHARNESS_CAPACITY_PROJECT=2) will force it, but then you are watching a demo, not your real lifecycle.
What went wrong
[[ALI: the honest part. Some prompts for yourself, keep only what actually happened:
- Did any skill encode something wrong, or right only for one file?
- Did a skill go live that you would have rejected in review?
- Did the reflector learn from a session where you were experimenting and later reverted?
- Did anything get rejected by the promoter, and was the rejection right? (check runs/*.json)
- Did the session-start index cost noticeable context?
- Anything about the redacted evidence files you would not want committed?]]
[[ALI: one paragraph on what you could NOT tell from a few days. For example: whether any skill made your sessions measurably better, because you had no baseline to compare against.]]
The gate I would require before a self-written skill reaches the team
Here is the position I ended up with. AutoHarness can write whatever it likes into my local .claude/skills/. That is my sandbox and the blast radius is me. But .claude/skills/ is in git, and the moment a skill is committed it loads into every teammate's session. That commit is the deployment. So the gate sits at the pull request.
One limitation I want to say out loud: this does nothing for my own session. Once AutoHarness promotes a skill locally, it is live for me immediately. The gate protects the shared repo, not my laptop.
The gate has five parts. All of them run in CI on any PR that touches .claude/skills/.
+---+-----------------------+-----------------------------------------------+
| # | Gate | Fails the PR when |
+---+-----------------------+-----------------------------------------------+
| 1 | Provenance | A self-authored skill has no ledger/evidence |
| 2 | Static lint | Banned patterns, size cap, bad frontmatter |
| 3 | Behavioural eval | Candidate is not better than baseline, or |
| | | breaks a case the baseline always passed |
| 4 | Human diff review | No CODEOWNERS approval on .claude/skills/ |
| 5 | Rollback rehearsal | The revert path is not one command |
+---+-----------------------+-----------------------------------------------+
1 and 4: provenance and ownership
CODEOWNERS is the cheapest control in this whole post:
# .github/CODEOWNERS
/.claude/skills/ @your-github-handle
/evals/skills/ @your-github-handle
Turn on “require review from code owners” in branch protection. Now nothing changes what the agent is told without a named human approving the diff. Chandi’s line about this is the one I would print out: if nobody is going to read the diffs, all the versioning is theatre.
2: static lint
This catches the cheap, dangerous stuff before spending a single model call. The banned list is .NET-flavoured and deliberately paranoid: these are the shortcuts an agent takes when it “learns” how to make a red build go green.
#!/usr/bin/env bash
# scripts/lint-skills.sh: static checks on changed skills
set -euo pipefail
BASE="${BASE_REF:-origin/main}"
MAX_LINES=40
fail=0
changed=$(git diff --name-only "$BASE"...HEAD -- '.claude/skills/*/SKILL.md' || true)
[-z "$changed"] && { echo "No skill changes."; exit 0; }
banned=(
'--no-verify'
'push --force'
'reset --hard'
'rm -rf'
'<NoWarn>'
'#pragma warning disable'
'TreatWarningsAsErrors>false'
'Skip *=' # xUnit [Fact(Skip = ...)]
'\[Ignore' # NUnit/MSTest ignore
'dotnet ef database drop'
'appsettings.Production'
)
for f in $changed; do
[-f "$f"] || continue # deleted skill, handled by review
dir=$(dirname "$f")
head -1 "$f" | grep -q '^---$' || { echo "FAIL $f: missing frontmatter"; fail=1; }
grep -q '^name:' "$f" || { echo "FAIL $f: no name"; fail=1; }
grep -q '^description:' "$f" || { echo "FAIL $f: no description"; fail=1; }
lines=$(grep -cv '^\s*$' "$f")
["$lines" -le "$MAX_LINES"] || { echo "FAIL $f: $lines lines > $MAX_LINES"; fail=1; }
for pat in "${banned[@]}"; do
if grep -qiE -- "$pat" "$f"; then
echo "FAIL $f: contains banned pattern '$pat'"; fail=1
fi
done
# Self-authored skills must carry their paper trail into review
if [-f "$dir/.ledger.jsonl"] && ! ls "$dir"/references/evidence-*.md >/dev/null 2>&1; then
echo "FAIL $f: self-authored skill without evidence files"; fail=1
fi
done
exit $fail
A skill that genuinely needs one of those patterns (say, a migration skill that must mention dotnet ef) can be allow-listed explicitly in the PR. The point is that it gets noticed.
3: the behavioural eval
This is the part neither tool gives me for my own codebase, and the part that does the real work. The idea is SkillOpt’s gate, scaled down to a solo developer: a small set of real tasks from this repo, each run several times with and without the candidate skill, scored by the toolchain rather than by a model.
Each case is a JSON file. I keep two kinds: tasks the skill is supposed to help with, and unrelated tasks it must not make worse.
{
"id": "add-endpoint-with-validation",
"prompt": "Add a GET /api/orders/{id}/status endpoint that returns 404 for unknown ids. Follow the existing controller and test conventions.",
"test_args": "--filter FullyQualifiedName~Orders",
"forbidden_paths": ["tests/", "Directory.Build.props", "global.json"]
}
forbidden_paths is the important line. An agent that makes the build pass by editing tests or loosening Directory.Build.props should fail the case even though dotnet test is green. (If the task legitimately needs new tests, put those in a separate case and drop tests/ from that one's list.)
The runner uses a throwaway git worktree per trial so runs never touch each other:
#!/usr/bin/env bash
# scripts/eval-skill.sh <skill-dir> e.g. .claude/skills/ef-migration-check
set -euo pipefail
SKILL_DIR="$1"
TRIALS="${TRIALS:-3}"
OUT="eval-results.csv"
: > "$OUT"
for case in evals/skills/*.json; do
id=$(jq -r .id "$case")
prompt=$(jq -r .prompt "$case")
test_args=$(jq -r '.test_args // ""' "$case")
for mode in baseline candidate; do
for i in $(seq 1 "$TRIALS"); do
wt=$(mktemp -d)
git worktree add --detach "$wt" HEAD >/dev/null 2>&1
["$mode" = "baseline"] && rm -rf "$wt/$SKILL_DIR"
( cd "$wt" && claude -p "$prompt" \
--allowedTools "Read,Edit,Write,Bash(dotnet build *),Bash(dotnet test *)" \
--output-format json > /dev/null 2>&1 ) || true
pass=1
( cd "$wt" && dotnet build --nologo -warnaserror > /dev/null 2>&1 ) || pass=0
# shellcheck disable=SC2086
( cd "$wt" && dotnet test --nologo $test_args > /dev/null 2>&1 ) || pass=0
changed=$(cd "$wt" && git status --porcelain | awk '{print $NF}')
for p in $(jq -r '.forbidden_paths[]?' "$case"); do
if printf '%s\n' "$changed" | grep -q "^$p"; then pass=0; fi
done
echo "$id,$mode,$i,$pass" >> "$OUT"
git worktree remove --force "$wt" > /dev/null 2>&1
done
done
done
echo "Wrote $OUT"
And the decision, kept separate so the rule is readable:
#!/usr/bin/env python3
# scripts/gate.py eval-results.csv [min_gain]
import csv
import sys
from collections import defaultdict
path = sys.argv[1]
min_gain = float(sys.argv[2]) if len(sys.argv) > 2 else 0.0
runs = defaultdict(lambda: {"baseline": [], "candidate": []})
with open(path) as fh:
for case_id, mode, _trial, ok in csv.reader(fh):
runs[case_id][mode].append(int(ok))
def rate(xs):
return sum(xs) / len(xs) if xs else 0.0
regressions = []
for case_id, r in sorted(runs.items()):
b, c = rate(r["baseline"]), rate(r["candidate"])
print(f"{case_id:40s} baseline={b:.2f} candidate={c:.2f}")
# A case the baseline always passed must not get worse at all
if b == 1.0 and c < 1.0:
regressions.append(case_id)
all_b = rate([x for r in runs.values() for x in r["baseline"]])
all_c = rate([x for r in runs.values() for x in r["candidate"]])
print(f"\nOVERALL baseline={all_b:.2f} candidate={all_c:.2f}")
if regressions:
print("REJECT: regressions on " + ", ".join(regressions))
sys.exit(1)
if all_c < all_b + min_gain:
print(f"REJECT: gain below {min_gain:.2f}")
sys.exit(1)
print("PASS")
Wiring it into GitHub Actions:
# .github/workflows/skill-gate.yml
name: Skill gate
on:
pull_request:
paths: ['.claude/skills/ **', 'evals/skills/**']
jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-dotnet@v4
with:
global-json-file: global.json
- run: npm install -g @anthropic-ai/claude-code
- name: Static lint
env:
BASE_REF: origin/${{ github.base_ref }}
run: bash scripts/lint-skills.sh
- name: Behavioural eval
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
TRIALS: "3"
run: |
for d in $(git diff --name-only origin/${{ github.base_ref }}...HEAD -- '.claude/skills/*/SKILL.md' | xargs -rn1 dirname | sort -u); do
[-f "$d/SKILL.md"] || continue
bash scripts/eval-skill.sh "$d"
python3 scripts/gate.py eval-results.csv 0.0
done
Some honest caveats about this eval, because I do not want to oversell my own gate either:
- Three trials is small. With three runs per case, one flaky run moves a pass rate by 33 points. That is why the regression rule is strict (“always passed before, must always pass now”) and the gain threshold defaults to zero: I mainly want “not worse”, not “proven better”.
- It is not cheap. Cases times modes times trials is a lot of Claude Code runs. I started with [[ALI: number]] cases. [[ALI: rough cost or time per gate run, if you measured it.]]
- Your evaluator is your ceiling. Chandi makes this point and it applies to me too. If my cases only check “build and tests pass”, a skill that writes worse but passing code sails through. The forbidden-path check catches the worst shortcuts, not taste.
- I do not yet check triggering. A skill that never loads cannot regress anything, so it passes trivially. I need positive cases where the skill should clearly fire, and I have not built that check well yet.
5: rollback
If the gate is the front door, rollback is the fire exit, and it should be boring:
# Undo a merged skill change for the whole team
git revert <merge-commit-sha>
# Or retire one AutoHarness skill locally, the same way it retires them itself
mkdir -p .claude/skills/.archive
mv .claude/skills/<skill-name> .claude/skills/.archive/
Because the skill is just a folder in git, rollback is a revert, and because AutoHarness archives instead of deleting, the local history survives either way.
What about canaries?
Chandi’s canary design (5% of traffic, 24 hours, at least 200 tasks) makes sense for a production agent that serves real requests. A coding skill in a repo has no traffic to split. My version of a canary is cruder: a new self-written skill lives on my machine for a week of real work before I open the PR, and the PR description has to say what I saw in that week. It is not statistically meaningful. It is a forcing function to actually look.
A local-only variant: running the optimizer loop on Ollama
If you want to experiment with the SkillOpt side without sending anything to a hosted API, the repo now ships a generic openai_compatible backend, and its docs list Ollama at http://localhost:11434/v1 as a supported target. Two caveats first: the docs say this backend landed after v0.2.0, so install from main, not from PyPI; and I have not measured how good a small local model is as the optimizer. I would expect it to be the weakest link.
Start Ollama in Docker (or use the native install if you have it):
docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama
docker exec ollama ollama pull qwen2.5:14b # optimizer: proposes the edits
docker exec ollama ollama pull qwen2.5:7b # target: the frozen agent being tested
Install SkillOpt from source:
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
Point both roles at Ollama. The API key is ignored by Ollama but the client expects one:
export OPENAI_COMPATIBLE_BASE_URL="http://localhost:11434/v1"
export OPENAI_COMPATIBLE_API_KEY="ollama"
Then run one of the built-in benchmarks first, to confirm the loop works end to end before you build anything of your own:
python scripts/train.py --config configs/searchqa/default.yaml \
--cfg-options \
model.optimizer_backend=openai_compatible \
model.target_backend=openai_compatible \
model.optimizer=qwen2.5:14b \
model.target=qwen2.5:7b
Splitting the roles across two model sizes is deliberate. The optimizer has the harder job (reading trajectories and writing a precise, small edit), so it gets the bigger model you can afford to run.
Getting from here to “optimize a skill for my .NET repo” is real work: SkillOpt expects a benchmark package with a data loader, a scored rollout and a config (docs/guide/new-benchmark.md in the repo describes the contract). The eval cases from the gate above are a reasonable starting dataset, since they already have a prompt and a deterministic pass/fail. I have not built that adapter yet, so I am not going to pretend I know how well it works.
Side quest: does Anthropic really run its SDLC on four files?
This came up in the same week and it is relevant, because it is the same question in a different shape: which markdown files does an agent-driven process actually depend on.
Yanli Liu’s article (All in AI, September 2026) frames Anthropic’s AI-native SDLC as coming down to four files per feature: intent.md, spec.md, plan.md and REVIEW.md. JIN's response (JIN System Architect, October 2026) argues in its headline that an AI-native SDLC does not need four files, it needs a verifiable engineering control plane.
What I could check against the primary source, Anthropic’s own “The AI-native SDLC playbook” on claude.com (August 21, 2026):
- Verified: all four filenames appear in the playbook. intent.md starts the loop, spec.md comes out of design, plan.md comes out of plan mode, and REVIEW.md holds the review policy at the repo root. Each comes with an example.
- Verified, and this is the part that undercuts “4 files”: the playbook’s list of committed artifacts is longer. It also counts the diff and its tests, the PR with its review findings, and the incident record, and from the build stage onward it says the artifact is code and its records rather than markdown. It also leans heavily on CLAUDE.md, skills, hooks and managed settings. The idea that runs through it is a chain of committed artifacts, each stage reading the previous one. "Four files" is a compression of that.
- Verified: the playbook describes itself as best practices from Anthropic’s Applied AI team, drawn from working with customers, and presents the pieces as modular plays an organization adopts in its own order. It reads as guidance and templates, not as a fixed internal standard.
- Not verified: whether Anthropic’s own engineering teams use exactly these filenames internally. The playbook’s wording does not settle that, and I found nothing else that does.
- Not verified: most of both articles. Both are member-only. I could read the openings, not the full arguments. So I cannot say whether Liu qualifies the “4 files” headline later, or exactly what JIN’s control plane consists of. JIN’s opening describes the artifact chain accurately, and the “control plane” framing is closer to what the playbook itself describes (hooks, evals, gates), but that is my reading of a headline and an intro, not of the whole piece.
My take: the useful idea is not the number four. It is “every stage commits something the next stage can check”. That is exactly what I want for skills too.
What I am taking away
I am keeping AutoHarness installed locally, for now. As a way to notice patterns in my own sessions it is interesting, and its paper trail (ledger, evidence, snapshots, archives) is better than what I had. What I am not doing is letting its output reach the team without the gate.
I am not running SkillOpt on this project yet. I do not have a scoreable task set big enough to trust a validation gate, and building one is the actual work. The paper’s results are strong on paper, on the authors’ benchmarks. My codebase is not one of them.
What I still do not know:
- [[ALI: whether any of the generated skills made your sessions better, as opposed to different.]]
- How small a bounded edit has to be before a human can review it in under a minute.
- How to test that a skill triggers when it should, cheaply.
- Whether a local model is good enough to be the optimizer, or only good enough to be the target.
The rule I am working by: anything that changes what the agent is told is a deployment. It needs a reason, a test, a reviewer and a way back. It does not matter whether a human or the agent wrote it.
Sources
- Chandi Bhandari, “Prompts Aren’t Enough Anymore: What SkillOpt Taught Me About Trainable AI Agent Skills”, Medium, Sep 2026: https://medium.com/@chandi.academic/prompts-arent-enough-anymore-what-skillopt-taught-me-about-trainable-ai-agent-skills-05be51b9132d
- Yang et al., “SkillOpt: Executive Strategy for Self-Evolving Agent Skills”, arXiv:2605.23904: https://arxiv.org/abs/2605.23904
- SkillOpt repository and backend guide: https://github.com/microsoft/SkillOpt
- AutoHarness repository (README is the source for all behaviour described here): https://github.com/tigerless-labs/autoharness
- Anthropic, “The AI-native SDLC playbook”, Aug 2026: https://claude.com/blog/the-ai-native-sdlc-playbook
- Yanli Liu, “Beyond CLAUDE.md: The 4 Files Anthropic Uses to Build an AI-Native SDLC”, All in AI (member-only, intro read): https://medium.com/ai-all-in/beyond-claude-md-the-4-files-anthropic-uses-to-build-an-ai-native-sdlc-ef4227caec7c
- JIN, “Beyond CLAUDE.md: AI-Native SDLC Doesn’t Need Four Files. It Needs a Verifiable Engineering Control Plane”, JIN System Architect (member-only, intro read): https://medium.com/jin-system-architect/beyond-claude-mdai-native-a4b975bdf89a
Tags: Claude Code, AI Agents, Agent Skills, LLMOps, Software Engineering, Dotnet, Evals, Ollama, DevOps
Top comments (0)