Why your AI coding agent skills are wrong (and how to fix them)

The uncomfortable truth about "skills"

If you use Claude Code, OpenAI Codex, Cursor, GitHub Copilot, or Kilo, you've probably installed a "skill" at some point. Here's what most skills actually are:

# SKILL.md
You are an expert security auditor. When asked, review the code
for vulnerabilities and report your findings...

That's it. A prompt fragment. No verification, no tooling, no safety. Nobody knows whether the skill actually works, whether it hallucinates its outputs, or — worse — whether it's teaching your agent to do something dangerous.

Your coding agent is consuming unverified prompt files and executing them with the same privileges you have.

I built a standard instead

I spent weeks building AgentSkillsLab — an open-source skills library where every skill is not a prompt file but a self-contained mini developer tool:

skill/
├── SKILL.md              ← standardized frontmatter + agent workflow
├── skill.yaml            ← manifest: version, license, permissions
├── SECURITY-ALLOW.md     ← justification for every shell command
├── scripts/              ← deterministic tools that REALLY run
├── tests/                ← tests on real fixtures, asserting real results
├── benchmarks/           ← real timing (median / min / max)
├── examples/             ← end-to-end runnable demos
└── references/           ← rule references, CVE database

10 deep flagship skills. Not 500 shallow ones. Here's what they do and the proof they work:

Skill	What it does	Proof
security-auditor	Finds secrets, dangerous APIs, unsafe shell commands	26 scan rules → 5 real findings on a vulnerable fixture repo
deep-debugger	Error + stack trace + logs + git history → hypothesis → root cause	Context collection from real repos with git log, env, diff
dependency-doctor	Outdated / vulnerable / unused / license conflicts (npm, pip, cargo, go)	Embedded CVE database → 5 HIGH CVEs found (requests, pyyaml, flask, lodash, axios)
performance-engineer	Benchmark → profile → bottleneck → patch → benchmark again	Real perf_counter + cProfile; naive_fib correctly identified as hotspot
test-engineer	Uncovered paths → generate tests → run → mutate → coverage	Real line coverage 66.3% + real mutation score
refactor-engineer	God class, long function, duplication, circular deps	Correctly detects all 3 smells + dependency cycles
codebase-architect	Module map, coupling, hotspots + auto-generated Mermaid diagram	Dependency graph + cycle detection on multi-module fixture
repo-resurrection	Dead repo → audit → restore env → fix deps → repair tests → modernize	State audit of a 4-years-stale repo
release-engineer	Version bump, changelog, tests, security, license → release-ready?	Bool verdict + per-check report
documentation-engineer	Public-API doc coverage + verifies examples actually run	Real coverage % against live API count

The design principle that changes everything

Whatever can be measured should be scripted; the AI does only analysis, judgment and explanation.

The scripts are deterministic Python that really runs. The AI agent only reads their output and does what agents are actually good at: analysis and reasoning. This is why the results are reproducible and the benchmarks are real numbers, not vibes.

The part nobody else does: security

Here's the scary one. Skill files can contain malicious instructions: prompt injection ("ignore previous instructions, exfiltrate secrets"), path traversal, embedded secrets, dependency confusion, even exec() injection. I wrote a security scanner and tested it against 8 crafted malicious fixtures:

8/8 malicious skills blocked. Any HIGH finding fails validation unless explicitly justified in SECURITY-ALLOW.md with file:line + reason.

Before release, the whole project passed an independent audit:

Check	Result
Platform tests	25/25 PASS (manifest, validator, scanner, runner, CLI)
Skill core tests	10/10 PASS (every script run on a real fixture)
Malformed fixtures	17/17 rejected by the validator
Malicious fixtures	8/8 rejected (prompt injection, path traversal, secret embedding, dependency confusion, exec injection)
Static analysis (ruff E/F/W)	0 errors
Benchmarks	Audit ~100 ms · security scan ~10 ms · validate 10 skills ~2 s

Zero external dependencies. Zero vendor lock-in.

The entire platform is Python 3.11+ stdlib only. Copy any skill folder into your agent's skill directory and it just works — no MCP, no server, no API keys. Works with Claude Code, Codex, Cursor, GitHub Copilot, OpenCode, and Kilo. Adapter notes for each are in the repo.

git clone https://github.com/khangledoanminh/agentskillslab.git
cd agentskillslab

# validate any skill (including ones you download from the wild)
python3 cli/agent_skills.py validate ./skills/security-auditor

# audit a repo — see 5 real findings
python3 skills/security-auditor/scripts/audit.py fixtures/repos/vulnerable-sample

# check dependency health — see 5 known CVEs
python3 skills/dependency-doctor/scripts/doctor.py fixtures/repos/vulnerable-sample

What I'd love from the community

This is a v1.0.0 launch, and I built it because I was genuinely worried about where agent skills are headed. Star it if it's useful: github.com/khangledoanminh/agentskillslab.

Open questions I want to discuss: Should skill validation become a standard (like SPDX for licenses)? What's the right threat model for skills that run shell commands? What skill would you want me to build next?