I open-sourced 410 AWS agent skills. Every one ships its test scores.

Jacky Chan — Tue, 08 Sep 2026 05:20:55 +0000

410 AWS CloudOps agent skills where quality is committed, not claimed: per-skill eval scorecards (Nova Pro executes, gpt-oss-120b judges, median of 3), signed USF v1 security manifests, and a clean OWASP Agentic Skills Top 10 sweep.

Here's a real scorecard from the repo I just published, verbatim (justifications trimmed):

{
  "skill": "s3-public-access-auditor",
  "model_id": "amazon.nova-pro-v1:0",
  "total_score": 109,
  "max_score": 120,
  "grade": "A",
  "dimensions": [
    { "id": "D1", "name": "Knowledge Delta",                 "score": 17, "max": 20 },
    { "id": "D2", "name": "Mindset + Appropriate Procedures", "score": 14, "max": 15 },
    { "id": "D3", "name": "Anti-Pattern Quality",             "score": 13, "max": 15 },
    { "id": "D4", "name": "Specification Compliance",         "score": 15, "max": 15 },
    { "id": "D5", "name": "Progressive Disclosure",           "score": 13, "max": 15 },
    { "id": "D6", "name": "Freedom Calibration",              "score": 14, "max": 15 },
    { "id": "D7", "name": "Pattern Recognition",              "score": 10, "max": 10 },
    { "id": "D8", "name": "Practical Usability",              "score": 13, "max": 15 }
  ]
}

That's not documentation about quality. It's the quality check itself, checked into git, for every task skill in jhkchan/aws-agent-skills.

Why I built it this way

I write agent skills for AWS work: audits, deployments, debugging runbooks, cost optimizations. Somewhere along the way I noticed every skills repository, including the official ones, ships instruction files and claims they were evaluated. I've never seen the evaluation.

That bothered me more than it should. If an agent is going to act on my AWS account because a skill told it to, "trust me" isn't a quality process.

So I made the evidence structural. Each skill carries its own test cases. A harness feeds them to Amazon Nova Pro, which does the task the way the skill instructs. Then gpt-oss-120b grades the output on the softaworks skill-judge rubric: eight dimensions, 120 points, covering things like practical usability and actual knowledge a model wouldn't have. The judge varies ±5 points between runs, so each skill gets graded three times and the median wins. The scorecard is committed next to the skill it belongs to.

Current tally: 407 scored, 240 at Grade A (108+), 167 at Grade B.

About those B grades

I publish them. Each scorecard shows which dimension dropped points, and the repo includes a root-cause analysis of what separates A from B. Short version: context density. Skills that pack decision trees and exact CLI syntax into the model's context do measurably better than ones that summarize. If you fork a skill and improve it, you can rerun the same harness and compare numbers.

I'd rather ship a 96 I can defend than delete it and pretend everything is a 108.

The safety half

Skills are supply chain now. You're installing instructions into an agent that has your credentials. Two things in the repo address that directly.

First, every skill passed an independent implementation of the OWASP Agentic Skills Top 10 detectors: 410 skills, zero findings. I open-sourced the scanner too, so you can rerun the sweep yourself. What the static scan can't decide (prompt-level intent, update drift) is documented as undecided rather than quietly counted as clean.

Second, every skill ships skill.usf.yaml, a Universal Skill Format manifest: what the package may read and write, its risk tier (an auditor is L0; a skill that deletes S3 versions is L3), a hash of the instruction bytes, and an ed25519 signature I made with a key my domain publishes. Verifying it tells you who published the skill. It does not tell you the skill is safe, and the repo says exactly that rather than implying more.

What's actually in the library

410 skills across 13 service families (Security 52, Compute 44, Management 43, Analytics 41, Storage 40, Networking 39, App Integration 33, Databases 31, Governance 30, Dev Tools 24, AI/ML 16, FinOps 12, Migration 5) and six task types: audit, deploy, troubleshoot, optimize, operate, automate.

The part I find most useful day to day: the new services. S3 Tables, S3 Express One Zone, EKS hybrid nodes, VPC Lattice, CloudWatch Application Signals, Amazon Verified Permissions, DataZone. Things an assistant's training data barely knows. When your model has never seen a directory bucket, a skill that has is the difference between a right answer and a confident guess. The older services get the same treatment. Lambda has ten skills, including cold-start optimization and runtime deprecation tracking. Firecracker, the open-source microVM that powers Lambda and Fargate, is the layer underneath.

Try it

Claude Code:

claude plugin marketplace add jhkchan/aws-agent-skills
claude plugin install aws-cloudops@aws-agent-skills      # router, ~300 tokens

The router installs just a catalog and an orchestrator; it finds the right skill when you need it. There's also a full-library install, but be aware it costs about 106K tokens of context per session. The README warns you instead of letting you discover that the hard way.

Any other runtime that reads SKILL.md folders (Cursor, Codex, VS Code) works too, or grab one skill at a time through the @skills protocol:

node bin/atskills.js save gh:jhkchan/aws-agent-skills/skills/s3-public-access-auditor

There's also an npm discovery CLI if you just want to poke around: npx aws-agent-skills route "why is my lambda timing out".

If you want to help

Star it if it's useful. Open issues for skills that are wrong. Where a skill and current AWS docs disagree, the docs win, and I'll fix the skill. And if you write a skill, send it: the harness will grade it the same way, and the grade goes in the commit.

https://github.com/jhkchan/aws-agent-skills

Jacky Chan, AWS Community Builder (ML & GenAI), CTO of Votee AI and Beever AI. Personal, community-driven project, Apache-2.0. Not affiliated with or endorsed by AWS.

DEV Community: Jacky Chan