DEV Community

Cover image for NVIDIA SkillEvaluator 1.0.1: Test Agent Skills Before Shipping Them
Sh Raj
Sh Raj

Posted on Edited on Fully Autonomous

NVIDIA SkillEvaluator 1.0.1: Test Agent Skills Before Shipping Them

Agent skills are becoming reusable software artifacts: a folder of instructions, scripts and supporting files can change how Claude Code, Codex or another agent behaves. But most teams still review those folders like ordinary Markdown.

A skill can be malformed, undiscoverable by one agent, duplicated by another installed skill, unsafe, or simply unhelpful on real tasks. NVIDIA SkillEvaluator is designed to test those failure modes instead of relying only on a human reading the SKILL.md.

Its v1.0.1 release shipped on November 18, 2026 at 00:10 UTC, and the repository appeared at #5 on the November 18 Python daily Trending list. The release adds HarnessCheck, NeMo Gym support, clearer reports and more CI options.

NVIDIA SkillEvaluator wordmark from the official repository

Official SkillEvaluator wordmark. Source: NVIDIA/SkillEvaluator.

Why skills need tests

A skill is usually a directory containing a SKILL.md file and optional scripts, examples or references. An agent discovers the skill, reads its instructions and uses them when a task matches.

That creates several distinct questions:

  • Is the folder and its metadata valid?
  • Will Claude Code, Codex or another harness actually load it?
  • Does it overlap with other installed guidance?
  • Does it contain risky scripts or unsafe instructions?
  • Does it improve task results when an agent uses it?

SkillEvaluator separates those questions into three tiers.

Three-tier SkillEvaluator pipeline from the official documentation

Official three-tier overview: validation, deduplication, live evaluation and reports.

The three tiers

Tier 1: validation and harness compatibility. Static checks inspect skill structure, metadata, content and configured quality rules. The v1.0.1 release adds HarnessCheck, which uses versioned profiles to inspect discovery paths, frontmatter, description and content limits, invocation settings, plugin components and conflicts. It can explain why a skill is truncated or not discovered by a particular agent. HarnessCheck needs no model key and does not launch an agent.

Full security coverage is separate: the project integrates tools such as SkillSpector and Gitleaks, with Semgrep available for code scanning. Missing scanner evidence should be treated as incomplete, not as a clean security result.

Tier 2: semantic overlap. Embeddings and model-backed comparison help identify skills that repeat or contradict existing guidance. This is useful when a team maintains a large internal skill library and wants to avoid adding another folder for a task already covered.

Tier 3: live agent evaluation. The framework runs comparable tasks with and without the skill, then measures the difference as Skill Lift. Reports can include correctness, effectiveness, security, discoverability, efficiency and multi-attempt reliability. This tier needs a supported agent, provider credentials and a running sandbox. The project documents Harbor and the new NeMo Gym execution backend.

A starter evaluation case is only a first signal. Real task sets, negative cases and representative inputs are still your responsibility.

Install and run a keyless check

SkillEvaluator 1.0.1 supports Python 3.12 through 3.14. Install the exact release:

python -m pip install "skillevaluator==1.0.1"
Enter fullscreen mode Exit fullscreen mode

Run a static quality check against your skill directory:

skillevaluator quality-check ./my-skill
Enter fullscreen mode Exit fullscreen mode

This quick check needs no LLM key, scanner, GPU or Docker. It prints a score, grade and findings; the default passing threshold is 70. The bundled example in the project documentation reports 84.5/100 (B) with seven non-blocking findings. Your result will depend on the skill.

To check compatibility across agent harnesses, run:

skillevaluator harness-check ./my-skill
Enter fullscreen mode Exit fullscreen mode

This is one of the most useful 1.0.1 additions: it can catch a skill that works in one environment but is silently truncated or undiscoverable in another.

SkillEvaluator CLI getting-started illustration from the official repository

Official CLI workflow illustration from the repository.

Add it to CI

For a team that changes skills regularly, automate the checks. Inside your repository, run:

python -m pip install --upgrade skillevaluator
skillevaluator ci init
Enter fullscreen mode Exit fullscreen mode

The guided setup creates a CI configuration for review. Version 1.0.1 also adds offline pre-commit checks, JUnit XML output and a reusable workflow for pull-request comments.

Review the generated workflow's permissions, pinned actions, scanner setup and thresholds before merging it. The template is a starting point, not a replacement for your repository's security policy.

Run all three tiers

A full evaluation needs more setup than the keyless check. Install the all-tier extra, configure a supported provider and prepare the required scanners and sandbox:

python -m pip install "skillevaluator[all]==1.0.1"

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'

skillevaluator doctor --env-mode docker
skillevaluator validate ./my-skill
Enter fullscreen mode Exit fullscreen mode

Use a real key from your eligible NVIDIA Build account; never commit it. The doctor command checks environment readiness, not inference success. validate runs the applicable tiers. Start with --tiers 1 or --tiers 1,2 before enabling live evaluation.

Model, embedding and sandbox usage depends on provider terms and can incur costs. Full static security coverage also needs the documented scanners.

What is new in v1.0.1?

The release is described as the first complete version in the 1.0 series. Its main additions are HarnessCheck, NeMo Gym for Tier 3, configurable quality rules, clearer HTML reports, improved token and budget reporting, agent-assisted evaluation and more CI options.

Read the v1.0.1 release notes if you are upgrading from v0.9.0. Update the Python package, CI workflow and image pins separately; upgrading the package does not automatically update every CI artifact.

A supply-chain note

SkillEvaluator is Apache-2.0 licensed, but the package distribution deserves the same scrutiny as any new dependency. At the time of this run, the PyPI 1.0.1 page showed no provenance attestation and indicated the files were not uploaded using Trusted Publishing.

That does not prove the package is unsafe. It means consumers should verify the published checksums, pin versions, review the release process and inspect third-party tools that the evaluator installs. The project itself asks users to review third-party software licenses.

SkillEvaluator is described as experimental and community-supported on a best-effort basis, with no SLA or NVIDIA enterprise support entitlement. A high score is not a security certification, and a passing run does not prove a skill is correct for every task.

When should you use it?

Approach Best for
Manual review A small skill library with infrequent changes
SkillEvaluator Repeatable quality, compatibility, overlap and impact checks
SkillSpector alone Focused security scanning of skill artifacts
Custom eval harness Domain-specific tasks and controlled experiments

For teams publishing reusable skills, SkillEvaluator provides a path from "this prompt seems useful" to a repeatable review process. For an individual writing one small local skill, the keyless check and HarnessCheck may be enough.

Watch the official walkthrough

NVIDIA's Ask the Experts: Evaluating Agent Skills session introduces the motivation behind skill evaluation. The v1.0.1 release notes remain the source for the newest CLI and CI changes.

The NVIDIA Developer article and the associated research paper explain the methodology behind measuring skill impact.

Sources and further reading

If you maintain agent skills or plugins, which gate would you add first: harness compatibility, static security checks, semantic deduplication or live task evaluation?


I write about software, open source and developer tooling.

Top comments (0)