DEV Community

Jasveer Singh
Jasveer Singh

Posted on Originally published at github.com

Your agent skill says "offline". I checked.

Installing an agent skill means trusting a sentence. The description says what the skill does — "works offline", "reads only the files you pass it", "never touches your credentials" — and the loader, the marketplace, and you all take the sentence at its word.

How often does the sentence hold? Behavioral Integrity Verification for AI Agent Skills (arXiv:2605.11770) did the comparison nobody else had: declared behaviour vs actual behaviour, across 49,943 skills in the OpenClaw registry. 80.0% deviate from what they declare. And the part that changed how I think about it: 81.1% of the deviations were developer oversight. Only 18.9% were adversarial.

The skills problem is mostly not a malice problem. It's a drift problem — a description written for v1, code that grew a telemetry helper in v3, and nobody re-reading the promise.

The tooling gap

Security scanners ask "is it dangerous?" — a necessary question, and I built one of those too (skill-vet). But look at what happens to the honesty question inside the security tools: NVIDIA's SkillSpector compares claims vs behaviour inside an optional LLM "Deep Mode" and folds the answer into a 0–100 security score. SkillAuditor.dev runs the skill in a hosted sandbox and records the verdict onchain. The PyPI skill-auditor has one description-mismatch rule inside a larger scanner.

Nobody just… prints the receipt. Deterministically, locally, claim by claim, with the evidence next to each verdict.

So I built skill-claims:

pipx install skill-claims
skill-claims audit ~/Downloads/some-skill
Enter fullscreen mode Exit fullscreen mode

How it works

  1. Claims are extracted from the package's own words — the frontmatter description, the SKILL.md body, a root README — by a pattern library. No LLM. "Works fully offline", "only reads the files you pass it", "never touches your credentials", "formats markdown tables": each becomes a claim to receipt.
  2. Behaviour is extracted from the bundled scripts by static patterns: network calls, credential reads, reads and writes outside the package, shell use, deletion, environment reads. Every hit cites file + line.
  3. Each claim gets a verdict:
    • SUPPORTED — the evidence backs it.
    • UNEVIDENCED — static evidence can neither back nor break it. (A capability whose words appear nowhere in the code. Any "guaranteed 100%" quality claim — those are always unevidenced, and the receipt says why rather than pretending to test them.)
    • CONTRADICTED — hard evidence only: the package's own code does the thing the claim rules out.

The demo fixture

The repo ships a constructed formatter whose description promises full offline working, reads limited to your files, and no credential contact. Its helper script reads ~/.aws/credentials and POSTs it with urllib. The receipt:

  • capability "Formats markdown tables…" → SUPPORTED (it really does — a receipt lists what held, too)
  • offline → CONTRADICTED (urllib call, scripts/telemetry.py:22)
  • read scope → CONTRADICTED (~/.aws read, line 11)
  • credentials → CONTRADICTED (same line)

And a second fixture that keeps all five of its promises and comes back fully SUPPORTED — because a receipt that only ever says "liar" would be a scanner with better PR.

Watch it run (32 seconds): the demo video runs audit on that same lying formatter — the capability claim comes back SUPPORTED, then the offline, read-scope and credentials claims flip to CONTRADICTED with their evidence lines on screen, and the run closes on the verdict. Watch the demo

The design decisions I care about

A capability claim is never CONTRADICTED. Static analysis cannot prove a feature absent — maybe the translation feature is implemented in a way my token search doesn't recognise. So a capability with no code evidence is UNEVIDENCED, with the exact key words I looked for printed, so a human can overrule the receipt in seconds. Contradiction is reserved for what static evidence can actually prove: the "never" claims, broken by the code itself.

No score. A 0–100 number would pretend the verdicts are commensurable. They aren't. Three contradicted promises and one supported capability is not "62/100" — it's a receipt, and you read it.

Behaviour no claim covers is still printed. A skill that never claims to be offline is allowed to use the network — that's not a contradiction. But it goes on the record, because a receipt should be complete, not just claim-shaped.

Honest limits

Static patterns are evidence, not proof — the README and every receipt say so. A skill that downloads its real logic at runtime, assembles a payload from pieces, or hides behaviour in a bundled binary can look consistent. What skill-claims catches is the common, boring, 81.1% case: the description and the code disagreeing in plain sight.

Pure Python stdlib, no dependencies, MIT. It runs beside skill-vet in my own pipeline as the pre-install honesty gate — vet for danger, claims for honesty. I'd value feedback on the claim library: phrasings it should recognise, and verdicts you'd judge differently.

Top comments (0)