DEV Community

george
george

Posted on

AI Agent Skills Are Everywhere. But Which Ones Are Safe to Install?

#ai #opensource #security #githubactions

AI agents are becoming less like single chatbots and more like extensible runtimes.

Instead of using a bare model, people are adding skills: coding helpers, research workflows, browser automation, MCP integrations, content generation tools, project-specific instructions, and more.

That is powerful.

But it also introduces a new trust problem.

When you install a third-party skill, you may be giving your agent new instructions, new tool access patterns, new dependencies, and sometimes new scripts. A skill can look helpful in its README while still containing risky behavior.

For example:

  • reading .env, SSH keys, browser cookies, or API tokens
  • running curl ... | sh
  • uploading data to a webhook
  • using destructive shell commands
  • telling the agent to ignore higher-priority instructions
  • including huge prompt files that waste tokens
  • depending on unpinned or unnecessary packages

As skill ecosystems grow, we need better pre-install checks.

That is why I built SkillPreflight.

What is SkillPreflight?

SkillPreflight is an open-source CLI and GitHub Action that scans AI agent skills before installation.

It performs static analysis and generates a 100-point scorecard covering:

Category What it checks
Security Dangerous commands, secret access, exfiltration, prompt injection, remote script execution
Permission restraint Overbroad activation, unnecessary shell/network/file access
Token efficiency Oversized skill files, repeated content, poor progressive disclosure
Lightweight footprint File count, total size, dependencies, large assets
Maintainability README, license, metadata, examples, documentation hygiene
Reliability Tests, fixtures, deterministic workflow, error handling
Compatibility Hardcoded paths, OS-specific assumptions, fragile shell usage

Usage

You can run it without installing anything globally:

npx skill-preflight@latest scan ./my-skill
Enter fullscreen mode Exit fullscreen mode

You can also scan a GitHub repository:

npx skill-preflight@latest scan https://github.com/owner/repo.git
Enter fullscreen mode Exit fullscreen mode

Generate JSON:

npx skill-preflight@latest scan ./my-skill --format json --out report.json
Enter fullscreen mode Exit fullscreen mode

Generate SARIF for GitHub Code Scanning:

npx skill-preflight@latest scan ./my-skill --format sarif --out skill-preflight.sarif
Enter fullscreen mode Exit fullscreen mode

GitHub Action

SkillPreflight is also available as a GitHub Action:

name: SkillPreflight

on: [pull_request, push]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: agent-contracts/skill-preflight@v1
        with:
          target: "."
          fail-below: "70"
Enter fullscreen mode Exit fullscreen mode

Safety model

SkillPreflight does not execute scripts from the skill being scanned.

It only reads files and performs static analysis. The goal is not to guarantee that a skill is perfectly safe, but to surface obvious red flags before users install it.

Why this matters

Browsers have extension permission prompts.

npm has package audits.

Containers have image scanners.

AI agent skills should have pre-install checks too.

As the AI agent ecosystem grows, users need a simple way to ask:

What does this skill actually do, and is it safe enough to install?

SkillPreflight is an early attempt at that.

GitHub:
https://github.com/agent-contracts/skill-preflight

npm:
https://www.npmjs.com/package/skill-preflight

GitHub Marketplace:
https://github.com/marketplace/actions/skillpreflight

Iā€™d love feedback from people building or installing AI agent skills. What checks should be added next?

Top comments (0)