agent-ops-check v0.2.0 adds an optional live eval runner for AI agents

agent-ops-check is my open-source CLI that scans an AI agent for missing guardrails and writes eval cases. Until now it stopped at writing them. v0.2.0 adds an optional live eval runner so you can actually run those cases against your agent.

The static scan is unchanged: it still only writes cases, and a scan without --llm still makes no network calls.

What's new

• agent-ops-check eval: validates agent-evals.yaml and calls the agent only with --live or live: true. Dry-run is the default, and live mode warns that the agent may take real actions.
• Optional Python package agent-ops-evals (Python 3.11+), built on pydantic-evals 2.59.0 (MIT). Node and Python exchange files only: YAML in, results.json out. The Node tool keeps working when Python isn't installed, and if Python and uv are both missing the command prints install steps and exits.
• Targets in agent-evals.yaml: an HTTP endpoint, a Python callable, a CLI command, or a Pydantic AI agent.
• Packs: LLMail-Inject samples from Microsoft (MIT), OWASP-tagged cases generated from the same templates as the static scan, a JSONL file a scan already wrote, and benign emails.
• Deterministic yes/no checks: a planted canary leaked, a forbidden tool was called, a secret pattern leaked, or a benign case that should be answered was refused (over-block).
• Network promise: still true for the static scan. eval contacts only the agent you configure, nothing else.
• CI: a new python-eval job installs the package and runs the Python tests against fake HTTP and CLI agents. The existing Node jobs are unchanged.
• examples/agent-evals.yaml dry-runs from a checkout.

Not in this release

An AI/LLM judge, promptfoo export, and a static-versus-live scorecard are planned for later phases.

Release notes: https://github.com/ultimatixarup/agent-ops-check/releases/tag/v0.2.0

Feedback and issues are welcome on GitHub.

Personal project built on my own time. Not affiliated with or endorsed by my employer.