DEV Community: ultimatixarup

agent-ops-check v0.2.0 adds an optional live eval runner for AI agents

ultimatixarup — Thu, 22 Oct 2026 00:34:53 +0000

agent-ops-check is my open-source CLI that scans an AI agent for missing guardrails and writes eval cases. Until now it stopped at writing them. v0.2.0 adds an optional live eval runner so you can actually run those cases against your agent.

The static scan is unchanged: it still only writes cases, and a scan without --llm still makes no network calls.

What's new

agent-ops-check eval: validates agent-evals.yaml and calls the agent only with --live or live: true. Dry-run is the default, and live mode warns that the agent may take real actions.
Optional Python package agent-ops-evals (Python 3.11+), built on pydantic-evals 2.59.0 (MIT). Node and Python exchange files only: YAML in, results.json out. The Node tool keeps working when Python isn't installed, and if Python and uv are both missing the command prints install steps and exits.
Targets in agent-evals.yaml: an HTTP endpoint, a Python callable, a CLI command, or a Pydantic AI agent.
Packs: LLMail-Inject samples from Microsoft (MIT), OWASP-tagged cases generated from the same templates as the static scan, a JSONL file a scan already wrote, and benign emails.
Deterministic yes/no checks: a planted canary leaked, a forbidden tool was called, a secret pattern leaked, or a benign case that should be answered was refused (over-block).
Network promise: still true for the static scan. eval contacts only the agent you configure, nothing else.
CI: a new python-eval job installs the package and runs the Python tests against fake HTTP and CLI agents. The existing Node jobs are unchanged.
examples/agent-evals.yaml dry-runs from a checkout.

Not in this release

An AI/LLM judge, promptfoo export, and a static-versus-live scorecard are planned for later phases.

Release notes: https://github.com/ultimatixarup/agent-ops-check/releases/tag/v0.2.0

Feedback and issues are welcome on GitHub.

Personal project built on my own time. Not affiliated with or endorsed by my employer.

White paper: agent-ops-check, a free offline check for AI agents

ultimatixarup — Mon, 12 Oct 2026 19:52:59 +0000

I wrote a 12-page white paper on agent-ops-check, a free Node CLI (MIT license) that reads agent code, prompts and configs and reports risks. It runs fully offline: no SaaS, no account, no network calls. LLM review is optional and uses your own key.

npx github:ultimatixarup/agent-ops-check ./my-agent

What the paper covers

Five needs, each matched to checks:

Need	Tab	Count today
Personal data in work channels	1 Crossover	6 personal sources, 6 work sinks, 9 secret patterns, 4 PII patterns
Agents acting on other sites	2 Actions	27 irreversible verbs, 4 gate detectors, 5 risky reads
Governance and spend	3 Governance, 4 Labs	11 provider patterns, 8 framework patterns, 11 governance findings
Hidden instructions in documents	6 ADSF	18 rule IDs
Unclear instructions	5 STE	11 style rules, 25 phrasal verbs, 31 hedges

Architecture. Inputs are a folder, files, a .zip or stdin. The rules are YAML, the engine uses regex, and the reports come out as HTML, JSON, Markdown and SARIF. The local UI listens on 127.0.0.1 only. The score runs from 0 to 100: a high finding costs 10 points, a medium costs 4 and a low costs 1. The bad example in the repo scores 27.

Standards mapping. The checks map to 10 frameworks (112 items), including OWASP Top 10 for LLM Applications 2025, OWASP Agentic 2026, NIST AI RMF, ISO/IEC 42001, MITRE ATLAS, CSA AICM and the EU AI Act. This is "maps to", never "compliant". For the paper I re-scored each item strictly against the code: 29 ✅ / 64 🟡 / 19 ⬜, against 43 / 52 / 17 shipped. In 16 items my score is lower than the shipped status.

Limits. The checks are static only, so they read what code says, not what it enforces. They can miss problems and flag harmless code. A gate is found only if it is from 6 lines above to 12 lines below an action.

Feedback on false positives is welcome.

Personal project built on my own time. Not affiliated with or endorsed by my employer; views are my own.

I shipped three free client-side scorecards (agents, cloud bills, observability) and refused to upload your paste

ultimatixarup — Wed, 30 Sep 2026 10:57:52 +0000

Agent configs and billing exports often contain secrets: API keys, account IDs, internal service names. So when I built Arup Banerjee Labs, I set one rule first: whatever you paste stays in your browser.

The result is three small, free tools that turn a paste into a scorecard.

The constraint that shaped the suite

Most "analyze your config" tools want you to upload a file or grant OAuth access to your cloud org. That's a big ask when all you want is a quick sanity check. A static site that parses and scores locally removes the trust problem: there's no backend to leak to, because there's no backend.

Everything is hosted on GitHub Pages, and the source is public: https://github.com/ultimatixarup/arup-banerjee-labs

Tool 1: Agent Health Checker

Paste an AI agent config (JSON, YAML, or plain text) and get a reliability/privacy scorecard. It looks for the things demos tend to hide:

retries and backoff
timeouts
tracing hooks
secrets sitting in the config
tool-auth notes and injection-prone tool sinks
evals/tests
a single LLM with no fallback

https://ultimatixarup.github.io/arup-banerjee-labs/agent-health-checker/

Tool 2: Cloud Bill Smell Detector

Paste a cost CSV, invoice export, or billing JSON that you exported from your own account, and get an educational smell scorecard: idle-looking lines, missing tags, transfer-heavy patterns, and similar heuristics.

Explicit non-goals: it never connects to AWS, GCP, or Azure, and it only looks at text you paste.

https://ultimatixarup.github.io/arup-banerjee-labs/cloud-bill-smell/

Tool 3: Observability Gap Finder

Paste OpenTelemetry, Prometheus, or Splunk-style config snippets, or a service inventory YAML, and get a gaps scorecard: missing SLIs, services without tracing, no cardinality guards.

https://ultimatixarup.github.io/arup-banerjee-labs/observability-gap-finder/

What "heuristic" means here

These are prioritization aids, not audit stamps. They'll produce false positives, and they won't catch everything. The goal is a fast "what should I look at first?" before something hits production.

Verify the privacy claim yourself

Open your browser's DevTools, switch to the Network tab, paste something, and run a score. Your paste should never leave the page. If you ever see it go out, that's a bug and I want to know.

What's next

I'm collecting feedback on wrong and missing checks. If one of these flags something silly, or misses something obvious in your stack, open an issue on the repo or leave a comment here.

Try the suite: https://ultimatixarup.github.io/arup-banerjee-labs/

Arup Kumar Banerjee · Little Elm, Texas

Questions? Message my free help bot on Telegram: t.me/ArupLabsHelpBot_bot

Personal project built on my own time. Not affiliated with or endorsed by my employer; views are my own.