DEV Community

Paw from Oz
Paw from Oz

Posted on

Testing AI agents is hard. I built a framework for it.

#ai #testing #devtools #llm

Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly.

You don't find out until a user complains.

The problem: AI agents are non-deterministic. Traditional unit tests don't work. expect(output).toBe("transfer to billing") fails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways.

I built AgentSpec to fix this — a testing framework designed specifically for non-deterministic AI output.

What it looks like

Define tests in YAML:

name: "billing-agent-tests"
tests:
  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support"]
      tool_called: "transfer_to_billing"

  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "response is on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5
Enter fullscreen mode Exit fullscreen mode

Run them:

npm install -g @ozperium/agentspec
agentspec init
agentspec run
Enter fullscreen mode Exit fullscreen mode

Output:

AgentSpec v1.2.0

  ✓ routes billing question (312ms)
  ✓ handles expired token (891ms)
  ✗ response is on-topic
    Expected semantic similarity ≥ 0.5, got 0.31
    Input: "how do I reset my password?"
    Output: "Please contact support for account issues."

2 passed, 1 failed
Enter fullscreen mode Exit fullscreen mode

Why assertions built for AI

Standard test assertions assume deterministic output. AI output isn't.

AgentSpec provides:

  • contains / not_contains — substring presence (handles paraphrasing)
  • contains_any / contains_all — flexible multi-keyword matching
  • semantically_similar — word-overlap similarity score, not exact match
  • regex — pattern matching for structured output
  • tool_called — verify the agent invoked the right tool
  • max_latency_ms — catch regressions in response time
  • llm_judge — use a local model (Ollama) to evaluate quality in natural language

CI integration 

agentspec run --ci   # exits 1 on failure, JUnit XML output
Enter fullscreen mode Exit fullscreen mode

Drop it in GitHub Actions:

- name: Run AgentSpec
  run: agentspec run --ci
Enter fullscreen mode Exit fullscreen mode

Now every PR that changes a prompt, model, or tool gets behavior regression tests.

Behavior diff reports

Run with --diff to compare against a previous baseline:

agentspec run --diff baseline.json
Enter fullscreen mode Exit fullscreen mode

Shows exactly what changed between runs — useful when you swap models or update prompts.

HTTP agents

Point it at any running agent endpoint:

agentspec run --endpoint http://localhost:3000/chat
Enter fullscreen mode Exit fullscreen mode

No SDK integration required.

Install 

npm install -g @ozperium/agentspec
agentspec init
agentspec run
Enter fullscreen mode Exit fullscreen mode

GitHub: https://github.com/Ozperium/agentspec

Also in the stack: AICostTracker for tracking what you spend on AI APIs, and quota for monitoring rate limits before they stop you.

Top comments (0)