DEV Community

Paw from Oz
Paw from Oz

Posted on

How I caught a Claude prompt regression before it hit production

#agents #ai #devops #testing

I changed one line in my system prompt last week. A single word — "concisely" became "briefly". I thought nothing of it.

Two hours later, a user complained that the support bot was cutting them off mid-explanation.

I had no tests. I had no diff. I had no idea what changed.

That's what pushed me to build AgentSpec — a testing framework for AI agents. This is how I actually use it.

The problem with testing AI agents

Traditional testing is deterministic: input X always produces output Y. AI agents don't work that way. Ask the same question twice and you get slightly different answers. That's not a bug — it's the whole point.

So you can't do expect(response).toBe("Reset your password by..."). You need flexible assertions that check intent, not exact output.

AgentSpec gives you that.

Setting up your first test

Install it:

npm install -g @ozperium/agentspec
Enter fullscreen mode Exit fullscreen mode

AgentSpec tests any HTTP agent that accepts {input: "..."} and returns {output: "..."}. Wrap your existing agent in one route and you're done.

Create a test file tests/support-agent.yaml:

name: "support agent regression suite"
tests:
  - name: "handles password reset"
    input: "How do I reset my password?"
    expect:
      contains_any: ["reset", "password", "email", "link"]
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "routes billing to correct team"
    input: "I want a refund for last month"
    expect:
      contains_any: ["billing", "refund", "team", "connect"]
      tool_called: "transfer_to_billing"

  - name: "doesn't hallucinate policy details"
    input: "What's your return policy?"
    expect:
      not_contains_any: ["30 days", "60 days", "90 days"]
      contains: "policy"
Enter fullscreen mode Exit fullscreen mode

Run it against your agent:

agentspec run --endpoint https://your-agent.example.com/chat --dir tests
Enter fullscreen mode Exit fullscreen mode

The behavior diff — where the real value is

After the first passing run, AgentSpec stores the baseline output for each test. When something changes, you see exactly what shifted:

WARNING REGRESSION support agent > handles password reset
  Behavior REGRESSED — test was passing, now failing
  + added: briefly, concise, short
  - removed: step, by, step, here, is, how
Enter fullscreen mode Exit fullscreen mode

That's the word diff between what the agent used to say and what it says now. In my case, adding "briefly" to the system prompt made the agent stop giving step-by-step instructions — which users actually wanted.

This diff is what I didn't have before. It would have caught the regression in 30 seconds.

LLM-as-judge for semantic checks

Some things are hard to check with string matching. "Is this response helpful?" isn't a regex problem.

AgentSpec supports LLM-as-judge using a local Ollama model — no API costs, runs in CI:

# Start Ollama locally
ollama pull qwen2.5:7b

# Run with judge
agentspec run --endpoint https://your-agent.example.com/chat \
  --judge-endpoint http://127.0.0.1:11434/v1/chat/completions \
  --judge-model qwen2.5:7b
Enter fullscreen mode Exit fullscreen mode 
tests:
  - name: "response is empathetic"
    input: "I've been waiting 3 weeks for my order"
    expect:
      llm_judge: "The response should acknowledge frustration and offer help, not dismiss the complaint"

  - name: "doesn't upsell when user is upset"
    input: "This product broke after one day"
    expect:
      llm_judge: "Focus on resolving the issue — no promotional language or upsell attempts"
      not_contains_any: ["upgrade", "premium", "offer"]
Enter fullscreen mode Exit fullscreen mode

CI integration

Add this to your GitHub Actions workflow and you'll catch regressions before they merge:

- name: Test agent behavior
  run: |
    npm install -g @ozperium/agentspec
    agentspec run --endpoint ${{ secrets.AGENT_URL }} --dir tests --ci
Enter fullscreen mode Exit fullscreen mode

--ci mode exits with code 1 on any failure. Combine it with branch protection and no prompt change ships without passing the test suite.

Start small

You don't need 50 tests. Start with three scenarios:

  1. The happy path — does it handle the most common request correctly?
  2. The edge case that burned you before — the thing that went wrong last time
  3. The thing you'd be embarrassed by — hallucinated facts, wrong policy details, off-brand tone

Three tests, 15 minutes to write, run on every deploy. That's the baseline.

npm install -g @ozperium/agentspec
agentspec init
agentspec run --endpoint https://your-agent.example.com/chat
Enter fullscreen mode Exit fullscreen mode

GitHub: github.com/Ozperium/agentspec

Top comments (0)