How to Build AI Evals for Tool-Calling Agents

Every other week it feels like a new model shows up with a shiny score on some "trust me bro" benchmark. The numbers climb, people call it smarter, and suddenly you're ready to switch. If you build applications on top of these models, you might catch yourself assuming that higher benchmark score means better agent performance. But benchmarks measure narrow skills under controlled conditions; your agent has a specific job to do. If you're building a customer support agent, you don't care about a few extra points on a reasoning test. You care about whether it calls the right tool, passes the right arguments, and avoids risky or redundant actions. That's the behavior that actually matters.

Testing a tool-calling agent means testing its decisions, not just its words. An agent can sound perfectly convincing while quietly messing things up underneath. When it answers "Your refund has been processed", the response sounds fine but did it check the right order? Refund the correct amount? Accidentally hit the refund API twice? The final response won't tell you.

What you actually need is an eval suite: a set of automated tests that score how well an agent performs. Eval is short for evaluation: each test runs the agent on a scenario, asks a grader "was this behavior correct?", and turns the answer into a score you can track and gate on. Traditional software tests and prompt-and-response evals both fall short here. A unit test asserts that a function returns the right value for a given input. An agent, though, makes a sequence of decisions (which tool to call, with what arguments, in what order), and each decision is non-deterministic. The same prompt can produce a different tool-calling path on every run. A single manual test tells you what the agent can do, not what it typically does.

So the eval suite should answer three questions:

1. Selection: Did the agent pick the right tool for the request?
2. Trajectory: Did it call the right tools, with the right arguments, in the right order, without wasted or dangerous calls?
3. Outcome: Is the final answer correct and grounded in what the tools returned?

In this guide, you'll build exactly that. Using Mastra, you'll create a customer support agent that can look up orders, process refunds, and escalate to a human. Then you'll write a layered eval suite for it:

• Quick Checks: zero-LLM, deterministic assertions like "the agent must call lookup_order" and "no tool call may error"
• Gates and verdicts: hard pass/fail requirements that fail a run outright
• Trajectory scorers: validate the full sequence of tool calls, including arguments, step budgets, and blacklisted tools
• LLM-as-a-judge scorers: semantic grading for the cases where exact matching is too rigid
• A Vitest suite: so the whole thing runs in CI on every change

Before the code: what an eval is

Before we start typing, let's get one mental model straight, because it's the vocabulary the rest of the guide (and Mastra's API) assumes. An eval is three things put together: a test case, an expected behavior, and a grader.

Input: "I want a refund for order 1001"

Expected behavior: Look up order 1001, then process the refund.

Grader: Did those things happen, in that order, with the correct order ID?

Score: 1 if correct, 0 if not.

Unlike a traditional unit test, you usually run an agent through the same input multiple times, because an agent's behavior isn't deterministic. The same prompt can call different tools on different runs, so you score many runs and average the results rather than trusting a single pass.

A handful of terms come up constantly, and it helps to pin them down now:

Concept	Meaning
Test case / data item	The scenario you give the agent (input)
Expected behavior	What you believe the agent should do
Scorer / grader	The mechanism that judges whether the agent did it
Eval suite	The collection of test cases plus graders
Gate	A requirement that must pass, or the run fails
Threshold	A minimum acceptable average score

Two of these matter more than the rest, so let's call them out. A gate is a hard requirement: if it doesn't pass, the whole run is failed, full stop. A threshold is a softer quality bar, like "average relevancy above 0.8", that a run can miss and still be usable. Keep that difference in mind; we devote a whole section to it below.

One last principle, and it's arguably the most important lesson in the guide: a grader is only meaningful relative to the scenario being graded. The scorer "the agent must call lookup_order" is exactly right for an order-status request, but it should fail for the question "What's the capital of France?" and that failure isn't an agent bug. It just means that scorer belongs to a different scenario than the off-topic one. The way forward is never a weaker, vaguer checker; it's organizing your test cases so each scenario gets graders that match its own expectations.

Why testing tool-calling agents is different

When a user sends a request to an agent, several decisions happen in sequence. The agent decides which tool (if any) to call, builds the arguments, executes the call, reads the result, and either calls another tool or synthesizes a final answer. Each step introduces a distinct failure mode:

• Wrong tool selection. The user asks for a refund, and the agent calls lookup_order but never process_refund, then claims the refund was processed anyway.
• Wrong arguments. The agent calls the right tool but passes orderId: "100" instead of "1001", gets an empty result, and hallucinates around it.
• Bad ordering. The agent refunds before verifying the order exists, because process_refund was "easier" to call first.
• Inefficient or looping trajectories. The agent calls the same tool repeatedly with identical arguments, burning tokens and latency.
• Bad synthesis. Every tool call succeeded, but the final answer contradicts what the tools returned.

Notice that only the last of these is visible if you only test the final response. This is why the broader agent evaluation field has converged on a layered approach: use deterministic, code-based graders for everything that is objectively checkable (tool names, arguments, call order, error rates), and reserve LLM-as-a-judge graders for semantic questions (was this tool choice appropriate? is the answer helpful?). Deterministic checks are free, instant, and reproducible, so they should carry as much of your suite as possible.

The example agent in this guide is deliberately small, but the failure modes above are exactly what benchmarks like τ-bench and τ²-bench measure at scale: tool-using agents navigating realistic customer-support scenarios.

Prerequisites

Before you begin, ensure you have:

1. Node.js: Version 22 or later. Download from nodejs.org.
2. OpenAI API key: The agent and the LLM judges both use OpenAI models through Mastra's model router. Create a key at platform.openai.com. Any other provider works too; just change the model strings.

Set up the project

Create a new directory and initialize it:

mkdir support-agent-evals && cd support-agent-evals
npm init -y && npm pkg set type=module

Install the dependencies:

npm install @mastra/core @mastra/evals zod
npm install --save-dev vitest tsx

• @mastra/core: Mastra's core package, which includes agents, tools, and the runEvals evaluation pipeline.
• @mastra/evals: The evals package, which includes Quick Checks and all built-in scorers.
• vitest: The test runner you'll use to run evals in CI. Any ESM-compatible runner (Jest, Mocha) works.

Create a .env file with your OpenAI key:

OPENAI_API_KEY=your_openai_key_here

Build the agent under test

The agent you'll test is a customer support agent for an online store. It has three tools: look up an order, process a refund, and escalate to a human. A refund requires looking up the order first, a realistic business rule that gives you something meaningful to test.

Create a src/agent.ts file:

import { Agent } from '@mastra/core/agent'
import { createTool } from '@mastra/core/tools'
import { z } from 'zod'

// In-memory "database" of orders
const orders: Record<string, { id: string; status: string; total: number }> = {
  '1001': { id: '1001', status: 'delivered', total: 59.99 },
  '1002': { id: '1002', status: 'shipped', total: 129.0 },
  '1003': { id: '1003', status: 'processing', total: 24.5 },
}

const refundedOrders = new Set<string>()

export const lookupOrderTool = createTool({
  id: 'lookup_order',
  description: 'Look up an order by its ID to get status and total',
  inputSchema: z.object({
    orderId: z.string().describe('The order ID, e.g. "1001"'),
  }),
  outputSchema: z.object({
    found: z.boolean(),
    order: z.object({ id: z.string(), status: z.string(), total: z.number() }).optional(),
  }),
  execute: async ({ orderId }) => {
    const order = orders[orderId]
    return { found: Boolean(order), order }
  },
})

export const processRefundTool = createTool({
  id: 'process_refund',
  description: 'Process a refund for a delivered order. Always look up the order first.',
  inputSchema: z.object({
    orderId: z.string().describe('The order ID to refund'),
  }),
  outputSchema: z.object({
    refunded: z.boolean(),
    reason: z.string().optional(),
  }),
  execute: async ({ orderId }) => {
    const order = orders[orderId]
    if (!order) return { refunded: false, reason: 'order_not_found' }
    if (order.status !== 'delivered') return { refunded: false, reason: 'order_not_delivered' }
    if (refundedOrders.has(orderId)) return { refunded: false, reason: 'already_refunded' }
    refundedOrders.add(orderId)
    return { refunded: true }
  },
})

export const escalateTool = createTool({
  id: 'escalate_to_human',
  description: 'Escalate to a human agent when the customer is upset or the request is outside policy',
  inputSchema: z.object({
    reason: z.string().describe('Why the conversation is being escalated'),
  }),
  outputSchema: z.object({
    escalated: z.boolean(),
    ticketId: z.string(),
  }),
  execute: async ({ reason }) => {
    return { escalated: true, ticketId: `TICKET-${Math.floor(Math.random() * 10000)}` }
  },
})

export const supportAgent = new Agent({
  id: 'support-agent',
  name: 'Support Agent',
  instructions: `You are a customer support agent for an online store.

Rules:
- Always use the lookup_order tool to verify an order before answering questions about it.
- To process a refund, first call lookup_order to confirm the order exists and is delivered, then call process_refund.
- Only delivered orders can be refunded.
- If the customer is upset or asks for a human, use the escalate_to_human tool.
- Never make up order information. Only report what the tools return.`,
  model: 'openai/gpt-5-mini',
  tools: {
    lookup_order: lookupOrderTool,
    process_refund: processRefundTool,
    escalate_to_human: escalateTool,
  },
})

The above code does the following:

• Tools with schemas: Each tool is created with createTool, with Zod schemas for both input and output. The schemas matter for testing later: when an agent passes a malformed argument, Mastra validates it against inputSchema and records a tool error, which your tests can detect.
• A stateful side effect: process_refund mutates the refundedOrders set, and it refuses refunds for orders that aren't delivered. This gives your evals something real to check: did the agent call it correctly, at the right time, with the right ID?
• A policy in the instructions: The system prompt encodes the refund policy ("look up first, only refund delivered orders"). Your eval suite is, in effect, an executable version of this policy: every rule in the prompt should map to at least one test case.
• The model: 'openai/gpt-5-mini' uses Mastra's model router format (provider/model-name). Swap the prefix to use Anthropic, Google, or any other supported provider.

Your first eval: Quick Checks

Quick Checks are composable micro-scorers for common assertions: "the output contains X", "the agent called tool Y", "no tool errored". They make no LLM calls, so they run in microseconds and cost nothing. This makes them the foundation of your suite.

Create a src/evals.ts file:

import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { supportAgent } from './agent.js'

const result = await runEvals({
  target: supportAgent,
  data: [
    { input: 'Where is my order 1002?' },
    { input: 'Has my package 1003 shipped yet?' },
    { input: "What's the capital of France?" },
  ],
  scorers: [checks.calledTool('lookup_order'), checks.noToolErrors()],
})

console.log(result.scores)
console.log(`Processed ${result.summary.totalItems} items`)

Run it with:

npx tsx src/evals.ts

The output will look like this:

{
  "check-called-tool": 0.6666666666666666,
  "check-no-tool-errors": 1
}
Processed 3 items

The above code does the following:

• runEvals: The evaluation pipeline from @mastra/core/evals. It runs every item in data through the target agent and scores each run with every scorer in scorers. It returns average scores per scorer plus a summary.
• checks.calledTool('lookup_order'): Scores 1 if the agent called lookup_order at least once during the run, 0 otherwise. Notice the average above is ~0.67, not 1: the agent correctly used no tools for the off-topic question about France, so that item scored 0 for this check. This is the most important habit in eval design: a check that should pass for one case will legitimately fail for another. Keep reading: the fix is per-scenario grouping, not a looser check.
• checks.noToolErrors(): Scores 1 if every tool invocation completed without errors. This catches malformed arguments and schema violations regardless of which tools were called, so it's safe to apply to every scenario.

The natural way to organize your data is to group test cases by scenario and give each scenario its own expectations:

const result = await runEvals({
  target: supportAgent,
  data: [
    // Order status questions must trigger a lookup
    { input: 'Where is my order 1002?' },
    { input: 'Has my package 1003 shipped yet?' },
  ],
  scorers: [checks.calledTool('lookup_order'), checks.noToolErrors()],
})

// Off-topic questions are a separate scenario with opposite expectations
const offTopic = await runEvals({
  target: supportAgent,
  data: [{ input: "What's the capital of France?" }],
  scorers: [checks.usedNoTools()],
})

Here checks.usedNoTools() asserts the agent answered without calling any tool. Negative tests like this are how you catch an agent that over-eagerly calls tools it doesn't need, a real cost and latency problem in production.

Test what the agent must not do

Beyond "called the right tool", the most valuable tool-calling tests are often about what the agent avoided. The full set of tool-call Quick Checks:

Check	What it asserts	Failure it catches
checks.calledTool(name)	Tool was called (optionally { times: n })	Missing capability
checks.didNotCall(name)	Tool was never called	Overreach, policy violations
checks.toolOrder([a, b])	Tools were called in this relative order	Unsafe sequencing
checks.maxToolCalls(n)	No more than n total tool calls	Loops, inefficiency, cost blowups
checks.usedNoTools()	No tools called at all	Unnecessary tool usage
checks.noToolErrors()	No tool invocation errored	Malformed arguments, schema violations

Applied to the refund scenario:

const refund = await runEvals({
  target: supportAgent,
  data: [{ input: 'I want a refund for order 1001' }],
  scorers: [
    checks.toolOrder(['lookup_order', 'process_refund']),
    checks.didNotCall('escalate_to_human'),
    checks.maxToolCalls(3),
    checks.noToolErrors(),
  ],
})

console.log(refund.scores)
// {
//   'check-tool-order': 1,
//   'check-did-not-call': 1,
//   'check-max-tool-calls': 1,
//   'check-no-tool-errors': 1
// }

Each of these maps to a production failure mode. toolOrder enforces the "verify before refunding" policy from the system prompt. didNotCall('escalate_to_human') catches agents that give up and escalate routine requests. maxToolCalls(3) catches retry loops. Note that toolOrder checks relative order: the agent may insert extra calls between the expected ones. If you need exact-sequence validation, that's what trajectory scorers are for, covered below.

Gates and verdicts: turn checks into pass/fail

Averages are useful for tracking quality over time, but CI needs a sharper signal. Gates are scorers that must average 1.0 across all data items; if any gate slips, the whole run gets a failed verdict. Regular scorers can instead carry a threshold, and the run reports a three-state verdict:

• passed: all gates scored 1.0 and all thresholds were met
• scored: gates passed, but at least one threshold was missed
• failed: at least one gate averaged below 1.0

const result = await runEvals({
  target: supportAgent,
  data: [
    { input: 'Where is my order 1002?' },
    { input: 'I want a refund for order 1001' },
  ],
  gates: [checks.noToolErrors()],
  scorers: [checks.includes('order')],
})

console.log(result.verdict) // 'passed' | 'scored' | 'failed'
console.log(result.gateResults) // [{ id: 'check-no-tool-errors', passed: true, score: 1 }]

To see what a failure looks like, imagine a regression where the agent stops calling tools. The gate output makes the cause obvious:

verdict: failed
gateResults: [{ "id": "check-called-tool", "passed": false, "score": 0 }]

The mental model: gates are for invariants (never error, never call the dangerous tool, always look up before refunding), while thresholds are for quality metrics (relevancy above 0.8). Gates should almost always be deterministic checks, since you don't want your CI gate to depend on the mood of an LLM judge.

Test the full trajectory

Quick Checks answer questions about individual tool calls. Trajectory scorers answer questions about the sequence: did the agent take a sensible path, with the right arguments, within a reasonable budget? runEvals extracts the trajectory automatically and hands it to any scorer registered under the trajectory key.

There are two flavors:

1. createTrajectoryAccuracyScorerCode: deterministic comparison of the actual trajectory against expected steps. Free, instant, reproducible.
2. createTrajectoryAccuracyScorerLLM: an LLM judges whether the path was necessary, well-ordered, and complete, with written reasoning. Use when there is no single correct path.

For per-scenario expectations, put an expectedTrajectory on each data item:

import { createTrajectoryAccuracyScorerCode } from '@mastra/evals/scorers/prebuilt'

const trajectoryScorer = createTrajectoryAccuracyScorerCode()

const result = await runEvals({
  target: supportAgent,
  data: [
    {
      input: 'I want a refund for order 1001',
      expectedTrajectory: {
        steps: [
          { stepType: 'tool_call', name: 'lookup_order', toolArgs: { orderId: '1001' } },
          { stepType: 'tool_call', name: 'process_refund' },
        ],
      },
    },
    {
      input: 'Where is my order 1002?',
      expectedTrajectory: {
        steps: [{ stepType: 'tool_call', name: 'lookup_order' }],
      },
    },
  ],
  scorers: { trajectory: [trajectoryScorer] },
})

console.log(result.scores.trajectory) // { 'code-trajectory-accuracy-scorer': 1 }

The above code does the following:

• Per-item expectations: Each data item declares the steps it should produce. The scorer matches actual steps against them in relative order by default (extra steps in between are tolerated with a small penalty). Pass comparisonOptions: { strictOrder: true } to the scorer constructor for exact-sequence matching instead.
• Argument validation: When an expected step includes toolArgs, the scorer compares them against the actual arguments. This is how you catch the "right tool, wrong orderId" failure mode, a check no final-answer eval can perform.
• Score semantics: The scorer returns 0–1 in relaxed mode (fraction of expected steps matched, minus penalties) and binary 0/1 in strict mode.

Budgets, redundancy, and blacklists

For broader guardrails, createTrajectoryScorerCode evaluates four dimensions in one pass: step accuracy, efficiency budgets, blacklisted tools, and tool-failure patterns:

import { createTrajectoryScorerCode } from '@mastra/evals/scorers/prebuilt'

const guardrails = createTrajectoryScorerCode({
  defaults: {
    blacklistedTools: ['escalate_to_human'], // never escalate in these scenarios
    maxSteps: 4, // fail runs that take more than 4 steps
    noRedundantCalls: true, // fail runs that repeat identical tool calls
    maxRetriesPerTool: 2, // tolerate at most 2 retries per tool
  },
})

const result = await runEvals({
  target: supportAgent,
  data: [
    {
      input: 'I want a refund for order 1001',
      expectedTrajectory: {
        steps: [
          { stepType: 'tool_call', name: 'lookup_order' },
          { stepType: 'tool_call', name: 'process_refund' },
        ],
      },
    },
  ],
  scorers: { trajectory: [guardrails] },
})

The scorer produces a human-readable reason per run, which is exactly what you want when a CI job fails at 2am:

Score: 1
Accuracy (1): 2/2 expected steps matched.
Efficiency (1): all budgets met, no redundant calls.

A blacklist violation forces the score to 0 regardless of everything else, and says so:

Score: 0
Blacklist violation: forbidden tools used: escalate_to_human.

Add LLM-as-a-judge for semantic checks

Deterministic scorers answer "did the agent call lookup_order?" They can't answer "was calling lookup_order the appropriate response to this request?" For that, you need a judge that understands intent. Mastra ships LLM-based variants of both the tool-call and trajectory scorers:

import {
  createToolCallAccuracyScorerLLM,
  createAnswerRelevancyScorer,
} from '@mastra/evals/scorers/prebuilt'

const toolAppropriateness = createToolCallAccuracyScorerLLM({
  model: 'openai/gpt-5-mini',
  availableTools: [
    { name: 'lookup_order', description: 'Look up an order by its ID' },
    { name: 'process_refund', description: 'Process a refund for a delivered order' },
    { name: 'escalate_to_human', description: 'Escalate to a human agent' },
  ],
})

const relevancy = createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' })

const result = await runEvals({
  target: supportAgent,
  data: [
    { input: 'Where is my order 1002?' },
    { input: 'I want a refund for order 1001' },
  ],
  scorers: [
    { scorer: toolAppropriateness, threshold: 0.8 },
    { scorer: relevancy, threshold: 0.8 },
  ],
})

console.log(result.thresholdResults)
// [
//   { id: 'llm-tool-call-accuracy-scorer', passed: true, averageScore: 0.9, threshold: 0.8 },
//   { id: 'answer-relevancy-scorer', passed: true, averageScore: 0.87, threshold: 0.8 },
// ]

The LLM tool-call scorer evaluates each call against the user's request and the catalog of available tools, and returns fractional scores with written reasoning, including flags for tools that should have been called but weren't. The { scorer, threshold } wrapper turns the average into a pass/fail signal: a number means "at least this", and { max: 0.3 } inverts it for scorers where high is bad (hallucination, toxicity).

Two rules of thumb for the judge layer:

• Keep it small. LLM judges cost tokens and are themselves non-deterministic. Everything expressible as a deterministic check should be one; judges are for the residue that genuinely requires semantic understanding.
• Calibrate before you trust it. Read the judge's reasoning on a sample of runs and compare it against your own judgment. If you disagree with the judge more than occasionally, tighten the rubric or the tool descriptions before wiring it into CI.

Run evals in CI with Vitest

Everything so far composes into a standard test file. The pattern: one it block per scenario, gates for invariants, thresholds for quality metrics, and an assertion on the verdict. Create src/agent.eval.test.ts:

import { describe, it, expect } from 'vitest'
import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { createTrajectoryScorerCode } from '@mastra/evals/scorers/prebuilt'
import { supportAgent } from './agent.js'

const trajectoryGuardrails = createTrajectoryScorerCode({
  defaults: { maxSteps: 4, noRedundantCalls: true },
})

describe('Support Agent', () => {
  it('looks up orders before answering status questions', async () => {
    const result = await runEvals({
      target: supportAgent,
      data: [{ input: 'Where is my order 1002?' }, { input: 'Has my package 1003 shipped yet?' }],
      gates: [checks.calledTool('lookup_order'), checks.noToolErrors()],
      scorers: { trajectory: [trajectoryGuardrails] },
    })

    expect(result.verdict).toBe('passed')
  })

  it('verifies orders before refunding them', async () => {
    const result = await runEvals({
      target: supportAgent,
      data: [
        {
          input: 'I want a refund for order 1001',
          expectedTrajectory: {
            steps: [
              { stepType: 'tool_call', name: 'lookup_order', toolArgs: { orderId: '1001' } },
              { stepType: 'tool_call', name: 'process_refund' },
            ],
          },
        },
      ],
      gates: [checks.toolOrder(['lookup_order', 'process_refund']), checks.didNotCall('escalate_to_human')],
      scorers: { trajectory: [trajectoryGuardrails] },
    })

    expect(result.verdict).toBe('passed')
  })

  it('escalates upset customers instead of improvising', async () => {
    const result = await runEvals({
      target: supportAgent,
      data: [{ input: 'This is unacceptable! Get me a human right now!' }],
      gates: [checks.calledTool('escalate_to_human'), checks.didNotCall('process_refund')],
    })

    expect(result.verdict).toBe('passed')
  })

  it('answers off-topic questions without tools', async () => {
    const result = await runEvals({
      target: supportAgent,
      data: [{ input: "What's the capital of France?" }],
      gates: [checks.usedNoTools()],
    })

    expect(result.verdict).toBe('passed')
  })
})

Add a test script to package.json and run it:

npm pkg set scripts.test="vitest run"
npm test

 RUN  v4.1.10 /home/you/support-agent-evals

 Test Files  1 passed (1)
      Tests  4 passed (4)

The above test file does the following:

• One scenario per test: Each it block groups data items that share expectations. When a test fails, the test name tells you which behavior regressed, and gateResults tells you why.
• Gates plus guardrails: Each scenario combines hard gates (binary, deterministic) with the trajectory guardrail scorer (budgets and redundancy) that applies everywhere.
• CI-ready: vitest run exits non-zero on failure, so this drops into GitHub Actions or any other CI as-is. Your OPENAI_API_KEY goes in the CI secrets.

Tip: non-determinism in CI

Agent runs are non-deterministic, so an eval can flake the same way the agent does. Two mitigations: run each scenario with several data items so the averages are meaningful (runEvals supports concurrency to keep this fast), and reserve gates for checks that should pass at 100%; a gate at "mostly passes" is a flaky test by definition. For a fully deterministic CI lane, you can also swap the agent's model for the AI SDK's MockLanguageModel (from ai/test) that returns scripted tool calls, and reserve live-model evals for a scheduled job.

Best practices

A few principles that make agent test suites useful instead of noisy:

• Write a test for every rule in your system prompt. The prompt says "always look up the order before refunding"; the suite has toolOrder(['lookup_order', 'process_refund']). The prompt says "escalate upset customers"; the suite has an escalation scenario. When the prompt changes, the suite should change in the same commit.
• Grow the dataset from real failures. Every time the agent misbehaves in development or production, add the offending prompt as a data item with the correct expectations. Regression suites built from actual failures beat hand-invented happy paths every time.
• Deterministic first, judges second. Quick Checks and trajectory scorers are free, instant, and reproducible. Add LLM judges only for questions that exact matching can't express, and give them thresholds rather than gate status.
• Assert on arguments, not just tool names. The most damaging tool-calling bugs are argument bugs: right tool, wrong ID, wrong amount. toolArgs in expected trajectory steps exists for exactly this.
• Test the negatives. "Must not call process_refund for angry rants" and "must not call any tool for off-topic questions" catch overreach that positive tests never see.

Extending this workflow

The suite you built runs against curated datasets before shipping. Mastra's evals system extends in both directions from here:

• Multi-turn conversations: Use inputs (sequential turns on one thread) or turns (per-turn gates and scorers) in your data items to test behavior that emerges across a conversation, like an agent that correctly looks up an order on turn one but forgets the order ID by turn three. See Multi-turn evals.
• Live production scoring: Attach scorers directly to the agent with a sampling.rate to score a percentage of real traffic asynchronously, the same scorers, pointed at production instead of a dataset. Start with checks.noToolErrors() at rate 1: it's free and catches argument regressions immediately.
• Experiments and datasets in Studio: Register scorers with your Mastra instance and run experiments from Mastra Studio, including scoring historical traces and saving interesting results as dataset items, the production-failure-to-regression-test loop, with a UI.
• Custom scorers: When built-ins don't fit, createScorer gives you the same four-step pipeline (preprocess, analyze, generateScore, generateReason) to encode your own evaluation logic.

Conclusion

Testing a tool-calling agent means testing its decisions, not just its words. The final answer is the last place failures show up; by then, the wrong tool has been called, the wrong argument passed, or the same call repeated five times.

With Mastra, the layered suite looks like this: Quick Checks for deterministic invariants, gates and verdicts for a single CI signal, trajectory scorers for sequences, arguments, and budgets, and LLM judges, sparingly, for appropriateness and answer quality. Each layer catches what the layer above it can't, and the whole suite runs in CI on every prompt change and model upgrade, which is exactly when agents quietly regress.

Resources

• Mastra Evals Overview
• Quick Checks
• Gates and Verdicts
• Trajectory Accuracy Scorers
• Tool Call Accuracy Scorers
• runEvals API Reference
• Running Scorers in CI
• Demystifying evals for AI agents: Anthropic
• τ²-bench: Evaluating Conversational Agents

Comments

[Reid Marlow]

The layered suite is the part I would keep strict. Tool presence checks catch regressions early, but they also become noisy if they turn into snapshots of one lucky trajectory. I usually want at least one hostile fixture beside the happy path so the eval proves it can reject overreach, not just bless the demo run.

[Dean Lee]

I like the separation between selection, trajectory, and outcome. For tool-calling agents, trajectory is where most demos hide risk. A correct final answer can still come from one unnecessary write-capable call, or from arguments that only worked by accident. I’d be curious whether you treat dangerous-but-unused calls as a hard gate rather than a score penalty.