How to Test AI Agents Without Calling More LLMs

AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score is stable enough for CI.

LLM judges can be useful for genuinely semantic questions. They are a poor default for verifying tool order, retry limits, validation, budgets, state transitions, or error handling. Those behaviors have explicit contracts and can usually be tested with ordinary deterministic techniques.

The most effective agent test suite separates orchestration correctness from language quality. Test the first on every commit with fakes, traces, and rules. Evaluate the second with a smaller, calibrated model-and-human workflow.

What Can Be Deterministic?

An agent may produce variable language while still following a predictable execution contract. Useful deterministic checks include:

• Input validation runs before any model or tool call.
• Only authorized tools are available for the current state.
• Retrieval completes before generation uses its results.
• Tool arguments match a schema.
• Retries stop after the configured limit.
• A timeout or cancellation ends the run.
• A fallback is used only for approved error categories.
• Token and tool-call budgets are enforced.
• Every started span is completed exactly once.
• Sensitive payloads are absent from the trace.

None of these questions requires a second model. Most do not require a first model either.

Use a Testing Pyramid for Agents

Layer	Model access	Purpose
Pure unit tests	None	Routing, validation, budgeting, state reducers
Orchestration tests	Scripted fake	Tool order, retries, fallbacks, termination
Tool contract tests	None or sandbox	Schemas, timeouts, error mapping
Trace replay tests	None	Regression checks against recorded execution metadata
Model integration tests	Real model	Provider compatibility and a small set of end-to-end paths
Semantic evaluations	Real model and/or human	Helpfulness, groundedness, tone, nuanced correctness

The lower layers should contain most of the suite. They are fast, reproducible, and actionable. The upper layers are valuable, but they should be deliberately smaller.

Put Models and Tools Behind Interfaces

Dependency injection makes the orchestration testable without network calls.

type ModelReply =
  | { type: 'tool_call'; tool: string; args: unknown }
  | { type: 'final'; text: string; usage: { input: number; output: number } };

interface ModelClient {
  generate(input: {
    messages: Array<{ role: string; content: string }>;
    tools: string[];
  }): Promise<ModelReply>;
}

interface ToolClient {
  execute(name: string, args: unknown): Promise<unknown>;
}

interface Clock {
  now(): number;
}

Production code receives real implementations. Tests provide scripted implementations with known behavior.

Script Model Behavior Instead of Generating It

A scripted model returns a predefined sequence and fails if the agent makes an unexpected extra call.

class ScriptedModel implements ModelClient {
  private index = 0;

  constructor(private readonly replies: ModelReply[]) {}

  async generate(): Promise<ModelReply> {
    const reply = this.replies[this.index];
    if (!reply) {
      throw new Error(`Unexpected model call at index ${this.index}`);
    }

    this.index += 1;
    return structuredClone(reply);
  }

  assertConsumed(): void {
    if (this.index !== this.replies.length) {
      throw new Error(
        `Expected ${this.replies.length} model calls, received ${this.index}`,
      );
    }
  }
}

This fake does not imitate model intelligence. It controls the branch that the orchestration must handle. One script can request a tool and then return a final answer; another can repeatedly request an invalid tool call to verify termination.

Record a Test-Friendly Trace

The trace should expose behavior without coupling tests to prose output.

type TraceStep = {
  sequence: number;
  name: string;
  kind: 'run' | 'model' | 'tool' | 'policy' | 'fallback';
  status: 'ok' | 'error' | 'blocked';
  parentId: string | null;
  attempt?: number;
  metadata?: Record<string, string | number | boolean | null>;
};

type AgentTrace = {
  status: 'ok' | 'error' | 'blocked';
  steps: TraceStep[];
};

Use a monotonic sequence assigned by the recorder for causal assertions. Do not infer retry behavior from consecutive names: parallel work can interleave events, and unrelated steps can separate attempts.

Keep the trace schema stable and metadata-first. Tests should assert execution contracts, not depend on raw prompts or complete tool results.

Build Reusable Trace Assertions

Small domain-specific helpers make failures easier to understand than generic array comparisons.

function stepIndex(trace: AgentTrace, name: string): number {
  return trace.steps.findIndex((step) => step.name === name);
}

function expectStepBefore(
  trace: AgentTrace,
  first: string,
  second: string,
): void {
  const firstIndex = stepIndex(trace, first);
  const secondIndex = stepIndex(trace, second);

  if (firstIndex < 0) throw new Error(`Missing trace step: ${first}`);
  if (secondIndex < 0) throw new Error(`Missing trace step: ${second}`);
  if (firstIndex >= secondIndex) {
    throw new Error(`Expected ${first} before ${second}`);
  }
}

function countSteps(trace: AgentTrace, name: string): number {
  return trace.steps.filter((step) => step.name === name).length;
}

When operations run in parallel, assert parentage and required dependencies rather than total ordering. Two sibling tools may complete in either order and both be correct.

Test Retrieval Before Generation

test('retrieves policy before generating an answer', async () => {
  const model = new ScriptedModel([
    {
      type: 'final',
      text: 'Use the documented return process.',
      usage: { input: 420, output: 18 },
    },
  ]);

  const { trace } = await runSupportAgent(
    { model, tools: fakeTools, clock: fakeClock },
    'How do returns work?',
  );

  expectStepBefore(trace, 'retrieve_policy', 'generate_answer');
  expect(countSteps(trace, 'retrieve_policy')).toBe(1);
  model.assertConsumed();
});

The test says exactly what failed: a missing step, an invalid dependency order, or an unexpected model call. No quality score is needed.

Test Retry Limits by Attempt

test('stops a tool after two failed attempts', async () => {
  const tools = new FakeTools({
    lookup_invoice: [
      new TimeoutError(),
      new TimeoutError(),
      { invoiceId: 'should-never-be-returned' },
    ],
  });

  const { trace } = await runInvoiceAgent(
    { model: scriptedToolCaller, tools, clock: fakeClock },
    'Find the latest invoice',
  );

  const attempts = trace.steps.filter(
    (step) => step.name === 'lookup_invoice',
  );

  expect(attempts.map((step) => step.attempt)).toEqual([1, 2]);
  expect(tools.calls('lookup_invoice')).toHaveLength(2);
  expect(trace.status).toBe('error');
});

This catches an unbounded retry loop without waiting for a real service or model.

Test Guardrails as Control Flow

A blocked request should produce no downstream model or tool activity.

test('blocks unauthorized export before external calls', async () => {
  const { trace } = await runExportAgent(
    { model: modelThatMustNotRun, tools: toolsThatMustNotRun, clock: fakeClock },
    { action: 'export_all_accounts', authorized: false },
  );

  expect(trace.status).toBe('blocked');
  expect(stepIndex(trace, 'authorize_export')).toBeGreaterThanOrEqual(0);
  expect(trace.steps.some((step) => step.kind === 'model')).toBe(false);
  expect(trace.steps.some((step) => step.kind === 'tool')).toBe(false);
});

The test uses structured input instead of asking a model to recognize a particular malicious phrase. A separate security suite can test prompt-injection resistance with representative text fixtures.

Make Time Deterministic

Wall-clock assertions are often flaky under CI load. Inject a fake clock or scheduler and advance it deliberately.

class FakeClock implements Clock {
  private value = 0;

  now(): number {
    return this.value;
  }

  advance(ms: number): void {
    this.value += ms;
  }
}

test('uses fallback after the configured timeout', async () => {
  const clock = new FakeClock();
  const tools = new FakeTools({ primary_search: [new TimeoutError()] });

  const { trace } = await runSearchAgent(
    { model: scriptedToolCaller, tools, clock },
    'pricing',
  );

  expect(stepIndex(trace, 'fallback_search')).toBeGreaterThanOrEqual(0);
});

Keep a small number of real timing tests for integration boundaries. Do not make every unit test depend on an overloaded runner finishing within an arbitrary number of milliseconds.

Test Budgets at the Enforcement Point

Model fakes can return explicit usage values. That lets you verify budget logic without paying for tokens.

test('stops before exceeding the session token budget', async () => {
  const model = new ScriptedModel([
    { type: 'final', text: 'first', usage: { input: 700, output: 200 } },
    { type: 'final', text: 'second', usage: { input: 700, output: 200 } },
  ]);

  const { trace } = await runBudgetedAgent(
    { model, tools: fakeTools, clock: fakeClock, maxTokens: 1_200 },
    'continue',
  );

  expect(trace.steps.some((step) => step.name === 'token_budget_block')).toBe(true);
  expect(countSteps(trace, 'model_call')).toBe(1);
});

This proves the enforcement behavior. A separate provider integration test can verify that production usage fields are mapped correctly into the internal token model.

Replay Traces for Regression Tests

Recorded metadata traces can drive tests without replaying sensitive content or calling a model. Store a compact fixture describing model decisions, tool outcomes, usage, and expected invariants.

Avoid treating the entire trace as a snapshot that must match byte for byte. IDs, timestamps, and harmless implementation details make snapshots brittle. Normalize the trace and assert durable properties:

• Required and forbidden steps
• Parent-child relationships
• Attempt limits
• Terminal status
• Budget totals
• Policy outcomes

Update a fixture only when the behavioral contract intentionally changes.

Test Failure Paths, Not Only Happy Paths

Agents fail at boundaries. Include fixtures for:

• Invalid model tool arguments
• Tool timeouts and rate limits
• Empty retrieval results
• Cancellation during streaming
• Partial tool success
• Unauthorized tool selection
• Context-budget exhaustion
• Trace-sink failure
• Fallback failure

Verify that the original error category is preserved, cleanup runs, and no further model or tool work occurs after a terminal state.

Where Real Models Still Belong

Deterministic tests cannot prove that an answer is clear, grounded, polite, or semantically correct. Real-model evaluation remains useful for those questions.

Use a curated dataset and an explicit rubric. Calibrate judge scores against human labels, track disagreement, and review threshold failures. Because model behavior can change, compare distributions and trends rather than treating one score from one run as an unquestionable fact.

A practical schedule is:

Trigger	Recommended checks
Local save	Unit and scripted orchestration tests
Every commit	Trace invariants, policy, retries, budgets, tool contracts
Pull request	Deterministic suite plus a small real-model smoke set
Nightly or weekly	Larger semantic evaluation and regression trends
Release candidate	Human review of high-risk or changed workflows

Teams with stricter cost or reliability requirements can move real-model tests out of pull requests entirely. The important point is to make that trade-off explicit.

Final Thought

Agent output is probabilistic, but agent architecture does not need to be opaque. Validation, tool access, retries, fallbacks, budgets, state transitions, and trace structure can all have deterministic contracts.

Put those contracts behind injectable interfaces, drive them with scripted dependencies, and assert behavior through a stable metadata trace. Save real models and LLM judges for the semantic questions that ordinary code cannot answer.

The next article will turn these ideas into fast CI quality gates with reusable trace rules, baseline comparison, and actionable failure reports.

Comments

[Giulio D'Erme]

The split between orchestration correctness and language quality is the right cut, and the monotonic sequence numbers in the traces are the part I would steal directly.

One number for the exception you carve out. Reserving LLM judges for genuinely semantic questions is reasonable, but that zone needs its own measurement before it can be trusted as a default. On LOCOMO's adversarial split, the standard judge accepts 62.8% of answers that were constructed to be wrong. Not ambiguous ones, not near misses, deliberately wrong ones. So "reserve the judge for semantics" can quietly become "the judge is now part of what you are measuring", unless its agreement rate is itself a number you publish next to the result.

The rung I would add to your pyramid is: did the agent correctly decline to answer? It looks like a clean deterministic contract, and at the contract level it is one. The problem is that the signal underneath is graded rather than binary. Measuring discrimination between answerable and unanswerable questions, sorted by how far the supporting evidence was removed, gives AUC 0.968 when the whole surrounding topic is gone and 0.567 when only the single supporting turn is, which is barely above chance. Both numbers describe the same system. So a green abstention test says much less than it appears to when the miss is a near one, and a pyramid cannot express that on its own.

Curious about one thing ahead of your CI piece. Could a trace rule assert the abstention decision the same way it asserts retrieval-before-generation ordering, or does a decision whose underlying confidence is continuous need a different layer than a contract test?

[Raju Dandigam]

@gde03, I would let a trace rule assert that an abstention decision occurred only when the runtime emits that decision as an observed fact. It cannot by itself prove that the underlying confidence estimate was well calibrated. The continuous discrimination question belongs in a separate evaluation layer with threshold or AUC reporting, while the contract can still verify that low-confidence cases followed the required abstain-or-escalate path.

[Giulio D'Erme]

That is a useful distinction, so I implemented it as a separate trace check.
The benchmark now records explicit runtime decisions such as answer, abstain, or escalate when the memory layer emits them. The contract checks whether low confidence led to abstention or escalation, while calibration and AUC remain separate evaluations.
I also replayed the official 003 artifacts. Across 2,536 admitted sessions, there were no structured runtime decisions recorded. That does not prove that no agent abstained. It shows that 003 cannot establish abstention behavior from its traces. The existing response keyword result therefore remains only a lower bound.
For future runs, this gives us a cleaner requirement: if we want to claim that a memory layer abstained, the layer has to emit that decision as an observable event, rather than leaving us to infer it from the final prose.

[Raju Dandigam]

@gde03, exactly—absence of a decision event is unknown evidence, not evidence of absence. Describing 003 as “cannot establish abstention behavior” is the defensible result. A typed decision: answer | abstain | escalate event would let the contract report behavior coverage explicitly while keeping calibration and discrimination metrics in their separate evaluation layer.

[leob]

Thanks, that's very detailed and (as far as I could see) solid and usable advice!

[Raju Dandigam]

@leob, thank you. The main goal was to keep the fast path usable in ordinary CI: fake the model and tools, capture the orchestration facts, and reserve slower model-graded evaluation for a much smaller semantic suite.

[Zira]

The partial-order point is important: a trace assertion should usually be a DAG of required dependencies, not a byte-for-byte event snapshot. One extra layer I have found useful is contract-preserving perturbation: replay the same scripted trace while varying tool latency, sibling completion order, and irrelevant model text, then assert the terminal status, budget, and policy decisions stay unchanged. That catches hidden timing and prompt-coupling bugs without another model call.

[Raju Dandigam]

@zira125, contract-preserving perturbation is a great addition. Varying sibling completion order and tool latency while holding the required dependency DAG constant is exactly how to expose accidental timing assumptions. It turns concurrency from an occasional production surprise into a deterministic regression fixture.

[Zira]

Yes, and I would make the perturbation itself part of the fixture’s evidence. Generate schedules that preserve the dependency DAG, then record the observed trace as a partial order rather than comparing raw timestamps. The regression should fail when a required happens-before edge disappears, not merely when wall-clock order changes. I’d also rerun the fixture with one worker paused after producing its output but before acknowledging completion; that exposes agents that confuse “result exists” with “stage is durably committed.”

[Raju Dandigam]

@zira125, recording the schedule as a partial order is the crucial improvement. The paused-before-acknowledgement fixture tests a real commit boundary: produced, persisted, and acknowledged need to remain distinct states. I’d generate several valid topological schedules from the same DAG and require invariant dependency edges and outcomes while allowing sibling timestamps to vary.