How to Test AI Agents Without Calling More LLMs

AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score is stable enough for CI.

LLM judges can be useful for genuinely semantic questions. They are a poor default for verifying tool order, retry limits, validation, budgets, state transitions, or error handling. Those behaviors have explicit contracts and can usually be tested with ordinary deterministic techniques.

The most effective agent test suite separates orchestration correctness from language quality. Test the first on every commit with fakes, traces, and rules. Evaluate the second with a smaller, calibrated model-and-human workflow.

What Can Be Deterministic?

An agent may produce variable language while still following a predictable execution contract. Useful deterministic checks include:

• Input validation runs before any model or tool call.
• Only authorized tools are available for the current state.
• Retrieval completes before generation uses its results.
• Tool arguments match a schema.
• Retries stop after the configured limit.
• A timeout or cancellation ends the run.
• A fallback is used only for approved error categories.
• Token and tool-call budgets are enforced.
• Every started span is completed exactly once.
• Sensitive payloads are absent from the trace.

None of these questions requires a second model. Most do not require a first model either.

Use a Testing Pyramid for Agents

Layer	Model access	Purpose
Pure unit tests	None	Routing, validation, budgeting, state reducers
Orchestration tests	Scripted fake	Tool order, retries, fallbacks, termination
Tool contract tests	None or sandbox	Schemas, timeouts, error mapping
Trace replay tests	None	Regression checks against recorded execution metadata
Model integration tests	Real model	Provider compatibility and a small set of end-to-end paths
Semantic evaluations	Real model and/or human	Helpfulness, groundedness, tone, nuanced correctness

The lower layers should contain most of the suite. They are fast, reproducible, and actionable. The upper layers are valuable, but they should be deliberately smaller.

Put Models and Tools Behind Interfaces

Dependency injection makes the orchestration testable without network calls.

type ModelReply =
  | { type: 'tool_call'; tool: string; args: unknown }
  | { type: 'final'; text: string; usage: { input: number; output: number } };

interface ModelClient {
  generate(input: {
    messages: Array<{ role: string; content: string }>;
    tools: string[];
  }): Promise<ModelReply>;
}

interface ToolClient {
  execute(name: string, args: unknown): Promise<unknown>;
}

interface Clock {
  now(): number;
}

Production code receives real implementations. Tests provide scripted implementations with known behavior.

Script Model Behavior Instead of Generating It

A scripted model returns a predefined sequence and fails if the agent makes an unexpected extra call.

class ScriptedModel implements ModelClient {
  private index = 0;

  constructor(private readonly replies: ModelReply[]) {}

  async generate(): Promise<ModelReply> {
    const reply = this.replies[this.index];
    if (!reply) {
      throw new Error(`Unexpected model call at index ${this.index}`);
    }

    this.index += 1;
    return structuredClone(reply);
  }

  assertConsumed(): void {
    if (this.index !== this.replies.length) {
      throw new Error(
        `Expected ${this.replies.length} model calls, received ${this.index}`,
      );
    }
  }
}

This fake does not imitate model intelligence. It controls the branch that the orchestration must handle. One script can request a tool and then return a final answer; another can repeatedly request an invalid tool call to verify termination.

Record a Test-Friendly Trace

The trace should expose behavior without coupling tests to prose output.

type TraceStep = {
  sequence: number;
  name: string;
  kind: 'run' | 'model' | 'tool' | 'policy' | 'fallback';
  status: 'ok' | 'error' | 'blocked';
  parentId: string | null;
  attempt?: number;
  metadata?: Record<string, string | number | boolean | null>;
};

type AgentTrace = {
  status: 'ok' | 'error' | 'blocked';
  steps: TraceStep[];
};

Use a monotonic sequence assigned by the recorder for causal assertions. Do not infer retry behavior from consecutive names: parallel work can interleave events, and unrelated steps can separate attempts.

Keep the trace schema stable and metadata-first. Tests should assert execution contracts, not depend on raw prompts or complete tool results.

Build Reusable Trace Assertions

Small domain-specific helpers make failures easier to understand than generic array comparisons.

function stepIndex(trace: AgentTrace, name: string): number {
  return trace.steps.findIndex((step) => step.name === name);
}

function expectStepBefore(
  trace: AgentTrace,
  first: string,
  second: string,
): void {
  const firstIndex = stepIndex(trace, first);
  const secondIndex = stepIndex(trace, second);

  if (firstIndex < 0) throw new Error(`Missing trace step: ${first}`);
  if (secondIndex < 0) throw new Error(`Missing trace step: ${second}`);
  if (firstIndex >= secondIndex) {
    throw new Error(`Expected ${first} before ${second}`);
  }
}

function countSteps(trace: AgentTrace, name: string): number {
  return trace.steps.filter((step) => step.name === name).length;
}

When operations run in parallel, assert parentage and required dependencies rather than total ordering. Two sibling tools may complete in either order and both be correct.

Test Retrieval Before Generation

test('retrieves policy before generating an answer', async () => {
  const model = new ScriptedModel([
    {
      type: 'final',
      text: 'Use the documented return process.',
      usage: { input: 420, output: 18 },
    },
  ]);

  const { trace } = await runSupportAgent(
    { model, tools: fakeTools, clock: fakeClock },
    'How do returns work?',
  );

  expectStepBefore(trace, 'retrieve_policy', 'generate_answer');
  expect(countSteps(trace, 'retrieve_policy')).toBe(1);
  model.assertConsumed();
});

The test says exactly what failed: a missing step, an invalid dependency order, or an unexpected model call. No quality score is needed.

Test Retry Limits by Attempt

test('stops a tool after two failed attempts', async () => {
  const tools = new FakeTools({
    lookup_invoice: [
      new TimeoutError(),
      new TimeoutError(),
      { invoiceId: 'should-never-be-returned' },
    ],
  });

  const { trace } = await runInvoiceAgent(
    { model: scriptedToolCaller, tools, clock: fakeClock },
    'Find the latest invoice',
  );

  const attempts = trace.steps.filter(
    (step) => step.name === 'lookup_invoice',
  );

  expect(attempts.map((step) => step.attempt)).toEqual([1, 2]);
  expect(tools.calls('lookup_invoice')).toHaveLength(2);
  expect(trace.status).toBe('error');
});

This catches an unbounded retry loop without waiting for a real service or model.

Test Guardrails as Control Flow

A blocked request should produce no downstream model or tool activity.

test('blocks unauthorized export before external calls', async () => {
  const { trace } = await runExportAgent(
    { model: modelThatMustNotRun, tools: toolsThatMustNotRun, clock: fakeClock },
    { action: 'export_all_accounts', authorized: false },
  );

  expect(trace.status).toBe('blocked');
  expect(stepIndex(trace, 'authorize_export')).toBeGreaterThanOrEqual(0);
  expect(trace.steps.some((step) => step.kind === 'model')).toBe(false);
  expect(trace.steps.some((step) => step.kind === 'tool')).toBe(false);
});

The test uses structured input instead of asking a model to recognize a particular malicious phrase. A separate security suite can test prompt-injection resistance with representative text fixtures.

Make Time Deterministic

Wall-clock assertions are often flaky under CI load. Inject a fake clock or scheduler and advance it deliberately.

class FakeClock implements Clock {
  private value = 0;

  now(): number {
    return this.value;
  }

  advance(ms: number): void {
    this.value += ms;
  }
}

test('uses fallback after the configured timeout', async () => {
  const clock = new FakeClock();
  const tools = new FakeTools({ primary_search: [new TimeoutError()] });

  const { trace } = await runSearchAgent(
    { model: scriptedToolCaller, tools, clock },
    'pricing',
  );

  expect(stepIndex(trace, 'fallback_search')).toBeGreaterThanOrEqual(0);
});

Keep a small number of real timing tests for integration boundaries. Do not make every unit test depend on an overloaded runner finishing within an arbitrary number of milliseconds.

Test Budgets at the Enforcement Point

Model fakes can return explicit usage values. That lets you verify budget logic without paying for tokens.

test('stops before exceeding the session token budget', async () => {
  const model = new ScriptedModel([
    { type: 'final', text: 'first', usage: { input: 700, output: 200 } },
    { type: 'final', text: 'second', usage: { input: 700, output: 200 } },
  ]);

  const { trace } = await runBudgetedAgent(
    { model, tools: fakeTools, clock: fakeClock, maxTokens: 1_200 },
    'continue',
  );

  expect(trace.steps.some((step) => step.name === 'token_budget_block')).toBe(true);
  expect(countSteps(trace, 'model_call')).toBe(1);
});

This proves the enforcement behavior. A separate provider integration test can verify that production usage fields are mapped correctly into the internal token model.

Replay Traces for Regression Tests

Recorded metadata traces can drive tests without replaying sensitive content or calling a model. Store a compact fixture describing model decisions, tool outcomes, usage, and expected invariants.

Avoid treating the entire trace as a snapshot that must match byte for byte. IDs, timestamps, and harmless implementation details make snapshots brittle. Normalize the trace and assert durable properties:

• Required and forbidden steps
• Parent-child relationships
• Attempt limits
• Terminal status
• Budget totals
• Policy outcomes

Update a fixture only when the behavioral contract intentionally changes.

Test Failure Paths, Not Only Happy Paths

Agents fail at boundaries. Include fixtures for:

• Invalid model tool arguments
• Tool timeouts and rate limits
• Empty retrieval results
• Cancellation during streaming
• Partial tool success
• Unauthorized tool selection
• Context-budget exhaustion
• Trace-sink failure
• Fallback failure

Verify that the original error category is preserved, cleanup runs, and no further model or tool work occurs after a terminal state.

Where Real Models Still Belong

Deterministic tests cannot prove that an answer is clear, grounded, polite, or semantically correct. Real-model evaluation remains useful for those questions.

Use a curated dataset and an explicit rubric. Calibrate judge scores against human labels, track disagreement, and review threshold failures. Because model behavior can change, compare distributions and trends rather than treating one score from one run as an unquestionable fact.

A practical schedule is:

Trigger	Recommended checks
Local save	Unit and scripted orchestration tests
Every commit	Trace invariants, policy, retries, budgets, tool contracts
Pull request	Deterministic suite plus a small real-model smoke set
Nightly or weekly	Larger semantic evaluation and regression trends
Release candidate	Human review of high-risk or changed workflows

Teams with stricter cost or reliability requirements can move real-model tests out of pull requests entirely. The important point is to make that trade-off explicit.

Final Thought

Agent output is probabilistic, but agent architecture does not need to be opaque. Validation, tool access, retries, fallbacks, budgets, state transitions, and trace structure can all have deterministic contracts.

Put those contracts behind injectable interfaces, drive them with scripted dependencies, and assert behavior through a stable metadata trace. Save real models and LLM judges for the semantic questions that ordinary code cannot answer.

The next article will turn these ideas into fast CI quality gates with reusable trace rules, baseline comparison, and actionable failure reports.

Comments

[Zira]

The partial-order point is important: a trace assertion should usually be a DAG of required dependencies, not a byte-for-byte event snapshot. One extra layer I have found useful is contract-preserving perturbation: replay the same scripted trace while varying tool latency, sibling completion order, and irrelevant model text, then assert the terminal status, budget, and policy decisions stay unchanged. That catches hidden timing and prompt-coupling bugs without another model call.