How to Test Non-Deterministic AI Agents (When temperature=0 Isn't Enough)

Your test passed on Monday. Same input, same code, temperature=0. On Tuesday it failed, and you changed nothing. The assertion expected an exact string, but the model returned the same answer with slightly different wording. The agent is fine; the test is not.

Try Apidog today

This is the cost of testing language-model output. Even at temperature=0, you cannot rely on byte-identical responses across runs. Instead of testing exact wording, test the API contract: structure, required fields, valid ranges, tool-call payloads, and safety constraints. This is a deeper look at failure mode three in our guide to why AI agents break in production.

Why temperature=0 does not mean deterministic

Temperature controls token sampling. At zero, the model selects the most probable next token, which sounds reproducible. In practice, the serving stack introduces variation.

GPU floating-point math is not associative: adding the same values in a different order can produce small numerical differences. Those differences can change which token ranks highest. Once one token differs, every token after it can diverge.

The order of operations can vary based on:

• Request batching
• GPU hardware
• Inference kernels
• Provider routing and regions
• Model-serving library versions
• Weight quantization changes

Providers can also update infrastructure without changing your API request. As this vLLM discussion explains, a fixed seed and temperature=0 are not enough for bitwise reproducibility.

Determinism is a property of the entire serving stack, not a request parameter. Design tests accordingly.

Why exact-string assertions make tests flaky

This assertion is fragile:

assert.equal(response, "Your order total is $42.00.");

It fails if the model returns a correct variation:

Your total comes to $42.00.

The failure does not indicate a product regression. It indicates that the test is coupled to wording that is expected to vary.

Flaky tests create a predictable failure pattern:

1. Developers rerun the suite.
2. The test passes eventually.
3. The team stops trusting failures.
4. Real regressions get ignored.

We have covered what causes flaky tests and why they spread. Non-deterministic model output is one of the fastest ways to create them.

Do not make this worse by snapshotting full text responses. Replace exact-output checks with assertions on stable properties.

Assert on structure and meaning, not exact text

A valid support-agent response may be phrased many ways, but it should still contain the same contract-level facts:

• A refund amount
• An order ID
• A status
• A valid response shape

Shift your test question from:

Did the model say exactly this?

To:

Does the response have the expected structure, fields, types, and valid values?

The following strategies make that shift practical.

Validate responses with a JSON Schema

If your agent returns structured data, define a schema and validate every response against it.

For example, a refund result might require:

{
  "type": "object",
  "required": ["order_id", "status", "amount"],
  "properties": {
    "order_id": {
      "type": "string",
      "pattern": "^ord_[A-Za-z0-9]+$"
    },
    "status": {
      "type": "string",
      "enum": ["refunded", "pending", "denied"]
    },
    "amount": {
      "type": "number",
      "minimum": 0
    }
  },
  "additionalProperties": false
}

Then validate the response instead of comparing prose:

import Ajv from "ajv";

const ajv = new Ajv();
const validate = ajv.compile(refundSchema);

const valid = validate(agentResponse);

assert.ok(valid, JSON.stringify(validate.errors));

This catches failures that matter:

• Required fields are missing.
• A field has the wrong type.
• An enum contains an unsupported value.
• The model returns prose instead of JSON.
• The object is nested incorrectly.

Load your response schema into Apidog to validate live agent responses against the contract. A schema failure identifies the broken field instead of showing a long string diff.

Assert tool calls by shape and target

When an agent calls a tool, test the tool call rather than the natural-language text that preceded it.

For a booking agent, assert that it:

1. Selected the correct tool.
2. Sent the request to the correct endpoint.
3. Produced a payload that matches the tool schema.

Example:

assert.equal(toolCall.name, "createReservation");
assert.equal(toolCall.method, "POST");
assert.equal(toolCall.path, "/reservations");

assert.equal(typeof toolCall.arguments.guests, "number");
assert.ok(Number.isInteger(toolCall.arguments.guests));
assert.match(toolCall.arguments.date, /^\d{4}-\d{2}-\d{2}$/);

The wording may vary, but POST /reservations still needs an integer guests value and a valid date.

Apply the same schema discipline to outgoing requests that you use for incoming responses:

• Verify required parameters.
• Validate types and formats.
• Reject unexpected fields.
• Confirm the endpoint and method.

The end-to-end method for testing an agent’s API calls covers how to capture tool schemas and assert against them.

Use numeric ranges instead of exact values

For model-generated or model-transformed numbers, assert valid bounds instead of a single expected value.

For example, an order total should not be negative and should not exceed the cart subtotal plus a defined maximum for tax and shipping:

const maxTotal = cartSubtotal + maxTax + maxShipping;

assert.ok(response.total >= 0);
assert.ok(response.total <= maxTotal);

This catches meaningful failures:

• Negative totals
• Totals far above the allowed maximum
• Zero totals for non-empty carts
• Missing or non-numeric values

Use range assertions for:

• Confidence scores
• Item counts
• Token usage
• Latency budgets
• Prices and totals
• Retry counts

Choose the widest range that still fails for a real bug.

Check required keys and forbidden fields

Two low-cost assertions provide strong coverage:

1. Required fields exist and are non-null.
2. Sensitive or internal-only fields are absent.

For a support-ticket response:

assert.ok(response.resolution);
assert.notEqual(response.resolution, null);

assert.ok(!("internal_notes" in response));
assert.ok(!("raw_prompt" in response));

This verifies the response contract without caring how the model phrases the explanation.

Forbidden-field checks are also a useful privacy guard. They can catch accidental exposure of internal context, raw prompts, or notes that should never be sent to a customer.

Use semantic and threshold checks for free text

Sometimes your API must return prose. Do not use exact matching. Test measurable properties instead.

For example:

assert.ok(response.message.includes(orderId));
assert.ok(response.message.length <= 500);
assert.ok(!response.message.includes("internal system prompt"));

Useful free-text checks include:

• The response includes an order number or ticket ID.
• The response stays under a length limit.
• The response avoids prohibited phrases.
• The response includes required disclaimers.
• The response contains a URL from an allowlist.

If you need a coarse check for meaning, compare embeddings against a reference response and assert a minimum similarity score:

assert.ok(similarityScore >= 0.8);

Treat semantic similarity as a broad relevance gate, not a factual correctness guarantee. It can catch an answer that wandered off topic, but it may miss subtle factual errors. Pair it with structural validation and domain-specific assertions.

Snapshot stable properties, not complete text

Snapshot testing can still be useful if you snapshot the stable contract instead of the full response body.

Avoid this:

expect(response).toMatchSnapshot();

Prefer a normalized snapshot:

const stableResponse = {
  keys: Object.keys(response).sort(),
  status: response.status,
  hasOrderId: Boolean(response.order_id),
  amountInRange: response.amount >= 0 && response.amount <= maxTotal
};

expect(stableResponse).toMatchSnapshot();

Your snapshot should capture:

• Response keys
• Field types
• Enum values
• Required-field presence
• Allowed ranges
• Tool names and endpoints

This makes a snapshot fail on a structural change worth reviewing, not on a synonym.

State and memory make testing harder

The earlier examples assume one request and one response. Agents often carry state across turns.

A stateful agent can vary because of:

• Retrieved documents
• Stored memory
• Conversation history
• Earlier summaries
• The order of prior tool calls

Two runs of the same conversation can diverge if retrieval ranks documents differently or if a summary from turn two affects the model’s reasoning on turn five. The guide to how AI agent memory works explains where that state can live.

Use two habits to keep stateful tests reliable.

1. Seed known state before every test

Start each test from a controlled memory state:

beforeEach(async () => {
  await memoryStore.clear(testConversationId);

  await memoryStore.seed(testConversationId, [
    {
      role: "user",
      content: "My order ID is ord_123."
    }
  ]);
});

This removes one source of variation. Your test can focus on the behavior you actually want to verify.

2. Assert path-independent invariants

Test properties that must hold regardless of the exact conversational path.

Examples:

assert.ok(account.balance >= 0);
assert.equal(reservations.filter(r => r.flightId === flightId).length, 1);
assert.ok(ticket.resolution);

Useful invariants include:

• A balance never becomes negative.
• A completed booking creates exactly one reservation.
• A refund never exceeds the original payment.
• A support ticket reaches a terminal status.
• Sensitive fields never appear in customer-facing output.

These checks survive both model variation and state variation.

Mock dependencies so tests repeat

Do not rely on live third-party APIs for repeatable tests. External services can:

• Rate-limit requests
• Change their data
• Return intermittent errors
• Add network variability
• Introduce another source of non-determinism

Mock the dependencies the agent calls and return fixed responses.

For example, make a payment API always return the same receipt:

{
  "payment_id": "pay_test_001",
  "status": "captured",
  "amount": 42.0,
  "currency": "USD"
}

Make a search API always return the same result set:

{
  "results": [
    { "id": "doc_1", "title": "Refund policy" },
    { "id": "doc_2", "title": "Shipping policy" },
    { "id": "doc_3", "title": "Order status guide" }
  ]
}

With stable dependencies, the agent sees the same external inputs every run. You can also force edge cases that production APIs may not return on demand:

• Payment declined
• Empty search results
• Timeout responses
• Invalid payloads
• Rate-limit errors

Use Apidog to mock the agent’s API dependencies with controllable response bodies, then combine those mocks with schema and range assertions. This is part of broader agentic AI testing, where mocking and contract assertions work together.

Where Apidog fits—and where it does not

Be precise about the tool boundary.

Apidog is an API design, testing, and mocking platform. It is not:

• An agent framework
• A model host
• An agent runtime
• A reasoning evaluator
• An observability platform

It does not build, run, orchestrate, or score your agent.

It fits at the API boundary where your agent exchanges requests and responses. Use it to:

• Validate agent response schemas
• Check response shape and required fields
• Assert numeric ranges and enums
• Validate tool-call payloads
• Detect forbidden response fields
• Mock external dependencies with stable data

That is the useful seam: the contract around the model, not the model’s internal reasoning.

Test the contract, not the wording

Non-deterministic output is not a problem you can configure away. temperature=0 reduces sampling variation, but it does not guarantee identical output across runs.

Reliable agent tests focus on what should remain stable:

• Schema
• Response shape
• Required fields
• Forbidden fields
• Valid enums
• Numeric ranges
• Tool-call payloads
• State invariants

Let wording vary. Make the contract strict.

This week, take one flaky exact-string assertion and replace it with a schema check plus a range or presence assertion. Use Apidog to validate the response contract and mock the dependencies that prevent repeatable tests.