Deterministic Agent Contracts Without Making CI Brittle

The easiest agent test to write is often the most fragile:

expect(result.text).toBe("Your refund has been approved.");

Change the model, prompt punctuation, or tool response order and the assertion breaks—even if the user experience is still correct.

The opposite approach is equally dangerous: accept any final response that looks plausible and ignore the execution path. An agent could skip authorization, call a destructive tool twice, or answer without retrieval while the test remains green.

I maintain AgentInspect, an open-source TypeScript toolkit for inspecting agent runs locally. This article describes how I use deterministic trace contracts to test stable behavioral boundaries while leaving room for legitimate nondeterminism. The examples were verified against agent-inspect@6.17.4, where TraceContract is explicitly a beta API.

Contract the invariant, not the transcript

An agent run contains many kinds of variability:

• prose can change while meaning stays constant;
• tools may return results in different orders;
• a cache can remove a network call;
• a fallback can replace a primary path;
• latency and token counts naturally fluctuate.

A useful CI contract should focus on behavior that is both important and stable.

For a refund workflow, examples might include:

• the run must complete successfully;
• lookup_order must occur;
• delete_account must never occur;
• the agent may call at most four tools;
• retrieval must precede refund execution;
• recorded failure observations must fail the check.

These claims are deterministic because they can be evaluated from the captured trace. They do not ask an LLM to grade another LLM.

A small contract over a local trace

The release exposes defineTraceContract and evaluateTraceContractRead from agent-inspect/checks.

import {
  defineTraceContract,
  evaluateTraceContractRead,
} from "agent-inspect/checks";
import { openTraceFile } from "agent-inspect/readers";

const contract = defineTraceContract({
  run: {
    requireCompleted: true,
    allowedStatuses: ["ok"],
    maxDurationMs: 15_000,
  },
  tools: {
    required: ["lookup_order", "refund_order"],
    forbidden: ["delete_account"],
    maxCalls: 4,
    requiredOrder: ["lookup_order", "refund_order"],
  },
  llm: {
    maxCalls: 3,
    maxTotalTokens: 8_000,
    allowedModels: ["approved-model"],
  },
  observations: {
    failOn: ["failed", "unknown"],
  },
});

const trace = await openTraceFile("./fixtures/refund-run.jsonl");
const result = evaluateTraceContractRead(trace, contract);

if (result.status !== "pass") {
  console.error(result.findings);
  process.exitCode = 1;
}

The model name is intentionally synthetic. Configure the allowlist for your own environment rather than copying it.

The contract compiles to ordinary trace-check rules. It does not monitor the agent while the run is executing, and it does not stop a dangerous call before it happens. It evaluates persisted evidence afterward. Runtime policy enforcement belongs in the agent application or tool gateway.

What a useful failure looks like

AgentInspect includes synthetic broken and fixed contract fixtures. The broken run produces two evidence-bearing findings:

Check status: fail
Format: agent-inspect-v0.1-jsonl
Run: contract-broken
Summary: 2 failed, 0 warning(s), 0 error(s)
- [contract-broken] run.status: Run status error did not match expected ok.
- [contract-broken] tool.usage: Required tool refund_order did not appear.

The command exits with status 1, which makes it usable as a CI gate. The corrected fixture reports:

Check status: pass
Format: agent-inspect-v0.1-jsonl
Run: contract-fixed
Summary: 0 failed, 0 warning(s), 0 error(s)

A useful contract failure should identify the violated rule and point back to trace evidence. “The agent test failed” is not enough; the reviewer needs to know which behavioral promise was broken.

Avoid the “one golden path” trap

The first version of a trajectory test often looks like this:

classify -> retrieve -> generate -> validate

It is tempting to require that exact sequence for every run. Then a legitimate cache-hit path arrives:

classify -> load_cached_answer -> validate

Or a low-confidence path requires clarification:

classify -> ask_for_context

If the test permits only the original sequence, product improvement becomes test breakage.

The current requiredOrder surface is narrower than a workflow language. It turns adjacent entries into ordering rules. A contract such as:

tools: {
  requiredOrder: ["retrieve", "generate", "validate"],
}

checks retrieve before generate and generate before validate. Intermediate tools are allowed. It does not express alternatives such as “either retrieve or load from cache.” Model alternative paths as separate fixtures and contracts, or use lower-level rules suited to your application.

That limitation is important because a contract API should not be described as a full agent workflow engine when it is not one.

Read ordering semantics literally

In the referenced release, the tool-ordering rule uses the first occurrence of each named tool. This means:

retrieve -> generate -> retrieve

passes a retrieve before generate rule. The first retrieve appears before the first generate.

This is reasonable for a basic precedence check, but it may not match a stronger invariant such as “no retrieval is allowed after generation begins.” That stronger rule needs a custom check.

Before placing any contract in CI, test its semantics with at least these fixtures:

1. expected success path
2. expected violation
3. repeated-tool path
4. fallback or cache-hit path
5. incomplete or errored run

Treat the fixtures as executable documentation for what your team means by the contract.

A practical stability ladder

Not every possible assertion has the same CI value. I use this mental model:

usually stable                         often noisy
------------------------------------------------------------>
forbidden tool
required authorization
run completion
bounded call count
relative ordering
token ceiling
duration ceiling
exact output text

This is not universal. A hard latency budget can be stable in a controlled test, and exact text may be required for a protocol. The point is to choose thresholds based on controlled evidence, not intuition.

Start with safety and correctness boundaries

A forbidden destructive tool or missing authorization step is a strong candidate. It reflects product behavior rather than model style.

Use ceilings with headroom

Token and duration limits can catch runaway behavior, but thresholds placed directly at today’s observed value will create noise. Establish a representative fixture set and leave intentional margin.

Separate structural and semantic gates

Use deterministic trace checks for structure. Use domain assertions or evaluators for answer quality. Keep the reports separate so a failure says whether the path or the content was wrong.

Review green results

A passing contract proves only that the encoded rules passed for the inspected trace. It does not prove that the trace is representative, the response is correct, or the agent is safe in production.

CI should preserve evidence, not only a red icon

A failing job is much easier to diagnose when it uploads the trace, check results, and a readable report as artifacts.

An illustrative workflow looks like this:

- name: Run synthetic agent fixture
  run: npm run test:agent-fixture

- name: Evaluate trajectory contract
  run: node ./scripts/check-refund-contract.mjs

- name: Preserve evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: refund-agent-evidence
    path: |
      .agent-inspect/
      agent-inspect-report/

This GitHub Actions snippet is an integration pattern, not a copy of a shipped AgentInspect workflow. The local trace and contract result are the important pieces; adapt artifact retention to your security policy.

The contract is a reviewable engineering decision

The strongest benefit of a deterministic contract may be social rather than technical. It forces a team to state what must remain true across prompt, model, and tool changes.

Instead of debating whether a new run “looks okay,” reviewers can discuss concrete questions:

• Is retrieval always required, or can a cache satisfy the same obligation?
• Should a fallback be allowed in the release gate?
• Is the call ceiling protecting cost, latency, or both?
• Which tools are forbidden under every circumstance?

Those decisions are versionable. When product behavior legitimately changes, update the contract and its fixtures in the same review.

Make nondeterminism explicit, not invisible

Agent systems will remain variable. The goal of a CI contract is not to force every run into one transcript. It is to protect the small set of structural promises that should survive that variation.

Start with one synthetic failure, one corrected trace, and one high-value invariant. Confirm the exact beta semantics against the pinned release, then grow the contract only when a real failure mode justifies it.

The tagged implementation and contract documentation used for this article are available on GitHub. Feedback on missing rules and surprising semantics is particularly useful while this surface is still beta.``