DEV Community: Jack M

AI Agent Status Updates: Build Progress Users Can Trust Without Exposing Chain of Thought

Jack M — Mon, 26 Oct 2026 03:34:16 +0000

An agent that goes quiet for ten minutes does not feel autonomous. It feels broken. A spinner and a made-up “82% complete” label are not much better: they encourage users to trust a number your system cannot justify.

The fix is not to stream the model's hidden reasoning. It is to publish a small, evidence-backed account of the work: what phase the run is in, the last verified milestone, whether it needs a person, and what will happen next. This article shows how to build that contract for long-running, tool-using AI agents.

The progress problem is really a trust problem

Users ask three practical questions while an agent runs:

Is it still alive?
Did it make meaningful progress?
Do I need to do anything?

A token stream answers none of these reliably. It can continue while a tool is stuck, and it can stop even though a durable workflow will resume later. Likewise, an LLM-generated status such as “almost finished” is a prediction, not evidence.

Treat status as a product contract. The user-facing timeline should be derived from workflow events your system can verify: a job was accepted, a connector returned records, validation passed, an approval is waiting, or a final artifact was stored. The model may suggest a label, but it should not be the source of truth for state.

This distinction also protects privacy. A useful update is “Checked 42 invoices; 3 need review,” not a transcript of the agent's scratch work, retrieved customer text, or chain of thought.

Start with a state machine, not a progress bar

Long-running agent work has more states than running and failed. Model the states explicitly and make transitions durable.

State	Meaning to the user	Required evidence	Useful UI action
`queued`	Work is accepted but has not started	run record created	Cancel
`running`	A worker currently owns the run	lease heartbeat	Stop
`waiting_on_tool`	The run is awaiting a bounded external call	tool request ID and deadline	Cancel
`waiting_on_user`	A decision or missing input is required	approval or question ID	Review / answer
`retry_scheduled`	The system will retry later	retry time and error class	Cancel / view details
`succeeded`	The requested result was verified and saved	output receipt	Open result
`failed`	The run cannot continue automatically	failure class and last checkpoint	Retry / contact support
`cancelled`	Work was intentionally stopped	cancellation receipt	Restart

Avoid a vague paused state. If the workflow is paused because it needs an address, show waiting_on_user. If it is paused because a vendor rate-limited it, show retry_scheduled. Specific state names let a user decide what to do without reading logs.

Here is a compact TypeScript event shape. It is deliberately about observable work, not hidden reasoning.

type RunEvent = {
  runId: string;
  sequence: number;          // monotonic per run
  at: string;
  type: "phase_started" | "milestone" | "approval_needed" |
        "retry_scheduled" | "completed" | "failed";
  phase?: "preparing" | "collecting" | "verifying" | "delivering";
  message: string;           // reviewed, user-safe copy
  evidence?: {
    kind: "artifact" | "tool_receipt" | "checkpoint" | "approval";
    id: string;
  };
  action?: { label: string; url: string };
};

Persist these events in the same transaction that changes workflow state, or attach them to an outbox. A UI event that says “report ready” before the report is stored is worse than no event at all.

Publish milestones, not a narrated thought process

Choose milestones at boundaries where the system can prove something changed. Good milestones are stable across model versions and useful if a run resumes tomorrow.

For a document-analysis agent, the timeline might be:

Request accepted
Source file scanned and indexed
42 records extracted
Policy checks completed
Three exceptions need a decision
Final report saved

Notice what is absent: “thinking about clause 7” or “I believe this source is probably relevant.” Those claims are neither a dependable progress signal nor an appropriate audit record.

Each milestone needs an owner. The workflow engine owns checkpoint messages; the connector owns data-fetch receipts; the policy service owns validation results. If no component can produce evidence for an update, leave it out.

That rule prevents a common failure mode: a chat interface keeps emitting cheerful prose while the actual job stopped moving. Your supervisor should detect stale progress independently of the agent. For example, mark a run stalled internally when its lease heartbeat is fresh but neither a checkpoint nor a tool receipt has advanced within the expected phase window. The user-facing text can then say, “Still working longer than expected; no action needed yet,” or escalate honestly to “This task needs attention.”

Design an honest progress model

Percent complete works only when units of work are known. A migration over 10,000 rows has a denominator. An open-ended research task generally does not.

Use one of these models instead:

Work shape	Best indicator	Example
Fixed batch	completed / total	`318 of 500 files checked`
Known workflow	completed phase	`Verifying findings (3 of 4 phases)`
Open-ended work	elapsed time plus latest verified event	`Last verified update: sources indexed`
Approval flow	blocked state and decision impact	`Needs approval before sending 3 emails`
Retryable external work	next retry and preserved checkpoint	`Retrying connector at 14:05; 18 items saved`

Do not convert phase count into a false percentage. Four phases do not necessarily take equal time, and the model can loop inside one phase. If you show an estimate, label it as a range derived from similar completed runs, and hide it when the sample is weak.

function statusCopy(run: Run) {
  if (run.status === "waiting_on_user")
    return "Your decision is needed before this task can continue.";

  if (run.completedUnits != null && run.totalUnits != null)
    return `${run.completedUnits} of ${run.totalUnits} items verified`;

  return `Last verified update: ${run.latestMilestone.message}`;
}

The word verified matters. It teaches users what the timeline represents and sets a standard your team can test.

Make waiting visible and actionable

The most damaging silence is not a slow model call. It is an agent waiting for a person while the product looks busy.

An approval event should include the decision, the consequence, the safe default, and an expiry rule. Keep the payload concise and reviewable:

{
  "type": "approval_needed",
  "message": "Three customer records have conflicting addresses.",
  "question": "Which address should be used for the renewal notice?",
  "options": ["CRM record", "latest support ticket", "skip these records"],
  "default": "skip these records",
  "expires_at": "2026-10-26T15:00:00Z",
  "evidence": { "kind": "approval", "id": "apr_8f2" }
}

Never bury an approval in a scrolling chat. Surface it as a run state, send a notification only when action is required, and ensure a resumed run re-reads the approved decision from durable storage. This pairs well with approval gates, but it solves a different problem: making the wait understandable to the person who initiated the work.

Transport updates without building a fragile realtime system

Start with the simplest transport that fits your product.

Polling is enough for minute-scale background jobs. Return status, latest_event, and etag; back off when nothing changes.
Server-sent events work well for one-way timelines and reconnect cleanly with Last-Event-ID.
WebSockets fit collaborative workspaces where users can also approve, stop, or redirect a run live.
Notifications should be sparse: completion, failure, and waiting_on_user are normally worth interrupting someone; routine milestones are not.

Regardless of transport, the client must tolerate duplicate and out-of-order events. Use the sequence field, make the event stream append-only, and fetch the authoritative run snapshot after reconnecting. The UI is a projection; the workflow database remains the authority.

Build the API around a durable run record

A small API surface is easier to secure and test than a channel full of ad hoc text.

GET /runs/run_123

{
  "status": "waiting_on_user",
  "phase": "verifying",
  "latest_event_sequence": 17,
  "latest_event": {
    "message": "42 records verified; 3 need a decision",
    "at": "2026-10-26T12:04:12Z"
  },
  "actions": [{ "kind": "approve", "approval_id": "apr_8f2" }]
}

Keep sensitive tool arguments, raw documents, and internal exception stacks behind a permissioned diagnostics view. The default run page should disclose only the minimum a user needs to understand progress and take the next action.

Test the status contract like an API

Status is part of the user experience, so add tests beyond the happy path:

A run cannot become succeeded without an output receipt.
A waiting_on_user run always exposes a decision action and cannot spend tools until it resumes.
Replayed events create the same final timeline.
A reconnect receives no duplicate visible milestones.
A cancelled run cannot later publish a success event.
A stale worker lease produces an honest recovery state, not permanent “Working…” copy.
User-safe messages contain no prompt, secret, customer text, or chain of thought.

it("does not claim success before output exists", async () => {
  await workflow.emit({ type: "completed", evidence: undefined });
  expect(await runs.get("run_123")).toMatchObject({ status: "failed" });
});

This is also where observability earns its keep. Measure time to first verified milestone, time spent waiting for users, stale-run recovery rate, and the share of runs that finish with a useful artifact. Avoid treating the number of progress messages as a health metric; a noisy timeline can conceal a poor workflow.

A practical rollout plan

Do not retrofit every agent at once. Start with one workflow where users currently refresh a page or ask support whether a job is stuck.

Define the state machine and ownership of each transition.
Add durable run records and one evidence-backed milestone per phase.
Replace generic spinners with the latest verified update.
Promote approval waits to a clear, resumable state.
Add stale-run detection and a recovery message.
Instrument completion quality, not just elapsed time.

The result is deliberately less theatrical than a stream of simulated thinking. It is far more useful: people can leave a long-running agent alone, return with context, and intervene exactly when their judgment is needed.

Keep the language stable across releases

Treat visible status copy as part of your API. Changing waiting_on_user from “Needs your decision” to “Paused” may seem cosmetic, yet it can break notification rules, customer runbooks, and accessibility labels. Version the event schema, localize messages at the presentation layer, and keep a small compatibility test suite for every public state. That discipline pays off when new models, tools, and workflow engines arrive: the user still receives the same clear promise about what has happened and what they can do next.

FAQ

Should an AI agent show its chain of thought as progress?

No. Show concise, evidence-backed milestones instead. They are more reliable, safer for sensitive data, and easier for users to act on.

How often should an agent send status updates?

Update on meaningful state changes, verified milestones, and action-required moments. For long silent phases, add a bounded heartbeat that says the job is still owned, not a stream of filler messages.

Is a percentage complete useful for AI agents?

Only when the work has a known denominator, such as files processed or records checked. Otherwise show the phase, elapsed time, and most recent verified milestone.

What is the best state for an agent waiting on a person?

Use a specific waiting_on_user state with the exact question, a safe default, and a resumable approval ID. Do not leave the run labeled running.

Can polling work for long-running AI workflows?

Yes. Polling is simple and reliable for background tasks that update every few seconds or minutes. Use ETags and backoff, then move to SSE or WebSockets only when the interaction requires it.

How do you detect an AI agent that is stuck?

Track worker leases, tool deadlines, and the time since the last verified milestone. A supervisor outside the model should classify stale runs and surface a recovery state.

AI Agent Trace Replay: Turn Production Failures Into Safe Regression Tests

Jack M — Sun, 25 Oct 2026 03:28:32 +0000

Your agent failed once in production. The tempting fix is to patch the prompt, glance at the next successful run, and move on. That is how the same failure returns after a model, tool, or routing change.

The more useful response is to turn the failed run into a replayable regression test. Not a full rerun against live customer data. Not a screen recording. A small, scrubbed fixture that preserves the decisions and tool boundary that mattered, then proves the fix without sending another email, creating another ticket, or charging another card.

This guide shows how to build that pipeline. It is aimed at engineers operating tool-using agents in production. The result is a durable feedback loop: incidents become tests, and tests make the next release safer.

What agent trace replay is — and is not

An agent trace is an ordered record of a run: user intent, policy version, prompt revision, model calls, tool requests, tool responses, retry decisions, and final outcome. Trace replay uses a captured run as controlled input for a later test.

That definition matters because replay can mean different things:

Replay mode	What stays fixed	What you learn
Deterministic fixture replay	Tool outputs and selected model output	Whether orchestration, parsing, policies, and recovery still work
Model-in-the-loop replay	Tool outputs are fixed; model runs again	Whether a prompt/model change preserves the expected trajectory
Load replay	Request timing and token shapes	Whether serving latency, cache behavior, and capacity changed
Live rerun	Almost nothing	Usually too risky for debugging a customer incident

For a regression suite, start with the first mode. It makes a production bug repeatable without repeating the side effect. Add model-in-the-loop replay only after you can score it reliably.

This is different from a generic evaluation harness. An evaluation suite begins with representative tasks. Trace replay begins with a real failure, preserves the evidence that explains it, and turns a specific incident into a release gate. It is also different from event sourcing: the goal here is not to reconstruct every business object, but to create a minimal, privacy-safe test case for the agent behavior.

The failure pattern that replay catches

Imagine an operations agent that receives a request to pause a subscription. A billing lookup times out, the agent retries with a malformed account identifier, then tells the customer the subscription was paused. The retry did not change the account, and the customer is still billed.

An answer-quality check may pass because the message is polite. A unit test for the billing tool may pass because the tool validates normal inputs. The failure lives in the trajectory: timeout, incorrect fallback identifier, write attempt, unverified success claim.

A replay fixture lets you test that exact sequence:

Return the recorded timeout from the first lookup.
Provide the recorded account data from the fallback lookup.
Require an operation receipt before a success response is allowed.
Assert that the agent either verifies the pause or reports an unresolved state.

The key idea is simple: assert what the agent did and what it was allowed to claim, not only its final prose.

Capture the smallest useful trace

Do not dump every prompt and response into a test repository. A useful capture is intentional and bounded. Store only what you need to reproduce the decision boundary.

type TraceEvent = {
  runId: string;
  sequence: number;
  kind: "input" | "model" | "tool_call" | "tool_result" | "policy" | "outcome";
  at: string;
  version: { prompt: string; model: string; tools: string; policy: string };
  payloadRef?: string;       // encrypted or redacted fixture reference
  tool?: { name: string; operationId?: string; idempotencyKey?: string };
  digest: string;            // tamper-evident hash of canonical event data
};

Use a separate protected store for raw payloads if you genuinely need them. The trace used for test selection can usually carry redacted values, stable identifiers, schemas, response classes, and hashes. Never record chain-of-thought. You do not need it to verify tool order, policy decisions, structured arguments, or an authoritative receipt.

Capture these fields at minimum:

A correlation ID distinct from the customer and conversation IDs.
Versions for the prompt, model, tool schema, retrieval corpus, and runtime policy.
Tool arguments after validation, with secrets and personal data replaced by typed placeholders.
Tool result class: success, retryable failure, permanent failure, unknown outcome, or rejected.
External operation ID and authoritative verification result for every write.
The final customer-visible claim and the rule that allowed it.

The recording should be append-only. If a trace is edited for a fixture, retain its source run ID and record each redaction. Otherwise a convenient test fixture can quietly stop representing the incident that justified it.

Build a side-effect firewall before replaying anything

The dangerous mistake is to point a replay at production tools and assume an idempotency key makes it safe. Idempotency reduces duplicate writes; it does not make an old action appropriate today.

Use a replay adapter that has an explicit mode and a deny-by-default tool policy.

const SAFE_REPLAY_TOOLS = new Set(["customer.lookup", "billing.status"]);

async function invokeForReplay(call: ToolCall, fixture: FixtureStore) {
  if (!SAFE_REPLAY_TOOLS.has(call.name)) {
    return {
      status: "blocked",
      reason: `Replay may not execute ${call.name}`,
      required: "recorded_fixture_or_verified_sandbox"
    };
  }

  const recorded = await fixture.match(call.name, call.normalizedArgs);
  if (!recorded) return { status: "missing_fixture" };
  return recorded.result;
}

Treat writes, browser actions, emails, payment actions, deployments, and permission changes as blocked by default. For a write you need to model, return a recorded receipt or run against a dedicated sandbox with synthetic identities. A good replay failure is missing_fixture; it tells you the test is no longer a faithful simulation. A bad replay silently calls the real service.

Turn an incident into a compact fixture

Raw production runs are noisy. Reduce them in a reviewable sequence.

1. Name the observable failure

Avoid labels such as “agent confused.” Write the event that should never recur: “The agent claimed a subscription was paused without a successful pause receipt.” This becomes the primary assertion.

2. Keep the causal path, not every span

Retain the user request, relevant retrieval result, tool calls, tool responses, policy decisions, and final output. Remove unrelated telemetry and tool calls that cannot affect the outcome. If deletion changes the failure, put it back.

3. Replace sensitive values consistently

Map alex@example.com to user_01@example.invalid everywhere. Preserve the shape, locale, formats, and relationships that affected parsing. Do not replace a malformed identifier with a clean one when malformed input triggered the bug.

4. Freeze external responses

Represent each tool result as a fixture, including error bodies and headers the agent reads. A generic 500 does not reproduce a timeout branch that contained a retry hint.

5. Version the expected contract

Keep the allowed trajectory with the fixture. The contract can specify required calls, forbidden calls, maximum retries, expected state, and required evidence for a success claim.

case: pause-with-unknown-outcome
must_call:
  - customer.lookup
  - billing.pause
  - billing.get_operation
forbid_call:
  - customer.notify_success
max_calls:
  billing.pause: 1
assert:
  final_state: unresolved_or_verified_paused
  success_requires: billing.get_operation.status == completed

This format is intentionally boring. It lets a reviewer reason about behavior without reverse-engineering a prompt transcript.

Use layered assertions, not one brittle snapshot

An exact full transcript comparison makes every harmless wording change look like a regression. A single LLM judge makes every serious policy violation subjective. Combine layers instead.

Structural assertions: valid tool names, JSON schema, required fields, and no secret-like values in output.
Trajectory assertions: tool order, retry ceiling, no forbidden tool, and approval step before a sensitive action.
State assertions: an independent fixture or sandbox read-back confirms the intended business state.
Outcome assertions: the response does not promise more than evidence supports.
Semantic assertions: use a calibrated judge only for meaning that cannot be expressed deterministically, such as whether an explanation is understandable.

Here is a deliberately small trajectory test:

expect(events.toolCalls.map(x => x.name)).toEqual([
  "customer.lookup", "billing.pause", "billing.get_operation"
]);
expect(events.toolCalls.filter(x => x.name === "billing.pause")).toHaveLength(1);
expect(events.final.claim).not.toMatch(/paused successfully/i);
expect(events.final.claim).toMatch(/still verifying|could not confirm/i);

Prefer stable properties over wording. If your product truly needs a specific compliance phrase, assert that phrase separately and explain why.

Make model-in-the-loop replay measurable

Eventually you will want to replay the same fixture after changing a model or prompt. That run is non-deterministic, so design for variance instead of pretending it does not exist.

Run the case several times at the production-like temperature. Require every run to satisfy safety invariants, then set a threshold for softer outcomes. For example, all five runs must avoid an unverified success claim, four of five must choose the verified recovery path, and a human reviews any new tool sequence before it becomes accepted behavior.

Record the model and decoding settings with the result. A changed temperature, tool description, or retrieval revision can cause divergence even when the prompt file is untouched. If a replay diverges, classify it: acceptable alternative, new regression, fixture mismatch, or insufficient specification. Do not hide divergence by updating the snapshot automatically.

Put replay in the release path

Not every captured trace belongs in pull-request CI. Start with a small “incident corpus”: the highest-impact, recent, and recurring failures. Tag each case by workflow, risk, tenant shape, and the dependency it guards.

Run the cheap deterministic fixture suite on every change that touches prompts, policies, tool schemas, orchestration, or response rendering. Run model-in-the-loop cases on a scheduled or pre-release job where repeated samples are affordable. Run load replay separately; it answers latency and capacity questions, not behavioral correctness.

When a replay fails, show a compact diff:

case: pause-with-unknown-outcome
expected: verify operation before customer success claim
actual:   billing.pause -> customer.notify_success
first divergence: sequence 7, policy "receipt_required" skipped
changed inputs: prompt v42 -> v43; tool schema unchanged

That output is useful to the engineer on call. A wall of token traces is not.

Common replay traps

Treating replay as proof of live behavior. A fixture proves behavior against its captured conditions. It does not prove correctness for new customer data, a changed upstream API, or a different model response. Keep ordinary integration tests and sampled production monitoring.

Saving raw customer data forever. Expiring raw captures and retaining redacted fixtures reduces risk. Define who can access source traces and how a fixture is deleted when its underlying data must be removed.

Mocking away the failure. If the incident involved a malformed response or a missing field, preserve it. A happy-path mock creates false confidence.

Testing only the final answer. Agents can say the right thing after taking the wrong action. Test the trajectory and authoritative state.

Letting fixtures drift. Give every fixture an owner, source trace, schema version, and review date. When a tool contract changes, decide whether the old incident remains a valid regression or should be migrated with an explicit explanation.

A practical first week

Pick one incident where the agent caused an incorrect claim, a repeated action, an unsafe call, or a costly loop. Capture the trace, redact it, block all side effects, and write three deterministic assertions: one forbidden action, one required evidence check, and one final-state rule. Put that one case in CI.

Then repeat for the next real incident. In a few weeks, you will have a test corpus that reflects the failures your system actually has—not a generic benchmark someone else designed. That is the durable value of replay: each incident makes the next release a little harder to break in the same way.

FAQ

What is AI agent trace replay?

AI agent trace replay reruns a captured agent scenario using recorded or simulated inputs. It is used to reproduce a failure, compare behavior after a change, and make an incident into a regression test without invoking real side effects.

Should an agent replay call real tools?

Usually no. Default to recorded fixtures and deny real writes, emails, payments, deployments, and permission changes. Use an isolated sandbox with synthetic data only when the behavior cannot be tested from a recorded response.

What should an agent trace contain?

Keep ordered events, component versions, validated tool arguments, result classes, policy decisions, external operation receipts, and the final claim. Redact sensitive values and avoid storing private reasoning or unnecessary prompt content.

How do you test a non-deterministic agent with replay?

Test deterministic safety and trajectory rules on every run, then sample multiple model-in-the-loop runs for softer quality signals. Treat a new tool path as a review event rather than silently accepting a new snapshot.

Is trace replay the same as agent observability?

No. Observability helps you inspect what happened. Trace replay turns selected evidence from what happened into a controlled test you can run before the next release. Strong systems use both.

How many replay fixtures should a team start with?

Start with one high-impact incident and add cases after real failures. A small, maintained incident corpus is more valuable than a large collection of stale traces no one can explain.

AI Agent Dependency Lockfiles: Stop Silent Tool and Package Drift

Jack M — Sat, 24 Oct 2026 09:50:09 +0000

An agent can behave differently on Tuesday even when nobody changed its prompt or application code. A package resolves to a newer release, an MCP server updates a tool description, a model alias moves, or a tool image is rebuilt. The workflow still starts. The tests may still be green. But the agent is no longer the runtime you reviewed.

That is why a normal package-lock.json is necessary but incomplete for an AI feature. It pins part of the application, while the agent also depends on models, prompts, tool servers, policy bundles, retrieval configurations, and external packages. Treat those as one dependency graph. Then record the resolved graph in an AI agent dependency lockfile.

This is not a claim that a lockfile makes an agent safe. It gives you a specific, reviewable answer to a much more useful question: what exactly did this agent run with? That answer makes bad changes easier to spot, reproduce, and roll back.

What belongs in an agent dependency lockfile?

Start with dependencies that can change behavior or permissions without changing the business workflow. A useful lockfile records both the requested selector and the resolved identity.

Dependency	Weak reference	Useful locked identity
Model	`fast-model`	provider, model revision or immutable deployment ID
Prompt	`support-v3`	repository commit and content digest
MCP tool	`billing-tools:latest`	image/package digest, server version, tool-schema digest
Library	`agent-sdk:^4`	exact package version and integrity hash
Policy	`default-policy`	policy bundle digest and decision version
Retrieval config	`help-center`	index revision, embedding model, and chunking config digest

The point is not to freeze every choice forever. The point is to separate intent from resolution. latest can remain an input to an update job, but it should not be the only record of what production executed.

The trap: configuration looks less risky than code

Teams often review source changes carefully and accept an environment variable such as MODEL=fast-model or TOOLS_URL=https://tools.internal as harmless wiring. In an agent system, that wiring can change output quality, tool availability, permissions, latency, or data handling.

The same issue appears with agent-installed dependencies. If a coding agent adds a package because it recognizes a familiar name, the pull request can hide a new transitive tree behind one import. If an agent fetches a tool server by a mutable tag, an image rebuild can change the runtime after review. Current public guidance on agent supply chains explicitly includes models, agents, libraries, developer tools, and MCP-server packages in scope. The boundary is wider than requirements.txt.

A small lockfile format that is easy to review

Use a format your tools can parse and your reviewers can read. JSON is convenient for CI, though YAML or TOML works too. This example deliberately avoids storing credentials and raw customer data.

{
  "lockVersion": 1,
  "agent": {
    "name": "refund-assistant",
    "workflowCommit": "8e9b1f2",
    "policy": {
      "selector": "refund-policy",
      "resolvedDigest": "sha256:6b4d..."
    }
  },
  "model": {
    "selector": "support-fast",
    "provider": "example-provider",
    "resolvedId": "support-fast-2026-10-18"
  },
  "prompts": [{
    "path": "prompts/refund.md",
    "gitCommit": "8e9b1f2",
    "sha256": "f18c..."
  }],
  "tools": [{
    "name": "get_order",
    "serverImage": "registry.example/tools@sha256:a91d...",
    "serverVersion": "2.4.1",
    "schemaSha256": "ce33...",
    "scopes": ["orders.read"]
  }],
  "packages": {
    "agent-sdk": { "version": "4.2.3", "integrity": "sha512-..." }
  },
  "retrieval": {
    "indexRevision": "kb-2026-10-23-04",
    "embeddingModel": "embed-small-r17",
    "chunkerDigest": "sha256:9d70..."
  }
}

Two details matter here. First, the selector preserves the human-facing intent, while the immutable ID or digest preserves what actually resolved. Second, the lockfile captures only references and hashes. A lockfile should never become a second secret store.

Generate the file; do not hand-edit it in production

Create one command that resolves inputs, validates them, and writes a canonical lockfile. A minimal TypeScript-style sketch looks like this:

type ResolvedTool = {
  name: string;
  serverImage: string;
  schemaSha256: string;
  scopes: string[];
};

async function lockTool(name: string): Promise<ResolvedTool> {
  const server = await registry.resolveImmutableImage(name);
  const schema = await mcp.inspect(server.digest);

  if (!schema.tools.every(tool => allowedTools.has(tool.name))) {
    throw new Error(`Unexpected tool exposed by ${name}`);
  }

  return {
    name,
    serverImage: server.digest,
    schemaSha256: sha256(stableJson(schema)),
    scopes: minimalScopes(schema)
  };
}

Canonical serialization is important. If equivalent JSON can produce different bytes, a hash diff becomes noise. Sort keys, normalize line endings, and write the generated file deterministically. Commit it next to the workflow definition just as you commit an ordinary package lockfile.

Make dependency diffs a release gate

A lockfile only helps if a changed resolution gets attention. In CI, generate a candidate lockfile, compare it with the committed version, and classify the diff.

Change	Default handling	Why
Patch-level library update	test and review	behavior may still change
New MCP tool or expanded scope	block pending approval	expands agent capability
Tool-schema digest changed	contract tests + approval	arguments or semantics may drift
Model revision changed	evaluation suite + staged rollout	behavior can shift without API errors
Policy digest changed	policy-owner approval	decision boundary changed
Retrieval index revision	freshness and regression checks	evidence may have changed

Avoid a single “lockfile changed” checkbox. A new read-only tool and a tool that can issue refunds are not equivalent. The classifier should turn a mechanical diff into an appropriate review path.

Here is a shell-shaped CI outline:

agentctl lock --config agent.yaml --out agent.lock.next.json
agentctl lock verify --file agent.lock.next.json
agentctl lock diff agent.lock.json agent.lock.next.json --format json > diff.json
agentctl lock policy-check diff.json
agentctl test contracts --lock agent.lock.next.json
agentctl eval run --suite refund-safety --lock agent.lock.next.json

The policy-check command should fail closed for prohibited changes, such as a mutable image tag, an unapproved registry, a newly write-capable tool, or a missing integrity digest. It should produce a readable report for permitted changes, not merely a pass/fail bit.

Keep resolution separate from activation

Do not let a production process silently resolve new dependencies at startup. Instead, use a two-step path:

Resolve: an update job produces a candidate lockfile from declared selectors.
Validate: CI runs package scans, tool-contract tests, workflow tests, and representative evaluations.
Approve: reviewers accept the classified dependency diff.
Activate: deployment references the immutable lockfile digest.
Observe: logs attach the lockfile ID to each run; a canary receives a limited traffic slice.
Roll back: redeploy the prior known-good lockfile if failure signals cross a threshold.

This is deliberately less magical than allowing a runtime to “get the newest compatible tool.” Predictable change is the goal. If fast security patches are needed, the same path can be accelerated with an explicit emergency approval and a short retrospective requirement.

Record the lockfile ID in operational evidence

Attach agent_lock_digest to traces, decision records, and evaluation results. When a support team asks why an agent started choosing a different tool, you can group failures by the exact dependency set instead of guessing from deploy timestamps.

That also makes incident work smaller. You can reproduce an affected run in a sandbox using the workflow commit plus lockfile, then compare it with the previous digest. A release manifest tells you the bundle identity; the dependency lock tells you which mutable ecosystem references became concrete artifacts.

Common failure modes

Locking a name, not content. tools: 2.4.1 is better than latest, but a package version alone may not identify the exact artifact. Use an integrity value, immutable image digest, or registry-provided content hash where possible.

Treating tool schemas as documentation. A schema can alter required fields, defaults, descriptions, or side effects while the server name stays the same. Snapshot the inspected schema and run contract tests against it.

Locking only dependencies. A model alias, prompt file, retrieval index, and policy bundle can each change an agent’s behavior. Include the components that matter to your threat model and debugging path.

Making updates impossible. A lockfile that nobody refreshes becomes stale theater. Run a scheduled candidate-resolution job, send bounded diffs to owners, and maintain a tested fast lane for security patches.

Logging secrets by accident. Store a tool’s allowed scope and identity, not an access token, endpoint query string, or customer payload. Scan generated lockfiles before commit.

A practical adoption plan

Begin with one agent that has meaningful external effects. Inventory its model, prompt, tool servers, packages, policy, and retrieval configuration. Add immutable IDs for the two or three components most likely to drift. Then make one CI rule visible: any newly write-capable tool requires a human approval.

After that, add schema snapshots and trace correlation. Only when those pieces work should you expand the lock to every runtime dependency. The artifact should reduce ambiguity for engineers, not create a giant metadata file nobody can review.

Use the lockfile with, not instead of, an inventory

A lockfile answers “what did this release resolve?” An inventory answers “what agent components exist across the organization?” Keep both. The inventory is useful for ownership, patch notifications, and discovering an unmaintained tool server; the lockfile is useful for exact runtime reproduction. Mixing the two creates a document that is too broad for a deploy and too stale for security operations.

Think of the lockfile as a release-level snapshot and the inventory as a portfolio-level map. A tool may appear in the inventory with an owner, risk classification, and approved registry. Each agent lockfile then states the immutable tool artifact, schema digest, and scopes it actually used. This distinction gives a security reviewer a short answer during a release review and gives an operations team a larger answer when a vulnerability advisory arrives.

When a dependency advisory appears, first query the inventory to identify candidate owners and workloads. Next, query deployment records for lockfile digests that reference the affected artifact. That keeps response work evidence-based: do not restart every agent merely because a similarly named package exists somewhere in a repository.

What a good review looks like

Reviewers should be able to answer four questions from the pull request: What changed? Which capability or behavior could change? What tests exercised the new resolution? How do we return to the previous known-good digest? Include those answers in the generated diff summary.

For example, a tool image update with an unchanged schema may need integration tests and a vulnerability scan. A schema diff adding issue_refund requires a capability-owner approval even if the image has no known CVEs. A new model revision needs representative task evaluation, because its interface can be stable while its decisions move. The lockfile makes these different kinds of changes visible in one place without pretending they carry the same risk.

FAQ

Is an AI agent dependency lockfile different from a package lockfile?

Yes. A package lockfile resolves language dependencies. An agent lockfile extends the same idea to behavior-relevant dependencies such as models, prompts, MCP tools, policies, and retrieval revisions.

Should every model request use a pinned model version?

For production workflows where repeatability matters, record the resolved deployment or revision identity when your provider exposes one. If only an alias exists, record the alias, provider metadata available at resolution time, and the evaluation evidence used for the rollout.

How do I version an MCP server safely?

Pin its immutable image or package integrity value, capture the inspected tool-schema digest, test its tool contracts, and require approval when the exposed tools or scopes expand.

Can a lockfile prevent a malicious dependency?

No. It helps you know and reproduce what was selected. Combine it with trusted registries, integrity verification, vulnerability scanning, least-privilege credentials, code review, and runtime controls.

How often should we refresh an agent lockfile?

Generate a candidate on a predictable cadence and when declared dependencies change. Apply updates only after the appropriate checks; urgency can change the approval speed, not the evidence you retain.

What should happen when a lockfile changes during an incident?

Compare the new and previous resolved identities, correlate failures with the lockfile digest, deploy the last known-good lockfile if justified, then preserve the diff and test results for follow-up.

Do small teams need this much process?

Start small: pin package and tool identities, commit one generated file, and review capability changes. The same habit scales better than reconstructing a changed agent runtime from scattered environment variables later.

AI Agent Reconciliation: What to Do When a Tool Times Out After It May Have Succeeded

Jack M — Fri, 23 Oct 2026 03:35:27 +0000

A timeout after an agent sends a refund, deploy, booking, or customer message is not a failure. It is an unknown outcome: the remote system may have completed the work after your connection stopped waiting. If the agent retries because it sees the word “timeout,” one intended action can become two real-world actions.

The fix is not a better prompt that says “be careful.” Put a reconciliation protocol in the tool layer: assign one durable operation key, record the intended effect before dispatch, query for evidence after an ambiguous result, and only retry when the evidence says the effect did not happen. This guide shows the state model, API contracts, and tests that make that rule practical.

Why a timeout is not a failed action

There are two timelines in every state-changing tool call:

The caller sends a request and waits for an acknowledgement.
The downstream service receives the request and applies an effect.

A timeout only tells you that timeline one ended without a useful answer. It says nothing conclusive about timeline two. A worker can crash after sending the request. A load balancer can drop the response. A payment service can commit just after the client deadline. In each case, immediately sending the same logical request again can duplicate the effect.

This matters more for agents than ordinary forms. Agents are designed to recover and continue. They can interpret an error as a reason to retry, call a fallback tool, or resume a durable workflow later. A natural-language instruction cannot be the final authority over a write to an external system.

Treat these outcomes as separate states:

Tool result	Meaning	Next move
`confirmed`	The target returned a receipt or authoritative result.	Continue.
`rejected`	The target proved it did not accept the operation.	Correct input or stop.
`unknown`	A timeout, connection reset, process crash, or malformed acknowledgement occurred.	Reconcile; do not retry.
`not_found`	Reconciliation can prove the effect did not land.	Retry with the same operation key if safe.
`conflict`	Evidence is incomplete or contradictory.	Escalate; do not guess.

The important design choice is exposing unknown as a first-class outcome. Do not translate it into false, throw a generic exception, or let the model decide what it means.

The narrow problem this solves

Idempotency is necessary, but it is not the whole recovery story. It prevents many duplicate writes when a downstream API accepts an idempotency key. Reconciliation answers a different question: what actually happened to this intended business operation?

That distinction matters when an integration does not support client keys, when a write spans several systems, or when a successful response was lost. It also prevents the opposite bug: an agent tells a customer an action failed even though it succeeded.

Use this pattern for consequential writes such as:

creating a charge, refund, booking, order, ticket, or deployment;
sending an external email, Slack message, or webhook;
changing a permission, subscription, or account setting;
deleting or publishing a customer-visible record.

Pure reads can normally retry. A deterministic update such as “set plan to Pro” is safer than “append a note,” but it still needs an operation record if the user needs a truthful outcome.

Make the application own effect state

The model should propose an action. Your application should own its identity, authorization, execution, and recovery. A minimal operation ledger looks like this:

create table agent_operations (
  operation_id uuid primary key,
  tenant_id uuid not null,
  run_id uuid not null,
  tool_name text not null,
  request_hash text not null,
  state text not null check (state in (
    'prepared', 'dispatched', 'confirmed', 'unknown', 'reconciling',
    'not_found', 'escalated', 'cancelled'
  )),
  provider_reference text,
  receipt jsonb,
  reconcile_after timestamptz,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (tenant_id, tool_name, operation_id)
);

operation_id identifies one intended business effect, not one HTTP attempt. Store a hash of the normalized request too. If a resumed agent presents the same operation ID with different arguments, reject it. That is a serious programming or orchestration error, not a request to “try again.”

Create the row before making the external call. Otherwise, a crash in the gap between dispatch and persistence recreates the very ambiguity you are trying to control.

Design tool contracts for honest recovery

An agent-facing write tool should not return a raw network exception. Return a small structured result that forces the workflow into a safe branch.

type EffectResult =
  | { state: "confirmed"; operationId: string; receipt: Record<string, unknown> }
  | { state: "rejected"; operationId: string; reason: string }
  | { state: "unknown"; operationId: string; retryAllowed: false; nextCheckAt: string }
  | { state: "escalated"; operationId: string; reason: string };

The wrapper receives a stable operationId, writes it to the ledger, and passes it downstream as an idempotency header or external reference whenever possible.

async function sendRefund(input: RefundInput, operationId: string): Promise<EffectResult> {
  await operations.prepare(operationId, input);

  try {
    const response = await billing.refund({
      ...input,
      idempotencyKey: operationId,
      metadata: { agent_operation_id: operationId },
    });
    await operations.confirm(operationId, response.id, response);
    return { state: "confirmed", operationId, receipt: { refundId: response.id } };
  } catch (error) {
    if (isDefiniteRejection(error)) {
      await operations.reject(operationId, safeReason(error));
      return { state: "rejected", operationId, reason: safeReason(error) };
    }
    await operations.markUnknown(operationId, error);
    return {
      state: "unknown", operationId, retryAllowed: false,
      nextCheckAt: new Date(Date.now() + 30_000).toISOString(),
    };
  }
}

Notice what is missing: there is no catch { retry() }. The tool owns transport uncertainty so the agent never treats an unconfirmed write as a fresh task.

Reconcile before any retry

Reconciliation is a targeted read of an authoritative system of record. The best API pair is a write that accepts your client-generated key and a read such as GET /refunds?operation_id=.... That read should return the effect’s lifecycle state, not merely recent activity that happens to look similar.

Use this decision flow:

prepare operation → dispatch write → acknowledgement?
                                 ├─ yes → confirm receipt → continue
                                 └─ no  → mark unknown → query by operation ID
                                                     ├─ found → confirm
                                                     ├─ absent after safe window → retry same key
                                                     └─ partial/ambiguous → escalate

The “safe window” is integration-specific. A provider with eventual consistency may need several read attempts with exponential backoff before absence means anything. Record the policy per tool: maximum reconciliation delay, available lookup key, whether a retry is safe, and who owns escalation.

For systems that cannot query by a client key, look for a compound business identity: a draft ID, customer ID plus immutable request nonce, or a provider correlation field. If you cannot reliably identify the effect, the correct automated outcome is escalated, not a duplicate write disguised as resilience.

Do not confuse a similar record with proof

A tempting shortcut is to search for “a recent refund for this customer” or “a message with roughly this text.” That can turn reconciliation into a new source of bad decisions. A customer may have made two valid requests, an operator may have sent the message manually, or a background job may have changed the same record.

Define the minimum evidence needed to confirm every integration. Prefer a provider receipt that carries the operation key. If that is unavailable, require a combination of immutable business ID, normalized request hash, target identity, and a time bound that is documented by the provider. Mark the result as conflict when evidence is merely plausible rather than conclusive.

This is also where tenant boundaries matter. An operation lookup must be scoped by the requesting tenant before its evidence reaches an agent or reviewer. A global search for a matching customer email can accidentally confirm another tenant’s action. Keep the target reference, hashes, and receipts in application-owned storage, and authorize reconciliation reads exactly as you authorize the original write.

Give each operation a small state machine

Do not let every caller update the ledger with arbitrary strings. Define allowed transitions such as prepared → dispatched → confirmed, dispatched → unknown → reconciling → confirmed, and unknown → reconciling → escalated. Disallow confirmed → dispatched entirely. The guard belongs in the database transaction or service method, not in the prompt or UI.

This protects against races between a late provider webhook, a queue retry, and a human reviewer. When a webhook supplies a valid receipt after the reconciliation job has started, it can safely claim confirmed; the job then sees the terminal state and exits. When a reviewer declines an uncertain deletion, that decision becomes a terminal cancelled record rather than an instruction an agent can reinterpret later.

Include an optimistic version or row lock on transitions, and append an immutable event for each transition. The current state makes the workflow fast to query; the event history explains why it reached that state. This is especially valuable during incident review, where you need to distinguish “the provider acted late” from “two workers independently dispatched the same effect.”

Use a reconciliation queue, not a blocked agent turn

Avoid holding a model session open while waiting for a slow target. When a wrapper returns unknown, enqueue a durable reconciliation job keyed by operation_id. That job should lock the operation, perform a bounded read-back sequence, and write one terminal state.

async function reconcile(operationId: string) {
  const op = await operations.lockForReconciliation(operationId);
  if (["confirmed", "rejected", "escalated"].includes(op.state)) return;

  const evidence = await provider.findByOperationId(op.operation_id);
  if (evidence.kind === "found") return operations.confirm(op.operation_id, evidence.id, evidence);
  if (evidence.kind === "not_found" && canRetry(op)) {
    await operations.markNotFound(op.operation_id);
    return queue.enqueue("replay_operation", { operationId: op.operation_id });
  }
  return operations.escalate(op.operation_id, evidence.reason);
}

The replay uses the same operation ID and the same normalized request. Do not restart the whole agent run; that can repeat unrelated successful steps. If a workflow includes five writes, reconcile each effect independently and allow downstream steps only after their prerequisites are confirmed.

Add evidence, boundaries, and human decisions

An operations table without useful evidence becomes a second mystery. Store redacted dispatch time, provider request ID, response status, error class, attempt count, reconciliation reads, and final receipt. Never store secrets or unredacted customer content merely to debug retries.

Classify each tool in a policy registry:

Action class	Example	On unknown outcome
Safe replay	Set a named record to a known value	Reconcile, then replay same key.
Dedupe-supported	Refund API with idempotency key	Reconcile; same key may be replayed.
Compensatable	Create ticket then update local state	Reconcile; compensate only with evidence.
Irreversible	Send a legal notice or delete a record	Reconcile, then human review.

Require an approval or incident handoff for conflicts involving money, irreversible deletion, permissions, or customer communications. The reviewer needs the intended request, operation ID, timeline, evidence, and choices—not hidden chain-of-thought or a vague “agent failed” message.

Test the failures that happy paths hide

Your integration tests should simulate a target that commits the effect and drops the response. That is the bug that ordinary mock tests miss.

At minimum, test these invariants:

the same operation ID with changed input is rejected;
a timeout produces unknown, never automatic replay;
a reconciliation read that finds a receipt reaches confirmed without a second dispatch;
an absent result only enables retry after the configured consistency window;
two workers reconciling the same operation produce one terminal state;
a resumed workflow uses existing operations rather than creating new ones;
escalation contains redacted evidence and no model reasoning trace.

For a simple test double, let POST /refunds persist a refund and then throw a timeout. Run the wrapper twice. The expected result is one refund record, one operation ID, and a second call that returns the original receipt or waits for reconciliation.

Track unknown-outcome rate by tool, time-to-reconciliation, escalations, duplicate effects detected after the fact, and disagreements between an agent’s message and the reconciled state. Those metrics reveal fragile integrations before a rare timeout becomes a customer incident.

A rollout plan that does not freeze delivery

Start with one high-impact tool rather than wrapping every integration at once.

Inventory state-changing tools and rank them by irreversibility and customer impact.
Add operation IDs and ledger rows in observe-only mode; do not change retry behavior yet.
Capture whether each provider supports idempotency keys, lookup by key, and reliable receipts.
Switch the selected tool so ambiguous outcomes become unknown and enter reconciliation.
Add alerts for operations stuck past the tool’s expected window.
Run timeout, crash, duplicate-delivery, and delayed-consistency drills before expanding coverage.

This sequence produces evidence before policy changes. It also makes conversations with API owners concrete: ask for a client-generated key, a lookup endpoint, and a receipt—not a promise that timeouts are rare.

FAQ

Is an AI agent timeout the same as a failed tool call?

No. A timeout only means the caller did not receive a conclusive response. The target may have applied the effect, so a state-changing call should enter an unknown-outcome reconciliation path.

What is an operation key for an AI agent?

It is a durable ID for one intended business effect. Reuse it across attempts, send it to the target when possible, and use it to find the effect during reconciliation.

Are idempotency keys enough for agent retries?

They are the best protection when every target honors them, but you still need a ledger and read-back path to report the outcome honestly and handle APIs without key support.

When can an agent retry a timed-out action?

Only after a provider receipt or authoritative lookup proves the effect did not happen, or when the action is intrinsically safe to replay. Use the same operation key, not a new one.

What should happen when reconciliation cannot determine the result?

Stop automatic execution and escalate with a redacted operation timeline. Guessing can create a duplicate charge, message, deletion, or permission change.

Should the language model decide whether an action succeeded?

No. Let the model propose the action and explain the user-facing result, but derive success from deterministic receipts and reconciled system state.

How is reconciliation different from a workflow retry?

A workflow retry repeats a task. Reconciliation checks whether the original business effect happened and chooses confirm, safe replay, compensation, or escalation for that one operation.

MCP Schema Versioning: Keep AI Tool Calls Working Through API Changes

Jack M — Thu, 22 Oct 2026 03:29:41 +0000

An agent that worked yesterday can start failing after a change that looks harmless in a pull request: rename location to query, make a field required, or reshape a result object. The model has not suddenly become worse. It is still following the tool contract it saw before the deployment.

That is why MCP schema versioning needs the same care as a public API migration. Tool names, descriptions, input JSON Schema, output shape, and runtime behavior all influence an agent's next action. A server can successfully deploy while a cached client, a long-running workflow, or a prompt template continues to make yesterday's call.

This guide shows a practical way to evolve MCP tools without turning routine backend work into mysterious agent failures. The goal is not to freeze every tool forever. It is to make changes visible, compatible when possible, and reversible when they are not.

Separate protocol versions from tool-schema versions

MCP has protocol-version compatibility rules. Clients and servers use those rules to agree on the protocol revision they speak. That does not automatically version your application tools.

For example, both of these servers might support the same MCP protocol revision:

// Yesterday
{
  "name": "find_customer",
  "inputSchema": {
    "type": "object",
    "properties": { "email": { "type": "string" } },
    "required": ["email"]
  }
}

// Today
{
  "name": "find_customer",
  "inputSchema": {
    "type": "object",
    "properties": { "email": { "type": "string" }, "tenant_id": { "type": "string" } },
    "required": ["email", "tenant_id"]
  }
}

The protocol can be perfectly compatible while old agent calls now fail validation. Treat the tool schema as its own public contract. Give it an owner, a baseline, a change review, and a migration path.

Know which changes break agents

Start every tool change by asking a narrow question: can a call that was valid yesterday be rejected or mean something different today?

Change	Compatibility	Safe response
Add an optional input with a default	Usually compatible	Release normally; snapshot the change.
Add a new tool	Usually compatible	Keep existing tools stable and test tool selection.
Add a required input	Breaking	Keep the old input contract or introduce a versioned tool.
Rename/remove an input	Breaking	Accept the old name during a deprecation window.
Narrow a type or enum	Breaking	Preserve old values or map them explicitly.
Rename/remove an output field	Often silently breaking	Return both fields until clients migrate.
Change a description	Behavior-changing	Review it like code; models use descriptions to choose tools.
Keep the schema but change the business meaning	Breaking	Publish a new contract and require a migration.

The last two are easy to miss. A description is not merely documentation when it enters an agent's context. And a JSON Schema diff cannot see that status: "complete" now means “queued” instead of “executed.”

Make dynamic discovery a controlled deployment boundary

MCP makes tool discovery convenient, but convenience can hide a release. If an agent fetches a live tools/list response on every connection, an upstream definition can change between two user sessions without a code deployment in the agent service.

Choose deliberately which of these modes each integration uses:

Pinned: the agent uses an approved tool manifest for its release. This is the safest default for write-capable tools.
Monitored dynamic: the agent discovers live tools, but a fingerprint diff creates an alert and can disable a changed high-risk tool.
Fully dynamic: the agent accepts current definitions immediately. Reserve this for low-risk, read-only experimentation.

Pinning does not mean copying a tool implementation. It means recording the interface your agent was evaluated against. Refresh that manifest through the same review process as an SDK upgrade. For dynamic integrations, record the discovered fingerprint with the call trace; otherwise, a later investigation cannot distinguish a prompt regression from an upstream contract change.

This boundary also helps with tool selection. Adding ten unrelated tools may not invalidate any JSON payload, yet it can change which tool the model chooses. Keep the exposed tool set small per workflow, and run a representative selection test whenever the manifest changes.

Build a schema snapshot before you need it

Do not discover the old contract by reading a production incident. In CI, call tools/list, normalize the result, and commit a snapshot for every supported server and environment.

import { createHash } from "node:crypto";

function stableToolFingerprint(tool: unknown) {
  const canonical = JSON.stringify(tool, Object.keys(tool as object).sort());
  return createHash("sha256").update(canonical).digest("hex");
}

const current = (await client.listTools()).tools.map((tool) => ({
  name: tool.name,
  fingerprint: stableToolFingerprint({
    name: tool.name,
    description: tool.description,
    inputSchema: tool.inputSchema,
    outputSchema: tool.outputSchema
  })
}));

In real code, use a recursive canonical JSON serializer; sorting only top-level keys is not sufficient. Keep two artifacts:

A readable JSON file for review.
A fingerprint manifest for a fast deploy check.

Avoid storing credentials, tenant examples, or live tool results in the snapshot. The contract should be enough to detect a change without copying sensitive data into the repository.

Diff by risk, not just by bytes

A byte diff says that something changed. A compatibility classifier says whether it can break work.

Your classifier should flag at least these conditions:

a tool disappeared or was renamed;
a property was removed or became required;
a type, numeric range, regex, or enum narrowed;
a default value changed;
an output property disappeared or changed type;
a description changed for a high-impact tool;
an authorization, side-effect, or idempotency expectation changed.

Put the result in the pull request. A useful summary is more actionable than a large JSON patch:

BREAKING: find_customer now requires tenant_id
COMPATIBLE: list_invoices gained optional page_size (default 50)
REVIEW: refund_payment description changed; execution remains side-effecting

For tools that create records, move money, send messages, or alter infrastructure, make a breaking diff block the release until an approved migration plan is attached.

Prefer adapters over abrupt replacements

Suppose location is no longer precise enough and your API now needs latitude and longitude. Do not immediately force every agent to learn the new fields. Accept the old call, translate it, and emit a deprecation signal that your logs can count.

type WeatherArgs =
  | { location: string; unit?: "c" | "f" }
  | { lat: number; lng: number; unit?: "c" | "f" };

function normalizeWeatherArgs(args: WeatherArgs) {
  if ("location" in args) {
    return {
      kind: "legacy" as const,
      query: { place: args.location, unit: args.unit ?? "c" }
    };
  }

  return {
    kind: "current" as const,
    query: { lat: args.lat, lng: args.lng, unit: args.unit ?? "c" }
  };
}

The adapter is a temporary compatibility boundary, not permanent clutter. Give it a removal date and measure use of the legacy branch. If old calls are still common, find out where they originate: a scheduled workflow, a prompt stored in a database, a client that caches tools, or an integration you do not control.

When the semantics truly differ, use a new tool name such as get_weather_by_coordinates rather than overloading the old name. An agent can reason about two clear options better than one tool with surprising behavior.

Version outputs as carefully as inputs

Input errors usually fail loudly. Output changes often fail silently.

Imagine a tool used to return this:

{ "ok": true, "ticket_id": "T-42" }

Then a refactor changes it to:

{ "state": "accepted", "job_id": "J-96" }

An agent may interpret missing ok as success, retry unnecessarily, or tell the user a task is complete before it runs. During a transition, return an envelope that preserves the old fields while introducing the new ones:

{
  "ok": true,
  "ticket_id": "T-42",
  "state": "accepted",
  "job_id": "J-96",
  "deprecation": { "fields": ["ok", "ticket_id"], "remove_after": "release-2027.03" }
}

Use a release identifier or a policy window in that metadata, rather than a vague promise to remove it “soon.” Validate that old and new fields always agree while both exist.

Test three layers of the contract

Schema validation alone is necessary but not enough. Add three test layers to each MCP tool.

1. Snapshot tests

Compare tools/list against the approved manifest. This catches accidental schema regeneration after a Pydantic, Zod, OpenAPI, or SDK upgrade.

2. Compatibility fixtures

Run old valid payloads and current payloads through the handler. The old fixture should either succeed through the adapter or fail with an explicit, actionable migration error. Never let an old call silently map to a different action.

3. Behavioral contract tests

Assert the promise that JSON Schema cannot express: a read tool does not write, a dry run does not execute, a retry does not duplicate a charge, and a returned accepted state does not mean completed.

it("keeps legacy lookup behavior during migration", async () => {
  const result = await callTool("find_customer", { email: "a@example.test" });
  expect(result.isError).toBe(false);
  expect(result.structuredContent.customer).toMatchObject({ email: "a@example.test" });
  expect(result.structuredContent.deprecation).toBeDefined();
});

Use synthetic tenants and fake side-effect adapters for these tests. Contract tests should be safe enough to run on every pull request.

Roll out schema changes in stages

A careful rollout makes the blast radius small and observable.

Inventory consumers. Find interactive agents, scheduled jobs, API clients, prompt templates, and external integrations that call the tool.
Ship additive support. Add the new shape or new tool while the old contract remains valid.
Release the snapshot and fixtures. Make the intended diff visible in code review.
Canary by tenant or workflow. Send a limited, reversible population to the new contract.
Observe tool errors and outcome errors. Track validation failures, fallback use, retries, and task completion—not only HTTP status.
Notify and deprecate. State the replacement, an example payload, and the removal condition.
Remove only after evidence. Delete the adapter after legacy traffic stays at zero for the defined window.

Keep a server build identifier and tool-manifest fingerprint in every trace. When a failure appears, this lets an engineer answer “which contract did the agent see?” without reconstructing a deployment timeline.

A small release checklist

Before merging an MCP tool change, answer these questions:

Does any formerly valid request now fail or mean something different?
Did input, output, description, side effect, authorization, or idempotency change?
Is the schema snapshot updated and classified by compatibility risk?
Do old and new fixtures run in CI?
Is there an adapter, a new tool name, or an explicit migration error?
Can the change be canaried and rolled back without deleting state?
Is there an owner and a measurable removal condition for deprecated behavior?

If any answer is “no,” the tool is not ready for a routine deployment. That is not bureaucracy. It is how you prevent an API change from masquerading as an agent reliability problem.

FAQ

What is MCP schema versioning?

MCP schema versioning is the discipline of tracking and evolving an MCP tool's name, description, input schema, output shape, and behavior without breaking agents that use an older contract. It is separate from the MCP protocol version negotiated by clients and servers.

Is adding a required MCP tool parameter a breaking change?

Yes. Any older call that omits the new parameter can fail validation. Prefer an optional field with a safe default, a compatibility adapter, or a new tool name when the new information is essential.

Should MCP tool descriptions be versioned too?

Yes. Agents use descriptions to select tools and construct arguments, so a description change can alter behavior even if the JSON Schema is identical. Snapshot descriptions and require review for changes to sensitive or side-effecting tools.

How long should an MCP schema deprecation window last?

Base it on observed legacy traffic and the longest-running consumer in your system. Define a minimum window, publish a removal condition, and remove the adapter only after telemetry shows the old shape is unused for that period.

Can JSON Schema tests catch every breaking agent change?

No. They catch structural differences, but not changed business meaning, tool-selection behavior, authorization assumptions, or unwanted side effects. Pair schema snapshots with compatibility fixtures and behavioral contract tests.

Should I create a new MCP tool instead of changing an old one?

Create a new tool when the input or output semantics change materially. For a small additive change, keep the existing tool and use an optional field. A clearly named second tool is safer than one overloaded tool whose old calls now do something surprising.

AI Code Invariant Tests: Catch Regressions That Green Test Suites Miss

Jack M — Wed, 21 Oct 2026 03:31:59 +0000

An AI coding agent can make every visible test pass and still break the one rule your product cannot afford to violate.

That is the uncomfortable failure mode behind many “safe” agent refactors. The diff looks tidy. Unit tests are green. The new code may even have more coverage than before. But a permission check moves after a database lookup, a retry stops being idempotent, or a list endpoint turns into an accidental N+1 query. Nothing in a happy-path test has to notice.

The fix is not a longer prompt asking the agent to be careful. It is to name the behavior that must remain true, then test it as an invariant. This article shows a practical, framework-neutral way to add invariant tests around AI-generated changes—especially where money, access, state, and latency matter.

Why green tests can still hide an AI-generated regression

Most unit tests answer a narrow question: “Does this example produce the expected output?” That is useful, but it leaves room for an implementation that passes known examples while breaking a rule between examples.

Coding agents amplify that gap. They are effective at repairing the assertion directly in front of them. If a refactor changes a test, they may update the test too. The suite turns green, but the original contract has quietly moved.

Recent developer discussions around agentic refactoring repeatedly describe the same pain: tests can remain green while an agent weakens an alert, rewrites an assertion, or creates a slow query loop. Research and benchmark work on coding agents also points toward specification-first and adversarial validation rather than trusting generated code by default.

An invariant is different from a single expected result. It is a property that must hold across many inputs and implementation paths.

Ordinary example test	Invariant test
A member can read their own project.	A request can never return a project from another organization.
Retrying this request returns `200`.	Retrying the same operation never creates a second charge.
A search result is sorted in this fixture.	Results are always sorted by the promised key, including ties and empty input.
This endpoint completes quickly in a test.	Loading 50 records does not issue more than a bounded number of queries.

The aim is not to replace example tests. Keep them. Invariants give them a harder-to-game companion.

Pick invariants before asking an agent to refactor

Do not begin with “clean up this module.” Start with a short, testable list of things that cannot change. This becomes the acceptance contract for the task and a small map for review.

For a typical application, useful categories are:

Authorization: one tenant or role never sees another’s data.
State: records move only through allowed states.
Money and side effects: retries do not duplicate emails, charges, or writes.
Data integrity: identifiers, totals, and ownership stay consistent.
Reliability: failures leave the system in a recoverable state.
Performance: a refactor does not multiply database calls or network work.
Compatibility: a public API preserves its documented shape and meaning.

Write each one in plain language first. A good invariant has a subject, a boundary, and an observable result:

For every invoice event with the same idempotency key, at most one payment record exists.

For every authenticated request, every returned document belongs to the requester’s organization.

For a valid workflow, terminal states cannot transition back to pending.

Avoid vague rules such as “keep the code secure” or “do not change behavior.” They are review intentions, not executable claims.

Turn an invariant into a test oracle

An oracle is the thing that says whether behavior is correct. Existing snapshot tests are weak oracles when an agent can update them. For critical changes, use an oracle the change is not allowed to edit without an explicit human decision.

One practical pattern is to put invariant tests in a protected directory and declare them read-only for the task.

repo/
  src/
  tests/
  safety-contracts/
    organization-isolation.spec.ts
    payment-idempotency.spec.ts
    workflow-transitions.spec.ts

Your task brief can then be concrete:

Refactor src/billing/renewal.ts to remove duplication.

Constraints:
- Do not edit safety-contracts/.
- Run npm test and npm run test:contracts after each logical step.
- Preserve one payment per idempotency key.
- Report changed files, commands, and any failing contract.

That instruction alone is not protection. Enforce it in CI: fail a pull request that changes protected contract files unless a designated reviewer approves it. The important part is separation—an agent should not be able to write both the implementation and the only proof it is safe without making that visible.

Example: protect tenant isolation with generated inputs

Imagine a document query that an agent is about to “simplify.” A normal example test might check that one user can fetch a document. An invariant test should try many organizations and document IDs, then assert that every returned record belongs to the caller’s organization.

Here is TypeScript-style pseudocode using a property-testing library:

import fc from "fast-check";

it("never returns another organization's document", async () => {
  await fc.assert(
    fc.asyncProperty(
      fc.uuid(),
      fc.uuid(),
      fc.uuid(),
      async (requesterOrgId, otherOrgId, documentId) => {
        fc.pre(requesterOrgId !== otherOrgId);

        await seedDocument({ id: documentId, organizationId: otherOrgId });
        const result = await listDocuments({ organizationId: requesterOrgId });

        expect(result.every((doc) => doc.organizationId === requesterOrgId)).toBe(true);
      },
    ),
  );
});

The exact library is optional. You can loop over generated cases yourself. The principle matters: test the rule across values the author did not hand-pick.

For a multi-tenant system, add negative cases too:

a guessed ID from another organization
an empty role list
an admin role from the wrong organization
soft-deleted records
pagination cursors created by another tenant

These are the cases a polished refactor often fails because the happy path still works.

Example: protect idempotency when an agent changes retries

Retries are a classic regression surface. An agent may correctly add timeout handling but accidentally put the side effect before the deduplication check.

The invariant is simple: repeating a request with the same stable key produces one durable effect.

it("creates at most one payment for the same request key", async () => {
  const key = "payment:order-42";

  await Promise.all([
    createPayment({ orderId: "order-42", amount: 2500, idempotencyKey: key }),
    createPayment({ orderId: "order-42", amount: 2500, idempotencyKey: key }),
  ]);

  const payments = await db.payment.findMany({
    where: { idempotencyKey: key },
  });

  expect(payments).toHaveLength(1);
  expect(payments[0].amount).toBe(2500);
});

Run it with sequential and concurrent calls. Then add a failure injection between “reserve key” and “write payment.” If the invariant cannot survive a timeout, it is not really an invariant—it is a best-case assumption.

Test state machines instead of only endpoints

Many business rules are state transitions disguised as API behavior. A support ticket should not reopen after it is permanently closed. An order should not ship before payment. A deleted user should not receive a new session.

Model the allowed transitions in one small table before the refactor:

From	Allowed next states
`draft`	`submitted`, `cancelled`
`submitted`	`approved`, `rejected`, `cancelled`
`approved`	`fulfilled`, `cancelled`
`fulfilled`	none

Then test every pair, not just the common routes:

const allowed: Record<Status, Status[]> = {
  draft: ["submitted", "cancelled"],
  submitted: ["approved", "rejected", "cancelled"],
  approved: ["fulfilled", "cancelled"],
  rejected: [],
  cancelled: [],
  fulfilled: [],
};

for (const from of Object.keys(allowed) as Status[]) {
  for (const to of allStatuses) {
    const canMove = allowed[from].includes(to);
    await expect(changeStatus({ from, to }))
      .resolves[canMove ? "toBeDefined" : "toThrow"]();
  }
}

Use your test framework’s normal assertion style; the point is to enumerate the system’s real state space. If the agent replaces conditionals, extracts services, or switches an ORM call, this test protects the business meaning rather than its old implementation.

Add performance invariants where refactors hide cost

Correct output is not the whole contract. A coding agent can preserve results while turning one query into one query per row.

For known hot paths, count the work. Inject a database query counter in an integration test, execute with a small and a medium fixture, and set a reasonable ceiling.

it("loads project members without an N+1 query", async () => {
  const meter = trackDatabaseQueries(db);
  await seedProjectWithMembers({ memberCount: 50 });

  const page = await getProjectMembers({ projectId: "project-1" });

  expect(page.members).toHaveLength(50);
  expect(meter.count()).toBeLessThanOrEqual(3);
});

Do not turn every test into a brittle microbenchmark. A query ceiling, outbound-call count, or payload-size limit is usually more stable and more useful than an elapsed-millisecond assertion in CI.

A small contract loop for AI-assisted changes

Here is a repeatable workflow that does not require a formal-methods team.

Name the risk. Identify what can be lost, duplicated, leaked, or slowed down.
Write one invariant. Make it observable in code or a controlled integration environment.
Prove the test can fail. Temporarily introduce the forbidden behavior or use a mutation tool. If the test stays green, strengthen it.
Protect the oracle. Keep contract tests out of the agent’s editable scope for the task, or require explicit review for any change.
Give the agent a bounded goal. Specify files, behavior, test commands, and prohibited changes.
Run the contract suite in CI. Store the command output with the pull request.
Review the invariant, not only the diff. Ask whether the rule still means what the product needs.

The third step is the one teams skip. A test that never demonstrates a failure can be theater. For a payment invariant, change the uniqueness guard in a local branch and confirm the test catches two rows. For tenant isolation, deliberately remove the organization predicate and watch the contract fail.

Give coding agents a task contract they can follow

Clear instructions help, especially when they connect a change to a non-negotiable rule. Here is a concise template:

Goal: extract the notification scheduling logic from src/jobs/run.ts.

Behavioral contracts:
1. A job with the same idempotency key sends at most one notification.
2. A cancelled job never sends a notification.
3. A tenant can never schedule a job for another tenant.

Boundaries:
- Do not modify safety-contracts/.
- Do not add dependencies.
- Keep the public handler signature unchanged.

Verification:
- npm run typecheck
- npm test
- npm run test:contracts
- Summarize any command that fails; do not weaken tests to make them pass.

This is stronger than “refactor safely” because it tells the agent what success means. It is also easier for a reviewer to reject a superficially green change: the contract is visible before the code.

Common mistakes when adding invariant tests

Testing implementation details. A contract should survive a refactor. Assert a durable result—number of writes, access boundary, state transition—not which private helper ran.

Using only random input. Generated input expands coverage, but important named cases still matter. Keep regression fixtures for incidents and known edge cases.

Letting the same change rewrite every oracle. If an agent alters production code, snapshots, unit tests, and contract tests in one sweep, reviewers lose their independent evidence. Separate those changes or require a stronger review path.

Making every rule a blocking invariant. Start with the small set whose failure would cause financial loss, data exposure, irreversible writes, or serious reliability damage. A hundred weak contracts will be ignored.

Mistaking test coverage for behavioral coverage. Coverage reports show which lines ran. They do not show whether the line preserved ownership, idempotency, or a state boundary.

Where this fits in an AI-assisted engineering workflow

Invariant tests sit beside, not above, the rest of your quality system:

Types and schemas catch malformed values.
Unit tests catch local logic mistakes.
Integration tests prove boundaries with real infrastructure.
Contract tests protect cross-module promises.
Invariant tests protect rules across many inputs and paths.
Code review checks intent, readability, and operational consequences.

For AI-assisted work, the practical payoff is trustable speed. You can let an agent make a broad mechanical improvement when the most important promises are executable and independently checked. That is safer than restricting agents to trivial tasks—and safer than trusting a green checkmark with no stated contract.

FAQ

What is an invariant test for AI-generated code?

It checks a rule that must stay true across many inputs and execution paths, such as tenant isolation, one payment per idempotency key, or valid state transitions. It complements example-based unit tests.

Are invariant tests the same as property-based tests?

No. Property-based testing is a useful technique for generating many inputs. An invariant is the behavior you want to preserve. You can test an invariant with generated values, fixed fixtures, integration tests, or all three.

Which invariants should a small team add first?

Start with data isolation, authorization, idempotency, money movement, irreversible deletes, and the most expensive performance paths. Choose rules where one regression would create real user or business harm.

Should an AI coding agent be allowed to edit invariant tests?

It can propose a change, but treat that change as a contract decision rather than routine cleanup. Require explicit review and a reason the product rule itself changed; otherwise the agent can make failing evidence disappear.

How do I know an invariant test is strong enough?

Prove it can fail. Temporarily introduce the forbidden behavior, or use mutation testing, and confirm the contract catches it. If it stays green, it is not protecting the rule yet.

Do invariant tests make refactoring slower?

They take time to write, but they reduce the slowest part of refactoring: diagnosing a production regression after a large, plausible-looking change. Start with high-risk paths and keep each contract narrow.

RAG Context Budget: Pack Evidence Without Burying the Answer

Jack M — Tue, 20 Oct 2026 03:31:48 +0000

When a RAG answer is wrong, the reflex is often to raise top_k. More retrieved chunks feels safer. In production, it can do the opposite: the one useful passage gets surrounded by repeated, stale, or merely related text, while the model has less room to answer.

A large context window is capacity, not a retrieval strategy. Treat the input as a RAG context budget: reserve room for the answer, retrieve widely, then admit only the evidence that earns its token cost. This guide shows a vendor-neutral policy you can implement around any vector store and model API.

The mistake: confusing retrieval recall with prompt admission

Retrieval and generation have different jobs. A retriever should find a broad candidate set so a good passage is not missed. The prompt assembler should build a small, diverse, attributable evidence set that a model can use.

Those are not the same top_k.

For example, a support assistant may retrieve 30 candidates to protect recall, rerank the best 12, and send four short, non-overlapping sections to the model. A policy question may need the newest authoritative policy plus one exception. A “list every affected region” question may need more coverage and a different answer shape.

Sending every candidate creates four problems:

Attention dilution: Related passages repeat terms without answering the question.
Unstable cost and latency: A broad request can quietly turn into a huge input.
Crowded-out answers: The model has little output room for a careful explanation or citations.
Weak debugging: A trace says “top 20 retrieved,” but not which evidence actually mattered.

Recent retrieval systems increasingly select useful highlights rather than whole pages. That direction is sensible, but it does not remove the engineering decision: your application still needs an explicit policy for what may enter the generation prompt.

Define the budget before tuning chunks

Start from the smallest context you want to support reliably, not from a model’s maximum window. Reserve fixed and variable space first:

Allocation	What it contains	Why it needs a cap
Instructions	safety rules, format, tool policy	grows with every feature request
Conversation	relevant turns and user constraints	can become the dominant input
Evidence	retrieved passages and metadata	must compete fairly with the answer
Answer reserve	response plus citations	prevents truncation and rushed output
Safety margin	tokenizer and message-format overhead	avoids boundary failures

The basic equation is:

evidence_budget = input_limit
                - instruction_tokens
                - conversation_tokens
                - answer_reserve
                - safety_margin

Use the provider tokenizer for the final count. A rough estimator is fine while choosing candidates, but do not assume words equal tokens or that chat-message formatting is free.

The budget is a product decision too. If the user asks for a comparison with sources, reserve enough output for distinctions and citations. If the task returns a short JSON classification, the answer reserve can be smaller. A single global budget hides that trade-off.

Admit evidence in stages, not by a fixed top-k

A practical context packer has four gates:

Retrieve candidates broadly. Apply tenant, access-control, language, and freshness filters before similarity search where possible. Never let a relevance score override an authorization decision.
Rerank against the actual question. Embedding similarity is a candidate generator, not proof that a passage answers the request. A cross-encoder, rules, or a small model can score query-passage relevance.
Remove redundancy. Adjacent chunks, copied help-center pages, and syndicated content often say the same thing. Keep the best representative and retain its source identity.
Pack to the remaining token budget. Prefer authority, answer coverage, recency when relevant, and source diversity—not score alone.

This means a lower-scoring chunk can win a place because it covers a missing part of a multi-part question. Likewise, a high-scoring duplicate should lose once its fact is already represented.

Make the query class explicit

One cap cannot serve every intent. Begin with a small classifier that chooses a policy, and log its decision:

Query class	Primary goal	Packing rule
Point lookup	precise fact	one authoritative section; add a second only for verification
How-to	ordered steps	preserve a complete procedure and its prerequisites
Comparison	distinctions	reserve space for each option and a shared criterion
Aggregation	coverage	collect distinct sources; state what may be incomplete
Troubleshooting	diagnosis	favor version, error, and environment matches over generic advice

Start with deterministic routing based on obvious cues and test data. You can later use a classifier, but its output should remain observable and easy to override. A wrong class should fail safely by returning a narrower answer, asking a question, or disclosing limits—not by stuffing every result into the prompt.

A small packing policy in Python

The example below accepts already-authorized, reranked candidates. It leaves a fixed answer reserve, rejects near-duplicates, and gives each source a small diversity allowance. It is intentionally simple: production systems also need timeout handling, tracing, and model-specific token counting.

from dataclasses import dataclass

@dataclass
class Chunk:
    id: str
    source_id: str
    tokens: int
    relevance: float
    novelty: float       # 0..1; lower means it repeats accepted evidence
    authority: float     # 0..1; derived from your source policy
    text: str

def pack_context(candidates, evidence_budget, per_source_limit=2):
    accepted, used, source_counts = [], 0, {}

    # Relevance gets us close; novelty and authority stop a score-only pack.
    ranked = sorted(
        candidates,
        key=lambda c: 0.60 * c.relevance + 0.25 * c.novelty + 0.15 * c.authority,
        reverse=True,
    )

    for chunk in ranked:
        if chunk.novelty < 0.20:
            continue
        if source_counts.get(chunk.source_id, 0) >= per_source_limit:
            continue
        if used + chunk.tokens > evidence_budget:
            continue
        accepted.append(chunk)
        used += chunk.tokens
        source_counts[chunk.source_id] = source_counts.get(chunk.source_id, 0) + 1

    return accepted, used

Do not mistake this heuristic for ground truth. The purpose is to make an admission policy visible and testable. Your novelty signal could be cosine-distance from accepted chunks, heading overlap, a deduplication hash, or an evaluator. Your authority signal should come from a source registry rather than an LLM’s opinion.

If one excellent section is larger than the budget, do not blindly slice it at a character boundary. Prefer a precomputed passage, a parent section with an anchored excerpt, or a targeted extraction step. Keep the citation identifier with every excerpt so the answer can point back to what it used.

Reserve room for the answer before assembling evidence

The most subtle failure is calculating evidence first and letting the answer take whatever remains. That makes answer quality depend on retrieval volume.

Instead, choose an answer contract first. A short factual answer may reserve 300 tokens. A troubleshooting response with diagnosis, safe steps, and cited evidence may reserve 900 or more. A structured output contract should reserve room for its full schema and error case.

Set an evidence ceiling below the model limit even if you could fit more. This makes cost and latency predictable, and it makes behavior portable when you later route a request to a smaller model. If the candidate pack does not fit, apply a known fallback:

return the highest-authority answer with a scope note;
ask the user to narrow an aggregation request;
run a second retrieval pass for a missing sub-question; or
summarize a trustworthy parent document, while preserving provenance.

Avoid silently dropping the last chunks. Tail-drop creates position bias and makes results depend on incidental sort order.

Keep instructions, evidence, and tool output separate

Context packing is also a safety boundary. A retrieved document is evidence, not an instruction. Label it as untrusted source material, keep it outside the system instruction channel, and do not let its text decide which tools are available. This matters even for an internal knowledge base: imported tickets, copied documentation, and web pages can contain text that looks like a command.

Use a small envelope for every admitted passage, such as a source ID, title, version or fetch time, access label, and the excerpt itself. The model can cite the ID, while application code resolves it to a safe link. Do not invite the model to invent an origin from raw text.

Tool results deserve their own cap. A database query that returns 5,000 rows is not a better answer than a bounded aggregation with a continuation path. For tools, define a response schema, a maximum row or byte count, a pagination contract, and a redaction step before results reach the packer. If an action needs additional evidence, make a second bounded call instead of carrying every prior result through the conversation.

This separation makes incident review practical. You can answer three useful questions from a trace: which instruction governed the response, which sources supported it, and which tool result changed its state. A single concatenated prompt cannot answer those questions reliably.

Evaluate the policy, not just the final answer

An answer score alone cannot tell you whether the failure came from retrieval, reranking, packing, or generation. Add a trace for every sampled request:

{
  "query_class": "troubleshooting",
  "evidence_budget": 2600,
  "candidate_count": 30,
  "reranked_count": 12,
  "accepted_chunk_ids": ["docs-18#4", "release-7#2"],
  "evidence_tokens": 1412,
  "answer_reserve": 900,
  "dropped": {"duplicate": 5, "budget": 4, "source_cap": 1}
}

Then evaluate representative questions by query class. For each, record whether the expected evidence was retrieved, admitted, cited, and actually supports the answer. This separates four common diagnoses:

Observation	Likely issue	Next action
Needed source never appears	retrieval recall	improve indexing, filters, or candidate depth
Needed source is retrieved but excluded	packing/reranking	inspect budget, diversity, and score rules
Source is included but ignored	prompt or ordering	improve instructions, placement, or answer format
Source is cited but answer is wrong	generation/verification	add claim checks or require abstention

Measure cost, latency, groundedness, and task success together. Lowering context tokens is not a win if correct answers disappear. Conversely, sending more text is not a win if it only increases cost and makes attribution worse.

Ship it with a narrow rollout

Use a shadow trace first: calculate the proposed pack while still sending the current one. Compare token counts and whether expected evidence would have been admitted. Next, run a small traffic cohort with a kill switch.

Test deliberately awkward cases: duplicated policy pages, an old and a new version, a query that needs two documents, a broad “list all” request, and a long conversation with a short current question. Include authorization changes in this suite. A perfect pack that leaks a cross-tenant passage is still a failure.

The goal is not the smallest possible prompt. It is the smallest sufficient evidence set for a task, with a predictable answer reserve and a record of why each passage was admitted.

FAQ

What is a RAG context budget?

It is the portion of an LLM request deliberately reserved for retrieved evidence after instructions, relevant conversation, the answer, and a safety margin have been allocated. It turns context assembly into an explicit policy instead of a fixed top_k setting.

How many chunks should a RAG system send to the model?

There is no universal number. Retrieve enough candidates for recall, then evaluate how many diverse passages each query class needs. A point lookup may need one authoritative section; an aggregation question may need several. Use a token ceiling and test against real tasks.

Should I increase top-k when RAG answers are wrong?

Not automatically. First find out whether the needed source was missing, retrieved but excluded, included but ignored, or cited incorrectly. Raising top_k helps only the first case and can make the other three worse.

How do I prevent duplicate chunks in RAG context?

Store stable source and section identities, detect overlap or semantic similarity among candidates, and cap passages per source. Preserve the strongest chunk and its provenance rather than sending several near-identical versions of the same fact.

Does a larger context window remove the need for RAG packing?

No. Larger windows can hold more material, but they do not make every passage relevant, authorized, current, or easy for a model to use. Packing still controls cost, attention, provenance, and answer space.

What should I log for a RAG context budget?

Log the query class, token allocations, candidate and accepted IDs, source versions, drop reasons, latency, and the final citations. Avoid storing sensitive raw text unless retention and access rules explicitly allow it.

AI Agent Decision Logs: Record Evidence Without Storing Chain of Thought

Jack M — Sat, 10 Oct 2026 05:50:28 +0000

An agent can update a customer record, trigger a refund, or route a support case while every dashboard stays green. Then someone asks the only question that matters: why was this action allowed, and what actually changed? A trace may show tokens and latency. An application log may show an HTTP 200. Neither is a dependable answer.

The missing piece is an AI agent decision log: a small, structured record for each meaningful choice. It links the task, the bounded evidence, the policy and release in force, the chosen action, and the independently verified outcome. It is not a transcript, and it should not be a hidden-reasoning archive.

This guide shows how to build that record so an engineer can debug an incident, an operator can review a risky action, and a customer-data team does not inherit a second copy of every prompt.

A trace is not a decision record

Keep both. They answer different questions.

Artifact	Best for	Usually misses
Trace	latency, retries, spans, model calls	business intent and policy verdict
Tool log	request/response diagnostics	whether the response changed the target state
Prompt release manifest	which behavior bundle was deployed	why this request took this branch
Decision log	why the system was allowed to act and what was verified	low-level performance detail

Suppose a support agent changes a renewal status. A trace can show update_subscription completed in 180 ms. A useful decision log can say that the request came from an authenticated account owner, the account was eligible under renewal-policy-v4, the action stayed below the risk threshold, the expected version was 27, and the billing API confirmed version 28 with the requested status.

That is enough to investigate and reproduce the decision without pretending that a model's private intermediate text is a reliable audit artifact.

Record decisions, not every model turn

Logging every token is expensive, risky, and noisy. It also makes review harder. Define a decision boundary: log when the agent crosses from interpretation into a consequential branch or side effect.

Good boundaries include:

selecting a workflow or an external tool;
classifying a request into an approval or risk tier;
changing a record, sending a message, or charging money;
denying an action because policy blocked it;
escalating to a human;
declaring a long-running task complete.

Do not make a decision log for each retrieval chunk, retry, or sentence of a draft. Put those in traces and link their stable IDs when they matter. A compact log should be readable when an incident is active.

Start with one question

Before adding fields, ask: Could a teammate who was absent reconstruct the reason, authority, and result of this action from this one record?

If the answer is no, add the missing reference. If the answer needs the full prompt, full customer record, and every generated token, redesign the workflow so the record can cite safe, scoped evidence instead.

A practical decision-log schema

The schema needs stable identities, not a giant JSON blob. Here is a TypeScript shape for a material action:

type DecisionLog = {
  id: string;
  occurredAt: string;
  tenantId: string;
  runId: string;
  decision: "route" | "allow" | "hold" | "block" | "complete";
  task: { type: string; requestRef: string };
  release: { manifestId: string; model: string; policyVersion: string };
  evidence: Array<{ kind: string; ref: string; sha256?: string }>;
  action?: { tool: string; targetRef: string; idempotencyKey: string };
  policy: { verdict: "allow" | "hold" | "block"; reasons: string[] };
  outcome: { state: "verified" | "rejected" | "unknown"; resultRef?: string };
  redactionVersion: string;
};

Three choices matter here.

First, requestRef, targetRef, and resultRef point to access-controlled data; they do not copy it. A support transcript or invoice belongs in its existing system with its existing retention rules.

Second, store versioned identifiers for the release, model, and policy. An answer may be acceptable under one policy version and blocked under the next. Without IDs, you cannot explain a historical decision honestly.

Third, store the verdict and its public reasons, such as account_owner_verified or refund_below_limit. Do not store a fabricated summary of hidden reasoning. Design the policy engine to emit explicit reason codes that are safe to expose to operators.

Capture evidence before the action, then verify after it

The safest shape is a two-phase record. Write a planned decision before an external side effect; append the verified result after it.

async function changeRenewal(input: RenewalInput) {
  const decision = await decide(input); // typed facts + deterministic policy
  const record = await decisions.insert({
    ...decision,
    outcome: { state: "unknown" },
  });

  if (decision.policy.verdict !== "allow") return record;

  const result = await billing.updateRenewal({
    subscriptionId: input.subscriptionId,
    expectedVersion: input.version,
    idempotencyKey: record.action!.idempotencyKey,
  });

  return decisions.appendOutcome(record.id, {
    state: result.version === input.version + 1 ? "verified" : "unknown",
    resultRef: `billing-event:${result.eventId}`,
  });
}

Why not write the record only after success? Because a timeout, worker crash, or duplicate retry is itself important. The planned record gives you a durable idempotency key. The outcome distinguishes attempted, verified, and unknown instead of silently turning uncertainty into success.

For high-risk actions, send hold records to an approval queue. The reviewer should see the task summary, safe evidence links, policy reason codes, expected change, and expiry. Their approval becomes another append-only event, not an edit to history.

Keep private data out of the log by default

A decision log can become a data leak if it stores raw prompts, retrieval text, tool arguments, or secrets. Use a privacy budget as deliberately as a token budget.

Store directly	Store by reference or hash	Never store in the decision log
policy version, verdict, time, action type	request ID, document ID, sanitized evidence hash	API keys, access tokens, raw passwords
model and release IDs	encrypted artifact location	full hidden reasoning
idempotency key, result status	redacted tool-response reference	unrestricted customer transcript

Run a redactor before persistence, and version it. A later investigation needs to know which rule set removed or transformed a field. Do not solve this with a vague instruction like “avoid PII.” Make it a schema rule: only allow-listed fields can enter the decision table.

Also separate read permissions. An on-call engineer may need policy codes and status, while a privacy officer may be the only person allowed to open the underlying request reference. The decision log should remain useful even when its linked artifacts are unavailable to the reader.

Make records replayable without replaying production

Replay means testing whether the same typed facts and policy would produce the same verdict—not firing the live tool again.

Create a replay fixture with:

sanitized input facts;
the referenced release and policy versions;
expected decision, reason codes, and action contract;
a fake tool response or recorded result shape.

it("holds a renewal change when ownership evidence is missing", async () => {
  const result = await evaluate({
    accountOwnerVerified: false,
    requestedChange: "renew",
    riskAmount: 0,
  }, policy("renewal-policy-v4"));

  expect(result.verdict).toBe("hold");
  expect(result.reasons).toContain("account_owner_not_verified");
});

This is more stable than asking a model to narrate its prior thinking. Keep LLM interpretation upstream, constrained by a typed output schema. Make the consequential verdict deterministic where possible. If the model must classify, capture the allowed label, evidence references, confidence as a diagnostic signal, and the policy that converted the label into an action.

Query records like an incident responder

Good records turn vague questions into bounded queries:

“Which policy version allowed this action?”
“Did we verify the external change or merely receive a response?”
“Which release caused the first hold spike?”
“Did a retry reuse the same idempotency key?”
“Which evidence reference was unavailable during the decision?”

Build a small operational view around those questions. Show the decision timeline, state transitions, policy reason codes, release ID, and safe links. Do not start with a polished chat summary; start with data that makes a summary checkable.

An especially useful alert is a rising count of unknown outcomes. It catches the uncomfortable middle state: a tool call may have reached the destination, but your service cannot yet prove what happened. That deserves reconciliation, not an automatic retry.

Common failure modes

Logging an unbounded prompt blob

It feels thorough until a customer asks for deletion, a secret appears in context, or an operator needs one fact quickly. Store a constrained summary and references; keep original data in the system responsible for it.

Treating a model confidence score as authority

Confidence is not permission. A high score does not replace ownership checks, spend limits, or a reviewer. Policy should decide whether a classification may become an action.

Calling an HTTP response “verified”

An accepted request can still be rejected later, applied twice, or applied to the wrong version. Prefer a returned version, downstream event ID, read-after-write check, or reconciliation job.

Making the log editable

Corrections are valuable; overwriting history is not. Append a correction event that names the original record, the actor, and the reason. That preserves the investigation trail.

A rollout plan that will not stall the team

Start with one high-value tool action, such as changing a ticket state or creating an outbound message. Define its input facts, risk tier, policy reasons, idempotency key, and verification method. Put the log schema under test before adding a dashboard.

Next, add hold and unknown states. Teams often log only success and failure, but these two states capture the real operational ambiguity. Finally, link the record to your existing trace and release manifest. You get an explanation layer without replacing observability, deployments, or access control.

The result is less glamorous than an autonomous demo, but more useful in production: an agent can act quickly and still leave behind a clear, privacy-aware explanation of what it was allowed to do and what the system proved afterward.

Test the contract in CI

Treat a decision log as an API contract. If a release removes the policy version, changes a reason code, or starts serializing a raw email address, the build should fail before that change reaches production.

Start with three small tests:

Schema test: every required field is present and every reference uses an allowed prefix.
Redaction test: realistic prompts and tool responses cannot introduce secrets, direct identifiers, or unapproved fields.
Transition test: unknown may become verified or rejected, but a terminal outcome cannot be silently overwritten.

it("does not persist raw customer text", () => {
  const record = toDecisionLog({
    task: { summary: "Email ava@example.com about invoice 8841" },
    policy: { verdict: "hold", reasons: ["human_review_required"] },
  });

  expect(JSON.stringify(record)).not.toContain("ava@example.com");
  expect(record.task.requestRef).toMatch(/^request:/);
});

Use fixtures from incidents and near misses after redacting them. A fixture that proves a duplicated retry preserves its idempotency key is more valuable than a generic happy-path example. When a policy changes intentionally, review the fixture diff as carefully as an application permission change.

One final rule helps with multi-agent systems: assign a parentDecisionId when one agent delegates to another. The child still logs its own policy and result. This lets an investigator follow the tree from a user request to a tool action without merging all agents into one unsearchable transcript.

FAQ

What should an AI agent decision log contain?

Include a stable decision ID, time, task reference, release and policy versions, safe evidence references, policy verdict and reason codes, the planned action, idempotency key, and a verified or unknown outcome. Avoid raw prompts and hidden reasoning.

Are AI agent decision logs the same as observability traces?

No. Traces diagnose execution performance and call paths. Decision logs explain a consequential branch or action in business and policy terms. Link the two rather than forcing either one to do both jobs.

Should an AI agent decision log store chain of thought?

No. Store explicit, reviewable policy reasons and safe evidence references instead. They are more reliable for operations and avoid creating an unnecessary store of sensitive, unbounded model text.

How do I verify an AI agent action actually happened?

Use an idempotency key plus a domain-level confirmation: a returned version, durable downstream event, read-after-write check, or reconciliation process. Record unknown until that confirmation exists.

How long should decision logs be retained?

Match retention to the underlying business process, contractual needs, and privacy policy. Keep the smallest useful record, retain linked sensitive artifacts under their own controls, and test deletion and access rules.

Can small teams implement this without a new platform?

Yes. Start with an append-only database table or event stream, typed schema validation, policy reason codes, and links to existing traces and records. Add specialized tooling only after the basic contract is working.

Prompt Release Manifests: Ship AI Changes Without Breaking Production

Jack M — Sun, 27 Sep 2026 10:38:47 +0000

A one-line prompt edit can change what your application does. It can choose a different tool, omit a warning, produce invalid JSON, or spend twice as many tokens. Yet after a bad response reaches a customer, many teams still cannot answer the first incident question: what exact behavior was running?

Git history is necessary, but it is not enough. A production response depends on more than text in a prompt file. It also depends on the model and settings, output schema, retrieval index, tool descriptions, policy rules, and feature flags resolved at request time.

A prompt release manifest packages those dependencies into one immutable, testable unit. This guide shows a vendor-neutral way to create one, promote it safely, and trace a response back to the release that caused it.

Why a prompt is not the unit you deploy

It is tempting to treat a prompt as a string:

const instructions = "Answer from the account record. Return JSON.";

That works until a behavior change spans two systems. Imagine that you change the instruction to ask for a reason field, upgrade the model, and add a new billing tool. The application code still deploys successfully. But the old JSON validator rejects the new field, the model calls the tool for questions it previously answered directly, and traces only record the model name.

No individual change looks dramatic. The combined release is the behavior users experience.

Treat this tuple as the release unit:

Component	Why it belongs in the release
Prompt template and examples	Changes language, priorities, and tool choice
Model and generation settings	Changes reasoning, format reliability, latency, and cost
Output schema	Defines what downstream code may safely consume
Retrieval configuration	Changes the evidence available to the model
Tool contracts and permissions	Changes what the workflow can do
Policy version	Changes allowed actions and escalation rules
Evaluation dataset and thresholds	Proves why the candidate may be promoted

The goal is not to invent paperwork. It is to make a response reproducible when something goes wrong.

Separate artifact identity from deployment state

This distinction prevents a subtle class of production bugs. An artifact answers, “what did we test?” A deployment label answers, “what should eligible traffic use right now?” If a dashboard lets someone edit prompt text behind a production label, it has mixed those two jobs. You can no longer compare an evaluation result with the exact behavior served later.

Keep the release artifact append-only. Keep deployment state small and auditable: label, previous label target, actor, time, cohort rule, and change reason. That leaves a compact trail even when several people work on the same feature. It also lets a developer reproduce a customer issue locally by fetching one ID instead of guessing which combination of configuration happened to be live.

Define a small, immutable manifest

Start with plain YAML or JSON in the repository. A database-backed registry is useful later, but a reviewable file is a good first control surface.

# releases/support-answer/2026-09-27.3.yaml
id: support-answer-2026-09-27.3
owner: support-platform
prompt:
  id: support-answer
  sha256: "8b4a…d91e"
  variables: [customer_message, account_snapshot, policy_excerpt]
model:
  provider: openai
  id: gpt-6-luna
  temperature: 0.1
  max_output_tokens: 500
output:
  schema_id: support-answer-v4
  schema_sha256: "f03c…7ab2"
retrieval:
  index: billing-kb
  index_version: "2026-09-26.2"
  filters: [tenant_id, published]
tools:
  - name: get_invoice
    contract_version: v2
    permission_policy: read-billing-v3
policy:
  version: support-escalation-v5
evaluation:
  dataset: support-golden-v12
  dataset_sha256: "a10e…4d77"
  run: eval-1842
  gates:
    schema_valid_rate: { min: 0.995 }
    grounded_answer_rate: { min: 0.98 }
    unsafe_action_rate: { max: 0 }

Use an immutable id, not an editable label such as latest. A mutable label has a place—staging or production—but it should point to a release ID. The manifest itself should never change after promotion. A correction creates a new release.

Hashes matter when artifacts live outside Git. They let an incident responder distinguish “the prompt with this name” from the exact prompt, schema, or dataset used in a run.

Keep secrets and customer data out

Do not place API keys, raw customer examples, or complete production documents in the manifest. Store references, version IDs, and redacted test fixtures instead. A release file often becomes visible to every engineer who can review a change; make it safe to review.

Make compatibility explicit before an evaluation

An evaluation can tell you a candidate is worse. It will not automatically tell you whether your application can parse its output or whether a tool call violates a new policy. Add fast compatibility checks first.

For each change, ask four questions:

Can all declared template variables be rendered?
Does the output validate against the versioned schema?
Can the declared tool contracts be invoked with the release's permissions?
Does the retrieval index support the required tenant and publication filters?

The following TypeScript sketch fails a build before a candidate reaches an expensive model evaluation:

type Manifest = {
  id: string;
  prompt: { variables: string[] };
  output: { schema_id: string };
  tools: Array<{ name: string; contract_version: string }>;
};

export function validateManifest(m: Manifest, registry: Registry) {
  const missing = m.prompt.variables.filter(v => !registry.variableTypes.has(v));
  if (missing.length) throw new Error(`Unknown prompt variables: ${missing.join(", ")}`);

  if (!registry.schemas.has(m.output.schema_id)) {
    throw new Error(`Unknown output schema: ${m.output.schema_id}`);
  }

  for (const tool of m.tools) {
    if (!registry.toolContracts.has(`${tool.name}@${tool.contract_version}`)) {
      throw new Error(`Unknown tool contract: ${tool.name}@${tool.contract_version}`);
    }
  }
}

This is deliberately boring. Boring checks prevent incidents that no prompt wording can fix.

Evaluate the candidate against the current release

Run the same representative cases against the current production release and the candidate. Do not judge a candidate only against a hand-picked example that inspired the change.

Your golden set should include normal requests, short and ambiguous messages, missing records, stale retrieval results, permission boundaries, tool failures, and requests that require escalation. Keep the expected answer narrow where possible: schema validity, grounded claims, correct route, allowed tool use, and a human-review requirement can often be checked without asking another model for a vague quality score.

Store the comparison with the manifest rather than in a dashboard comment:

{
  "candidate": "support-answer-2026-09-27.3",
  "baseline": "support-answer-2026-09-18.1",
  "cases": 184,
  "schema_valid_rate": { "baseline": 1, "candidate": 1 },
  "grounded_answer_rate": { "baseline": 0.989, "candidate": 0.994 },
  "unsafe_action_rate": { "baseline": 0, "candidate": 0 },
  "review": "approved"
}

The numbers above are an example format, not a universal pass threshold. Define thresholds per workflow. A support-draft feature may tolerate a lower stylistic score than a workflow that changes a subscription or sends an email.

Promote with a pointer, not a code redeploy

The serving application should resolve one release ID at the start of a request and attach it to the request context. It should never resolve production again halfway through a tool-using workflow.

async function beginAiRequest(input: Input) {
  const releaseId = await releases.resolveLabel({
    label: "production",
    tenantId: input.tenantId,
  });

  const release = await releases.getImmutable(releaseId);
  const trace = tracer.startSpan("ai.request", {
    attributes: { "ai.release_id": release.id }
  });

  try {
    return await runWorkflow({ input, release, trace });
  } finally {
    trace.end();
  }
}

Promotion becomes a controlled pointer change:

production -> support-answer-2026-09-18.1
production -> support-answer-2026-09-27.3

That separation matters. A rollback is then a pointer change to a known-good release, not a frantic attempt to reconstruct an earlier deployment from commits, caches, and environment variables.

Use cohorts before full promotion

Start a candidate with a low-risk cohort: internal users, a test tenant, or a small percentage of eligible read-only requests. Preserve the resolved release ID on every trace, output validation failure, tool call, and user feedback event.

Compare candidate and baseline by release ID. Watch the measures that matter to the workflow: invalid responses, escalation rate, task completion, tool errors, latency, and token use. Do not promote because a general “thumbs up” average looks good when policy violations or parse failures are hiding underneath it.

For write-capable workflows, shadow evaluation is often safer than a traffic split. Let the candidate produce a proposed action, compare it with the baseline or a reviewed expectation, then discard it. Never let a shadow run repeat an external side effect.

Plan rollback as a test, not a hope

Rolling back the prompt alone may not restore behavior. The candidate might depend on a schema, tool contract, or retrieval index that changed at the same time. That is why the manifest contains the whole compatible set.

Rehearse this runbook:

Detect a release-scoped regression from traces or a gate alert.
Freeze promotion and identify the last healthy release ID.
Move the production label back to that ID.
Confirm new requests resolve the old ID in each region and cache layer.
Stop or safely drain in-flight work that has not resolved a release yet.
Compare errors and outcomes by release ID, then create a new candidate instead of editing history.

Cache behavior deserves special attention. If one process caches the production label for ten minutes and another for thirty seconds, you do not have an instant rollback. Include label-cache TTL and region in trace attributes, and test a rollback during a calm weekday—not during an incident.

Make traces useful to the person on call

At minimum, record this data on every AI request:

release_id
prompt_hash
model_id and provider response version when available
schema_id
retrieval_index_version
tool contract versions
policy version
tenant-safe evaluation cohort
label-cache state

Do not log the entire prompt or user content by default. Store redacted identifiers and use restricted debugging access for sensitive payloads. Traceability should improve incident response without turning observability into a second data leak.

A practical adoption path

You do not need a prompt platform to adopt this pattern.

Week one: move production prompts into reviewed files, add an immutable release ID, and log it.
Week two: add schema and tool-contract validation plus a small golden set of cases.
Week three: promote through a staging label and gate production on a recorded comparison.
Week four: add cohort rollout, release-scoped dashboards, and a rollback rehearsal.

The smallest useful rule is simple: every production response must name the release that produced it. Once that is true, quality work becomes much less mysterious.

Release checklist

Before moving a prompt release to production, confirm:

[ ] The manifest is immutable and reviewed.
[ ] Prompt, model, schema, retrieval, tools, and policy versions are pinned.
[ ] Template variables and tool contracts passed compatibility checks.
[ ] Candidate and baseline ran against the same relevant test set.
[ ] Gates match the risk of the workflow.
[ ] The rollout cohort and stop conditions are defined.
[ ] Every request records the resolved release ID.
[ ] Rollback points to a known compatible release and has been rehearsed.

FAQ

What is a prompt release manifest?

It is an immutable record of the full AI behavior you are deploying: prompt, model settings, schema, retrieval version, tool contracts, policy, and evaluation evidence. A label such as production points to one manifest at a time.

Is Git enough for prompt versioning?

Git is an excellent authoring and review surface, but it does not by itself show which prompt, model, index, and feature configuration a live request resolved. Pair Git history with an immutable runtime release ID and trace it on every request.

Should every prompt edit get a new release ID?

Any edit that can affect production behavior should. Drafts can change freely, but once a candidate is evaluated or exposed to users, give it a new immutable ID so results and incidents remain reproducible.

How is a prompt release manifest different from an evaluation harness?

An evaluation harness measures behavior. A release manifest identifies exactly what behavior was measured and later served. Use both: the manifest links a candidate to the dataset, thresholds, and evaluation run that justified promotion.

Can I roll back a prompt without redeploying the application?

Yes, if the application resolves a production label to an immutable manifest at request start. Move the label to a previously compatible release, then verify cache and region propagation. Do not rely on editing a live prompt in place.

What should a team log for AI release debugging?

Log the resolved release ID plus prompt hash, model identifier, output schema, retrieval version, tool contracts, policy version, and safe rollout cohort. Avoid logging raw sensitive content unless access controls and retention rules justify it.

How to Combine Rules and LLMs for Reliable AI Workflows

Jack M — Wed, 23 Sep 2026 11:39:17 +0000

An LLM can write a convincing answer while making the wrong decision. That is the uncomfortable gap behind many production AI failures: the system treats a probabilistic model as the final authority on facts that should be fixed, authorized, or calculated exactly.

The answer is not to remove the LLM. It is to give it the right job. Use deterministic code for boundaries and decisions that must be repeatable; use the LLM for interpretation, extraction, explanation, and ambiguity. This article shows how to build that split, test it, and keep it useful as the workflow grows.

The core pattern: rules decide, LLMs interpret

Think of an AI workflow as four separate layers:

Layer	Best job	Should it be deterministic?
Input boundary	validate identity, schema, size, and consent	Yes
Decision policy	permissions, pricing, eligibility, risk, routing	Yes
Language layer	classify messy text, extract fields, summarize, explain	No, but constrained
Action boundary	execute approved operations and record evidence	Yes

This is not a claim that models cannot reason. It is a design choice: do not ask a model to enforce a rule your application can enforce exactly. Recent AWS guidance for agentic systems similarly recommends server-side checks at the point of tool use, least-privilege boundaries, tracing, and canary testing for behavior changes.

For example, a support workflow may ask an LLM to read a customer message and return a structured intent. The application, not the model, decides whether a refund is allowed and whether a ticket can be changed.

customer message
  -> LLM extracts intent + evidence
  -> schema validation
  -> deterministic policy evaluates eligibility
  -> approved action or human review
  -> LLM explains the outcome in plain language

The key benefit is debuggability. When a refund is blocked, you can tell whether extraction failed, a required fact was missing, or a rule intentionally denied the action. “The model decided” is not an operational explanation.

Find the decisions that should never depend on a prompt

Start by listing every workflow decision and placing it in one of three buckets.

Hard rules

Hard rules have a single correct answer from trusted inputs. Put them in code, a policy engine, or a database constraint.

Is the caller authenticated for this tenant?
Is the requested action within their role and spend limit?
Is an invoice overdue according to the ledger?
Does a request satisfy a required JSON schema?
Is a tool call idempotent, rate-limited, and within its allowed scope?

An LLM can help explain a denial, but it should not be the source of truth for it.

Soft judgments

Soft judgments need context, language understanding, or a trade-off. These suit an LLM, ideally with an explicit output contract.

Which issue category best fits this free-form report?
Which facts in a document answer the user’s question?
Is this explanation clear enough for a nontechnical reader?
Which knowledge-base article is most relevant after retrieval?

Escalation cases

Some tasks are neither safe to automate nor simple enough for a static rule. Route them to a reviewer with the facts the reviewer needs. Do not hide uncertainty behind confident prose.

type Disposition =
  | { kind: "allow"; policyId: string }
  | { kind: "deny"; policyId: string; reason: string }
  | { kind: "review"; policyId: string; questions: string[] };

function decideRefund(input: {
  tenantId: string;
  actorRole: "admin" | "agent" | "viewer";
  amountCents: number;
  daysSincePurchase: number;
}): Disposition {
  if (input.actorRole === "viewer") {
    return { kind: "deny", policyId: "refund-role-v1", reason: "role cannot issue refunds" };
  }
  if (input.amountCents > 50_000) {
    return { kind: "review", policyId: "refund-limit-v1", questions: ["Is manager approval attached?"] };
  }
  if (input.daysSincePurchase > 30) {
    return { kind: "deny", policyId: "refund-window-v1", reason: "outside refund window" };
  }
  return { kind: "allow", policyId: "refund-standard-v1" };
}

Notice what is missing: no model call is needed to determine the limit. That makes the result stable, unit-testable, and easy to audit.

Give the model a narrow, typed contract

The LLM should return observations, not permissions. A good contract names the facts it may infer, records uncertainty, and leaves sensitive decisions to code.

import { z } from "zod";

const SupportSignal = z.object({
  intent: z.enum(["refund_request", "billing_question", "technical_issue", "other"]),
  orderId: z.string().regex(/^ord_[a-z0-9]+$/).optional(),
  requestedAmountCents: z.number().int().nonnegative().optional(),
  evidence: z.array(z.string()).max(3),
  confidence: z.number().min(0).max(1),
  needsHumanReview: z.boolean()
});

type SupportSignal = z.infer<typeof SupportSignal>;

In the prompt, ask for only this JSON shape. After receiving it, parse it with the schema. If parsing fails, retry once with a repair instruction or route the task to review. Do not silently coerce a malformed amount, unknown intent, or invented order ID into an action.

This division also protects against an easy mistake: treating text returned by a document, browser page, or tool as authority. Retrieved content can influence the LLM’s interpretation; it must not be allowed to override authorization or policy. Fetch authoritative facts from the systems that own them.

Build a decision pipeline, not a giant prompt

An implementation becomes easier to reason about when each stage has one responsibility.

async function handleSupportMessage(request: Request) {
  const context = await authenticateAndLoadTenant(request);
  const message = await validateIncomingMessage(request);

  const signal = SupportSignal.parse(await extractSignal(message));
  const order = signal.orderId
    ? await loadOrderForTenant(context.tenantId, signal.orderId)
    : null;

  const disposition = !order || signal.needsHumanReview || signal.confidence < 0.8
    ? { kind: "review", policyId: "missing-or-uncertain-facts-v1", questions: ["Verify order and request details"] }
    : decideRefund({
        tenantId: context.tenantId,
        actorRole: context.actorRole,
        amountCents: signal.requestedAmountCents ?? 0,
        daysSincePurchase: daysSince(order.purchasedAt)
      });

  await writeAuditRecord({ context, signal, disposition, orderId: order?.id });

  if (disposition.kind !== "allow") return renderSafeResponse(disposition, signal);
  return executeIdempotentRefund(context, order!, disposition.policyId);
}

This pattern makes threat boundaries visible:

authenticateAndLoadTenant gets identity from a trusted session, not model output.
loadOrderForTenant scopes the lookup before it reaches a business rule.
decideRefund is pure, so it can be tested with a table of cases.
executeIdempotentRefund owns the side effect and records the policy that allowed it.

The LLM can still make the experience feel natural. It can extract fields from a rambling request, draft a kind response, and suggest what information is missing. It cannot manufacture the right to change a record.

Handle disagreement without pretending the model is a judge

Many teams add a second model when the first output looks uncertain. That can be useful for language tasks, but it is not a substitute for policy. Two models agreeing that an action is allowed does not make it authorized.

Use disagreement as a routing signal instead:

Compare model extraction with trusted records and deterministic checks.
If the facts conflict, stop the automated path.
Preserve the candidate output, relevant evidence, and policy result for review.
Turn the resolved case into a regression fixture.

For a classification task, you might use a small model for a first pass and a stronger model only when confidence is low. For a payment, permission, deletion, or external message, the final authority should still be an application policy plus any required human approval.

Measure the workflow at three levels

HTTP success tells you very little. A 200 response can conceal a wrong classification, a denied action that should have been allowed, or a costly retry loop.

Track three metric layers for each version of the workflow:

Layer	Examples	Promotion question
Contract	schema-valid output, required facts present, tool argument checks	Did the interface remain intact?
Outcome	accepted resolution, correction rate, reviewer reversal, safety violation	Did it help the user without breaking policy?
Operations	p95 latency, error rate, cost per completed task, queue age	Can the system sustain it?

This mirrors a useful observation from a feature-flag guide for LLM features: computational metrics are easy, deterministic behavioral checks need more setup, and semantic quality needs a rubric or human labeling. Do not let a latency improvement hide a semantic regression.

Segment metrics by tenant plan, language, workflow type, and risk tier. A global average can be healthy while a low-volume but important category fails badly.

Test rules and model behavior differently

Rules deserve ordinary unit tests because they should behave the same way every time.

import { expect, test } from "vitest";

test("large refunds always require review", () => {
  expect(decideRefund({
    tenantId: "t_1", actorRole: "admin", amountCents: 50_001, daysSincePurchase: 2
  })).toMatchObject({ kind: "review", policyId: "refund-limit-v1" });
});

Model behavior needs a different test set: representative messages, malformed inputs, adversarial instructions inside quoted text, sparse evidence, multilingual requests, and cases that must be escalated. Assert hard properties exactly—schema validity, no unapproved tool call, correct tenant scope. Score soft properties with a rubric and sampled human review.

Keep a fixture whenever a production case is corrected. Over time, the suite becomes a map of real customer ambiguity, not a collection of polished demos.

Roll out a policy-model change safely

Treat a prompt, retrieval setting, tool description, model, and policy rule as a release bundle. Record version IDs for each piece. If two components change at once, you cannot tell which one improved or harmed the result.

Use a small progression:

Offline replay: run candidate and incumbent against saved, de-identified fixtures.
Shadow path: run the candidate beside production, but keep it away from side effects and user-visible results.
Limited exposure: give a stable, low-risk cohort the candidate path. Keep an immediate switch back.
Promotion: expand only when contract, outcome, and operational thresholds pass with enough representative traffic.

Before the first user sees a change, write veto conditions. Examples include any cross-tenant lookup, any unauthorized action, schema failure above a small tolerance, or a correction rate above the incumbent. These are not averages to trade away for a prettier answer.

A practical starting checklist

If your current workflow is one large prompt, do not rewrite everything. Pick one irreversible or high-volume path and work through this list.

List every proposed action and identify its authoritative data source.
Move permission, eligibility, spend, and tenant-scope checks outside the prompt.
Make the LLM output observations in a validated schema.
Use trusted records to confirm identifiers before an action.
Add an explicit review disposition; uncertainty should have a safe destination.
Make writes idempotent and log the policy version, input references, and outcome.
Test rules with tables and model behavior with representative fixtures.
Define release thresholds and vetoes before a new version receives traffic.

The goal is not a less capable AI feature. It is an AI feature whose strengths are useful and whose boundaries are inspectable. Let the model handle language and ambiguity. Let the system enforce the facts, permissions, and actions that customers depend on.

FAQ

When should I use rules instead of an LLM?

Use rules when a decision has a trusted input and a repeatable answer: authorization, pricing, eligibility, tenant scope, rate limits, schemas, and side-effect permissions. Use an LLM when the input is ambiguous language or the output is explanatory.

Can an LLM call a rule engine?

Yes. The safe pattern is for the LLM to request a narrow, validated operation, while the application or policy service evaluates the rule and returns the result. The model should not be able to bypass that service or supply its own authorization facts.

How do I prevent an AI workflow from acting on hallucinated IDs?

Validate the model’s output against a strict schema, then retrieve the record from an authoritative system using the authenticated tenant scope. Never use a model-generated identifier as proof that a record exists or belongs to the caller.

What should happen when the model is uncertain?

Make uncertainty explicit in the output contract and route it to a safe fallback: ask for missing information, use a deterministic alternate path, or create a review task. Do not lower a confidence threshold merely to increase automation.

Do deterministic rules make an AI feature less flexible?

They make high-stakes edges less flexible on purpose. The LLM remains free to understand varied language, summarize, and explain. Rules simply reserve fixed decisions and external actions for components that can be tested and audited exactly.

How should I test hybrid AI workflows?

Unit-test policies and side-effect guards with exact cases. Test the model layer with representative and adversarial fixtures, then assert strict properties such as schema validity, tenant scope, and forbidden-action avoidance. Evaluate softer qualities with rubrics and sampled review.

Conversation Regression Testing for AI Agents: Catch Multi-Turn Failures Before Production

Jack M — Sat, 19 Sep 2026 15:16:25 +0000

An agent can give a convincing final answer and still fail the user three turns earlier. It may forget an account constraint, call the wrong tool, accept a correction it should reject, or carry a stale fact into every later decision. A one-prompt test will happily pass.

That is why production agents need conversation regression testing: replaying a complete, realistic interaction after a change and checking properties that span the whole trajectory. This guide shows how to build a small, useful suite without pretending model output will be byte-for-byte deterministic.

The payoff is concrete: when you change a model, prompt, tool, retrieval source, or memory policy, CI can tell you whether a familiar customer journey still completes safely—and where it first went wrong.

Why single-turn tests miss the expensive failures

Single-turn tests are still valuable. They catch malformed structured output, unsafe tool arguments, and obvious retrieval errors quickly. But a customer-facing agent is stateful. Each turn changes what the next turn sees.

Consider a support agent helping a customer change a subscription:

The customer identifies their workspace.
They ask about plan options.
They clarify that an annual invoice is already paid.
The agent proposes a change.
The customer approves it.

An agent can answer turn 5 politely while violating the constraint introduced at turn 3. A final-answer judge may call the response helpful; your billing system will call it a defect.

Conversation tests reveal four failures that prompt tests routinely hide:

Failure	What a single prompt misses	Conversation-level assertion
Context loss	The final answer looks plausible	A previously confirmed constraint remains active
Bad trajectory	The answer is right for the wrong reason	Required tool calls happen in the approved order
Cascading error	Later turns inherit an early mistake	The first failing turn is recorded
Unsafe recovery	The agent retries an action after a correction	A correction invalidates pending state and side effects

The key idea is simple: test a conversation as a sequence of state transitions, not as a bag of independent answers.

Start with journeys, not a giant benchmark

Do not begin by generating thousands of synthetic chats. Start with 10–20 journeys that represent expensive, frequent, or risky work:

a customer changes a requirement halfway through;
an API returns incomplete data and the agent must ask a question;
a user asks for a write action, then withdraws approval;
a retrieved policy conflicts with a remembered preference;
a tool times out and the agent must not duplicate the side effect.

Good fixtures come from anonymized support transcripts, bug reports, sales-engineering handoffs, and incidents. Remove personal data, replace identifiers with stable test values, and record the business rule the journey protects. A fixture is not a transcript archive; it is an executable statement of what must remain true.

Here is a compact TypeScript shape:

type Turn = {
  user: string;
  expected?: {
    tool?: string;
    mustInclude?: string[];
    mustNotInclude?: string[];
  };
};

type ConversationFixture = {
  id: string;
  risk: "low" | "medium" | "high";
  initialState: Record<string, unknown>;
  turns: Turn[];
  invariants: string[];
};

const cancelBeforeWrite: ConversationFixture = {
  id: "cancel-before-plan-change",
  risk: "high",
  initialState: { workspaceId: "ws_test", plan: "annual" },
  turns: [
    { user: "Change us to the monthly plan.", expected: { tool: "quote_plan_change" } },
    { user: "Wait, do not make any changes yet." },
    { user: "What would the prorated amount be?", expected: { tool: "quote_plan_change" } }
  ],
  invariants: ["no plan_change write occurs", "withdrawal is acknowledged"]
};

This fixture is deliberately small. It tests a real product promise: a quote is not authorization, and a withdrawn request cannot leak into a later tool call.

Record enough evidence to replay the path

You cannot debug a failed conversation with only the final text. Store a trace for each turn with the input, selected tool, sanitized arguments, tool result, state revision, prompt revision, model ID, and an immutable run ID.

Keep secrets and raw customer data out of fixtures. Store references or redacted values instead. If a tool result changes outside your control, use a test double for CI and separately run a small nightly suite against a safe staging environment.

type TraceStep = {
  turn: number;
  stateVersion: string;
  assistantText: string;
  toolCalls: Array<{ name: string; args: unknown; resultCode: string }>;
  promptVersion: string;
  model: string;
};

The distinction matters. A replay fixture supplies stable inputs. A trace explains the observed trajectory. Together they let a reviewer answer, “Did the model change, did our prompt change, or did a tool contract change?”

Assert invariants, not exact prose

Exact-string assertions are brittle because legitimate wording changes. Purely subjective judging is brittle because it can miss deterministic safety failures. Use both, in layers.

1. Deterministic checks first

Make product and security rules ordinary code. Examples:

function assertNoWriteAfterWithdrawal(trace: TraceStep[]) {
  const withdrewAt = trace.findIndex(s =>
    /do not make|cancel|withdraw/i.test(s.assistantText)
  );
  const writes = trace.slice(withdrewAt + 1)
    .flatMap(s => s.toolCalls)
    .filter(c => c.name === "change_plan");

  if (writes.length) throw new Error("plan changed after withdrawal");
}

function assertWorkspaceScope(trace: TraceStep[]) {
  for (const call of trace.flatMap(s => s.toolCalls)) {
    if (call.name === "get_invoice" && !JSON.stringify(call.args).includes("ws_test")) {
      throw new Error("tool call escaped fixture workspace");
    }
  }
}

Prefer assertions on permissions, schema validity, tool order, idempotency keys, source citations, and state transitions. They are fast, explainable, and cheap enough for every pull request.

2. Use rubric judges for meaning

Then use a separate model or human review for properties that cannot be reduced to a rule: did the agent acknowledge the correction, ask a necessary clarifying question, or preserve the user’s goal?

Give the judge a narrow rubric and the relevant evidence. Do not ask, “Is this good?” Ask, “After the user withdrew authorization, did the agent promise or initiate a plan change? Return pass, fail, or uncertain with the first supporting turn.” Treat uncertain as review-needed, not as a pass.

3. Compare distributions when the model is variable

Run high-risk fixtures several times. A pass rate of 10/10 is more meaningful than a fortunate 1/1, but do not turn every CI job into an expensive Monte Carlo experiment. A practical split is:

PR checks: deterministic mocks plus one seeded or low-temperature run;
nightly: 3–10 runs for high-risk journeys and broader user simulations;
release candidate: full suite, with failures triaged before promotion.

Set a budget per suite. Regression testing should prevent surprise spend, not create it.

Attribute the first failure, not just the unhappy ending

The most useful output is not “conversation failed.” It is “turn 2 selected an unscoped invoice lookup; turns 3–5 inherited the wrong workspace.” That transforms an opaque judge score into an engineering task.

For each test, record:

first_failure_turn;
violated invariant or rubric item;
causal tool call, retrieval item, or state transition;
downstream turns affected;
prompt, model, tool, and knowledge-base versions.

This separation prevents a common mistake: fixing the last bad answer instead of the earlier decision that made it inevitable. It also makes failures comparable across releases.

Add forks for the moments where users change direction

Linear transcripts cover the happy path. Real conversations fork at moments of uncertainty: a user corrects an assumption, refuses an action, or reveals a constraint. You do not need to regenerate the identical first four turns for every branch.

Snapshot the safe state at a branch point and run alternatives from there:

const branches = [
  "Yes, apply the change.",
  "No, cancel that request.",
  "Use a different workspace instead."
];

for (const nextUserMessage of branches) {
  const result = await runFromSnapshot({ snapshot: quoteState, nextUserMessage });
  await scoreConversation(result);
}

Forking improves coverage while controlling token cost. It is especially useful for consent, payments, account access, and any agent that can trigger an external write.

Put the suite in the delivery path

Run conversation regression tests whenever any behavior-bearing input changes:

system prompt or workflow graph;
model, model parameters, or provider route;
tool schema, permission policy, or API version;
retrieval corpus, ranking rule, or memory summarizer;
frontend action that changes what users can approve.

Tag fixtures by risk. A low-risk content-answering journey may warn on a judge regression. A high-risk write journey should block a release when a deterministic invariant fails.

Keep a short review packet with the baseline trace, candidate trace, score changes, and first failure. This is far better than burying a 30-turn transcript in CI logs.

Turn incidents into durable tests

Every production failure should become one of three things: a fixed invariant, a replay fixture, or a reason not to add a test. The last category is legitimate when the incident was infrastructure-only and already covered elsewhere—but make that decision explicit.

Use this loop:

Redact and reconstruct the customer journey.
Capture the smallest sequence that reproduces the failure.
Add the missing invariant or rubric criterion.
Verify the fix against the old fixture and a nearby branch.
Keep the fixture permanently unless the product rule changes.

Over time, the suite becomes an operational memory of how your agent has actually failed. That is more valuable than a generic leaderboard score because it reflects the work your users ask it to do.

A practical rollout checklist

Before calling conversation regression testing complete, confirm:

[ ] You have 10–20 risk-ranked journeys from real work.
[ ] Fixtures use redacted, stable data and safe tool doubles.
[ ] Each high-risk journey has at least one deterministic invariant.
[ ] Traces identify prompt, model, tool, and state revisions.
[ ] Failures report the first failing turn and causal evidence.
[ ] Branches cover approval, cancellation, correction, and timeout paths.
[ ] CI runs a bounded suite; deeper sampling happens nightly.
[ ] Incidents feed new fixtures into the suite.

The goal is not to prove an agent will never fail. It is to make familiar failures difficult to reintroduce—and to detect a new class of failure before it reaches a customer.

FAQ

What is conversation regression testing for AI agents?

It is a repeatable test of a whole agent interaction, rather than one prompt and one reply. The test replays a realistic sequence of user turns and tool responses, then checks cross-turn rules such as context retention, permission boundaries, and task completion.

How is it different from an AI agent evaluation harness?

An evaluation harness can cover many workflow-level checks. Conversation regression testing is a focused layer within that practice: it preserves known multi-turn journeys and compares behavior after a change. Its central outputs are cross-turn invariants, replay evidence, and first-failure attribution.

Should conversation tests compare exact model responses?

Usually no. Exact prose makes tests fragile. Assert deterministic product rules first, then use narrow rubrics for meaning. Exact matches are appropriate for structured output, tool arguments, required disclaimers, and other stable contracts.

How many conversation fixtures should a small team maintain?

Start with 10–20. Prioritize journeys with writes, money, access, regulated information, high volume, or prior incidents. Add a fixture whenever a production failure reveals a behavior you do not want to see twice.

Can I run multi-turn agent tests in CI without spending too much?

Yes. Mock external tools, run deterministic checks on every pull request, keep the PR suite small, and reserve repeated simulations for nightly or release-candidate runs. Fork from saved state instead of regenerating shared conversation prefixes.

What should make a conversation test block a deployment?

Block on deterministic failures involving permissions, tenant boundaries, unintended writes, unsafe tool calls, invalid structured output, or required policy behavior. Treat subjective quality-score changes as warnings unless they cross a deliberate, reviewed threshold.

AI Agent Idempotency: Prevent Duplicate Charges, Emails, and Records

Jack M — Wed, 16 Sep 2026 09:34:47 +0000

A timeout is not a failed action. It is an unknown action.

That distinction matters the first time an agent calls a payment API, sends a customer email, or creates a CRM record—and loses the response. If the agent retries blindly, it may double-charge a card or send the same message twice. If it does nothing, the user’s request may never finish. A prompt telling the model to “avoid duplicates” cannot settle that ambiguity.

The fix is idempotency: make each logical write happen once, even when workers crash, queues redeliver, SDKs retry, or the model asks to try again. This guide shows a practical design for putting that guarantee in the tool layer, where it can be tested and enforced.

Why agents make ordinary retry bugs worse

Every distributed system has ambiguous failures. A request may reach the target, commit its side effect, and then lose its response. Traditional backends solve this with idempotency keys, unique constraints, durable jobs, and reconciliation.

Agents amplify the problem because they are built to keep trying. A model sees timeout, changes its plan, and asks the same tool to run again. Meanwhile, a queue may redeliver the job after a lease expires, and an HTTP client may retry underneath both of them. One user intent can become four writes.

Treat these as distinct outcomes:

Outcome	What the caller knows	Safe next step
Rejected before send	The target did not receive it	Correct input, then retry if appropriate
Explicit failure	The target responded with a permanent error	Stop or request correction
Explicit success	The target returned a durable receipt	Store and return the receipt
Unknown commit state	The request may have succeeded	Reconcile before retrying

The last row is the important one. A timeout after send must never be silently mapped to “failed.”

The design rule: one key per logical action

An idempotency key identifies the business action, not a network attempt.

For example, “send the approved invoice email for invoice inv_123 revision 4” is one action. It should keep the same key if a worker restarts five times. A later resend after a user edits the invoice is a new action and needs a new key.

Good keys are stable, scoped, and inspectable:

tenant:acme | run:run_8f2 | step:email_invoice | invoice:inv_123 | revision:4

Hash that canonical representation if it contains sensitive identifiers. Do not generate a new UUID on every retry; that turns a dedupe mechanism into a duplicate generator. Also do not use only the user prompt. “Email this invoice” can be a valid request more than once.

An action should usually include:

tenant and actor identity
workflow run and step name
target resource identity and version
normalized arguments or an argument hash
expiry policy and final result

Put a write firewall in front of every agent tool

The agent should request a business action, not manipulate retry behavior directly. Place a deterministic action runner between the model and every state-changing integration.

agent plan
   -> tool request (business intent)
   -> action runner (policy + operation ledger)
   -> external API / database
   -> receipt + reconciliation result
   -> agent

This boundary is useful even if your tools are ordinary functions. It gives the system one place to validate tenant access, lock the operation, pass provider keys, classify failures, and hide unsafe retry choices from the model.

Reads can often be retried. Writes need an operation record first.

Start with a small operation ledger

Postgres is enough for many teams. Create the operation before calling the external service; keep it until the action’s replay window is over.

create table agent_operations (
  id uuid primary key,
  tenant_id uuid not null,
  idempotency_key text not null,
  action_type text not null,
  args_hash text not null,
  status text not null check (status in (
    'pending', 'running', 'succeeded', 'unknown', 'failed', 'needs_review'
  )),
  provider_reference text,
  result_json jsonb,
  error_json jsonb,
  lease_expires_at timestamptz,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (tenant_id, idempotency_key)
);

The unique constraint is a hard concurrency boundary. Two workers can receive the same job, but only one can own the logical action. Store an args_hash too: if the same key arrives with different arguments, fail closed. Reusing a key with changed data is almost always a caller bug.

Use status as evidence, not a guess

pending means no worker has started the side effect. running means a worker holds a short lease. succeeded stores the exact receipt to return on a duplicate call. unknown means the downstream result is ambiguous, so a reconciliation job—not an agent—must decide what happened.

Avoid marking an operation failed simply because the HTTP request timed out. That destroys the evidence needed to prevent a duplicate.

A TypeScript action wrapper

Here is a simplified pattern. In production, the claim query should use a transaction and a lease so a crashed worker can be recovered safely.

type ActionResult = { receiptId: string; status: "sent" | "already_sent" };

async function sendInvoiceEmail(input: {
  tenantId: string;
  runId: string;
  invoiceId: string;
  revision: number;
  to: string;
}): Promise<ActionResult> {
  const key = `invoice-email:${input.tenantId}:${input.invoiceId}:${input.revision}`;
  const argsHash = sha256(JSON.stringify({ to: input.to, revision: input.revision }));

  const operation = await operations.claim({
    tenantId: input.tenantId,
    key,
    actionType: "invoice_email",
    argsHash,
    leaseSeconds: 60,
  });

  if (operation.status === "succeeded") return operation.resultJson;
  if (operation.status === "unknown") return reconcileInvoiceEmail(operation, input);
  if (operation.status !== "running") throw new Error("Action is not safe to execute");

  try {
    const response = await emailProvider.send({
      to: input.to,
      template: "invoice",
      metadata: { operationId: operation.id },
      idempotencyKey: key,
    });

    return await operations.succeed(operation.id, {
      receiptId: response.messageId,
      status: "sent",
    });
  } catch (error) {
    if (isAmbiguousTransportError(error)) {
      await operations.markUnknown(operation.id, serialize(error));
      return reconcileInvoiceEmail(operation, input);
    }
    await operations.fail(operation.id, serialize(error));
    throw error;
  }
}

Notice what is missing: no model instruction decides whether an unknown write gets replayed. The wrapper returns a receipt, a verified “already completed” result, or an explicit escalation.

Reconcile when the provider lacks idempotency keys

Many third-party APIs do not support idempotency keys—or claim to accept them without making them searchable. Your internal ledger still helps, but it cannot prove the external effect occurred.

Make the action observable at the target. Depending on the system, reconciliation can query:

a provider object by a stored request or message ID
a database row with a unique operation ID
a webhook event carrying correlation metadata
a target record by a natural business key and version
an outbox entry that is atomically written with your local state

For an email, attach operationId as provider metadata and persist the provider message ID. For a CRM create, send an external ID derived from the operation key. For an internal database mutation, use a unique operation_id column or an INSERT ... ON CONFLICT pattern.

If reconciliation cannot prove success or failure, keep the operation in needs_review. This is safer than inventing an answer. The agent can tell the user that the action is pending verification rather than claiming it completed.

Separate retries by failure class

“Retry three times” is too blunt. A safe policy distinguishes failure modes.

Failure class	Example	Retry policy
Safe read	Search endpoint returns 503	Exponential backoff with a limit
Validation error	Missing required field	Do not retry; return correction needed
Rate limit	HTTP 429	Wait for the provider signal, then retry same operation
Ambiguous write	Socket closes after request body	Mark unknown and reconcile
Known transient write failure	Provider confirms no commit	Retry same operation key
Auth or policy denial	Scope removed	Stop and escalate

Retry budgets should also be outside the model. Put limits on attempts, elapsed time, spend, and allowed action types. This makes a bad downstream day finite instead of turning it into an overnight retry loop.

Test the ugly paths before customers find them

Unit tests that expect a 200 response are not enough. Build a small fault-injection suite around every write tool.

Test at least these cases:

Two workers claim the same operation at once.
The provider succeeds but the response is dropped.
The process dies after the provider call and before succeeded is stored.
A queue redelivers a completed job.
The same key arrives with different arguments.
Reconciliation finds a target effect after a timeout.
Reconciliation finds no effect and allows a same-key retry.
The provider returns a permanent validation or authorization error.

An effective assertion is simple: after any number of retries, the target contains exactly one effect for the operation key. Also test the user-visible result: a duplicate request should return the original receipt, not an opaque error.

Make idempotency visible in operations

Track these counters by tenant, tool, and provider:

duplicate calls served from the ledger
operations entering unknown
reconciliation success and failure rate
lease expirations and takeovers
key/argument mismatches
age of operations waiting for review

These are more useful than raw timeout counts. A rise in unknown commits may reveal a provider regression, an overly short client timeout, or a worker shutdown problem. A rise in duplicate calls can mean your queue is redelivering as designed—or that an upstream client is misbehaving.

Add an audit event each time an operation moves state. Include the actor, agent run, tool version, key hash, arguments hash, provider reference, and decision made by reconciliation. Do not store sensitive prompt or customer data just for convenience.

Roll this out without freezing your product

Start with the highest-impact tools: payments, email sends, record creation, access changes, and anything that triggers work outside your system. Inventory every write-side tool and give it an action type, stable key recipe, reconciliation strategy, and owner.

Then ship in stages:

Observe: log proposed keys and duplicate attempts without changing behavior.
Protect internal writes: add unique operation IDs and return stored receipts.
Protect provider writes: pass stable provider idempotency keys and record references.
Reconcile ambiguity: route timeout-after-send cases through target checks.
Enforce: reject direct write tools that bypass the action runner.

This is a better investment than ever more prompt rules. Prompts can help an agent choose a valid action; they cannot offer exactly-once delivery across a network.

FAQ

What is AI agent idempotency?

AI agent idempotency means repeating the same logical agent action produces one durable effect, not multiple ones. It protects state-changing tools from retries, worker crashes, duplicate queue messages, and repeated model tool calls.

Should every agent tool use an idempotency key?

Every state-changing tool should have an idempotency strategy. Read-only tools can normally use conventional retry policies. For writes, use provider keys, internal unique constraints, or an operation ledger plus reconciliation.

Can a system prompt prevent duplicate agent actions?

No. A prompt influences model behavior but cannot coordinate concurrent workers, recover a lost response, or prove whether a remote API committed an action. Enforce deduplication in the action runner and target system.

What should happen after a timeout on a write tool?

Treat it as an unknown commit state. Record the ambiguity, query the system of record using a correlation ID or business key, and retry only if you can establish that no effect occurred.

How long should idempotency records be retained?

Keep them at least as long as every possible replay window: queue retention, client retries, scheduled job retries, and provider webhook delays. High-risk actions such as payments often need longer retention and durable audit evidence.

Is idempotency the same as exactly-once delivery?

No. Most infrastructure offers at-least-once delivery. Idempotency makes repeated delivery safe by ensuring the receiver commits a logical action once and returns the original result to later attempts.

Make unknown explicit

Reliable agent systems do not pretend a timeout means failure. They preserve the operation, reconcile the target, and only then decide whether a retry is safe. Give every write tool a stable identity and a durable receipt, and an agent can be persistent without becoming destructive.