Your AI Agent Passed Preflight. Which Receipt Proves the Change Landed?

A question for engineers building AI agents that actually change things.

Imagine this fictional workflow:

1. An agent is approved to update an ERP record.
2. The role, task scope, and review binding pass preflight.
3. The tool definition changes before dispatch.
4. The gateway forwards the operation, but the client times out.
5. Another agent takes over and considers retrying.

What evidence would you need before trusting the result?

I'm exploring four possible ingredients for a minimal evidence chain:

• The exact approved task/action digest
• The authenticated executor and effective tool version
• The logical operation ID and retry identity
• A fresh, authoritative readback of the actual result

But every extra receipt has a cost: storage, implementation, maintenance, and operational complexity.

I maintain Agent Role Contracts (ARC), an open-source project that checks supplied role, scope, review, and task/action declarations offline. Runtime enforcement and independent verification of external effects are separate responsibilities.

We're investigating these boundaries through a research pilot, with the real-world validation stages explicitly tracked.

My question: What's the smallest evidence chain you would trust in production?

Have you solved this with database logs, idempotency keys, event sourcing, workflow engines, or something simpler?

A real example would be especially useful:

system | side effect | failure | trusted evidence | retry rule | cost

I'm collecting concrete engineering perspectives in this GitHub discussion as well.

Comments

[Tom Zafred]

@suirindo This scenario exposes two subtle traps that standard idempotency keys and generic entity readbacks fail to solve:

1. The Stale Tool Contract (Step 3): If the tool definition changes before dispatch, a standard UUID idempotency key will still execute, applying an unverified payload against a mutated schema.
2. The Concurrency Illusion (Step 5): When the client times out, the handover agent cannot simply inspect the target record (GET /record/42). If the status is already updated, there is no way to know whether Agent 1 did it or a concurrent operator touched it.

In regulated biobanking LIMS and ERP systems (where duplicate specimen thawing or relocation is catastrophic), we solve this with a tripartite invariant:

Dimension	Production Implementation
System	Regulated Laboratory Information & Aliquot Inventory Management (LIMS / ERP)
Side Effect	State transition of cryogenic specimen containers (relocating boxes between freezers or thawing vials)
Failure	Client HTTP timeout/network partition after gateway dispatch (status strictly UNKNOWN, never FAILED)
Trusted Evidence	Atomic in-transaction audit ledger entry. The business mutation and an immutable correlation record (correlation_id, caller_role, tool_sha256, applied_at) commit within the exact same physical database transaction
Retry Rule	Probe Before Dispatch: Handover agent is barred from re-dispatching. It queries the audit ledger by correlation_id. If FOUND, it returns the committed state; if NOT_FOUND, it evaluates the live rowversion before authorising a single idempotent re-submission
Cost	1 auxiliary write to an indexed audit table per transaction; 1 indexed read query before handover execution. Zero distributed coordination daemons (Temporal/Kafka)

Two architectural rules close the remaining loopholes:

• Tool-Bound Idempotency Keys: We derive the idempotency key deterministically as HMAC(task_digest | tool_definition_sha256 | entity_id). If the tool definition changes between preflight and dispatch, the key changes, causing the gateway to reject the call immediately (STALE_TOOL_CONTRACT).
• Correlation-Attested Readback: The handover agent queries the audit journal for the specific correlation_id, not just the entity state. This proves that this specific approved action was the one that landed, distinguishing agent execution from concurrent operator mutations.

[asagiman]

Tom, this is a valuable production example. Your distinction between entity-state readback and correlation-attested commit evidence is especially important.

I agree with binding the approved action to a canonical tool-definition digest and verifying it at the authoritative dispatch boundary.

Two details seem worth testing explicitly:

1. Stale contract rejection: Changing the HMAC-derived idempotency key alone would not reject a stale contract. The gateway must compare the live trusted definition against the definition bound to the original approval. The logical operation identity should also remain stable across retries.

2. Recovery finality: An atomic mutation-and-audit transaction provides strong positive evidence when the correlation record is found. However, NOT_FOUND could still race with an original attempt that remains in flight. A rowversion check alone may not resolve that ambiguity.

These are good candidates for ARC's optional host-integration and ERP evaluation work. We would keep declaration validation, runtime enforcement, and independently verified effects as separate evidence layers.

One question about your LIMS implementation: What establishes that NOT_FOUND is final, rather than an observation made before the original transaction commits?

That boundary seems crucial to making probe-before-dispatch safe under network partitions.