An AI agent calls a business tool. The tool changes real state. Then the runtime fails while recording the result.
What should the caller see?
If the answer is a generic tool_error, the agent may do exactly what many systems have trained it to do:
The call failed.
Try again.
But the call may not have failed.
The refund may already exist. The employee record may already be changed. The deployment may already be running. Only the evidence write failed.
At that point, a blind retry is not resilience. It is a second attempt to create a business effect whose first outcome has not been reconciled.
This is a narrow failure mode, but it exposes a much larger design problem: tool execution, business commitment, caller-visible results, and durable audit evidence are different events. A runtime that compresses all of them into success or failed cannot make a safe retry decision.
The observable boundary matters
A common tool path has this shape:
run the tool
-> receive the tool result
-> append the result to the session log
-> return to the caller
If the tool has already run but the outcome append fails, the caller may receive a generic tool error. The caller cannot distinguish that event from a tool that never ran.
It is tempting to represent the failure with a state such as:
committed_but_evidence_degraded
That name is only accurate when some trusted component actually knows that the downstream business transaction committed.
A gateway often does not know that. It may observe dispatch and tool return, but not the database transaction, payment settlement, queue acknowledgment, or external side effect inside the tool.
So the portable failure distinction is narrower:
The operation crossed the dispatch boundary, and durable outcome evidence failed afterward.
That is enough to suppress a blind retry. It is not enough for the gateway to claim that a business commit occurred.
One generic error can hide four different events
Consider a write operation such as:
POST /refunds
A caller-visible failure might mean any of the following:
| Failure class | What is known? | Default retry posture |
|---|---|---|
| Pre-dispatch rejection | The operation did not leave the runtime | Retry may be possible after the cause is fixed |
| Dispatch not established | The runtime could not establish that the request was sent | Depends on transport and idempotency guarantees |
| Tool returned a confirmed business failure | The business system rejected or rolled back the operation | Do not retry unless the failure is explicitly retryable |
| Post-dispatch evidence failure | The tool ran or returned, but durable outcome evidence was not recorded | Do not retry blindly; reconcile first |
Returning the same error for all four cases transfers an impossible decision to the caller.
The model sees only:
{
"error": "tool failed"
}
It does not know whether another call is necessary, useless, or dangerous.
This decision should not be improvised by the model from error prose. It should be made by deterministic runtime policy using typed failure information.
The dual-write problem is not specific to agents
The underlying shape is familiar:
write business state
write evidence state
Unless both writes participate in one atomic transaction, they can disagree.
In an agent system, the stores are often intentionally separate:
- the business effect belongs to a SaaS application, payment provider, CRM, deployment system, or another tool;
- the execution trace belongs to an agent runtime, gateway, audit database, or SIEM pipeline.
The runtime usually cannot wrap both systems in one transaction.
That means the design must handle at least two asymmetric failures:
evidence exists, business effect did not occur
and:
business effect may exist, durable outcome evidence does not
The second case is especially dangerous because an ordinary failure response invites another execution.
Record intent before the effect
A useful starting pattern is write-ahead intent:
1. Persist invocation intent.
2. Dispatch the business operation.
3. Persist the outcome.
4. Return a typed result.
The intent record should be durable before dispatch and should contain enough stable identity to support reconciliation. Depending on the system, that may include:
task_id
invocation_id
capability identity
trusted acting subject reference
canonical argument hash
business idempotency reference
dispatch timestamp
It should not contain secrets merely because they were present in the tool request. Sensitive arguments can be redacted, summarized, or represented by a canonical digest.
If the outcome append fails, the runtime still has evidence that this invocation crossed the pre-dispatch boundary. A reconciler now has a starting point.
Without the intent record, the system may have only a generic error and a changed business system.
A distinct error must change retry behavior
Adding a new error name is not enough. It must change what the caller is allowed to do.
For example:
{
"status": "post_dispatch_evidence_failed",
"retryable": false,
"reconciliation_required": true,
"invocation_id": "inv-7f3..."
}
This state does not claim that the business operation committed.
It claims only that:
- dispatch already occurred, or the tool already returned;
- the runtime could not durably preserve the expected outcome evidence;
- another business invocation must not be created automatically;
- the original invocation must be reconciled.
The reconciliation path may query:
- a business idempotency ledger;
- a downstream result endpoint;
- the current business object state;
- a payment or deployment reference;
- an asynchronous callback record;
- a human operator when no deterministic query exists.
Only after reconciliation establishes that no business effect occurred, or the business system guarantees safe replay under the same idempotency identity, should another transport attempt be considered.
Idempotency is not a retry button
It is tempting to solve this problem by labeling the tool idempotent.
ACC v1, for example, can carry this execution hint:
x-agent-capability:
version: 1
enabled: true
scope: refund.create
execution:
readonly: false
idempotent: true
The declaration means that the operation may be safely retried with the same arguments. It does not create the business guarantee by itself.
A real idempotency path still needs a stable business identity and enforcement in the system that owns the effect:
same business intent
-> same idempotency key
-> one atomic business result
-> later requests return the original result
Several mistakes can make an idempotent: true declaration meaningless:
- generating a new idempotency key for every retry;
- deduplicating only in volatile runtime memory;
- allowing the subject, tenant, or arguments to drift between attempts;
- storing the idempotency key separately from the business transaction;
- treating a successful HTTP response as the only proof of the result.
For operations whose idempotency is false or unknown, the conservative rule is simpler:
A post-dispatch evidence failure is not automatically retryable.
Keep four identities separate
Reliable reconciliation becomes easier when the implementation distinguishes:
| Identity | What it represents |
|---|---|
task_id |
The durable user or business goal |
invocation_id |
One governed invocation of one capability |
attempt_id |
One transport attempt for that invocation |
idempotency_key |
The business-owned identity used to suppress duplicate effects |
One task may contain multiple invocations. One invocation may need more than one transport attempt. Those attempts must not silently become new business intents.
A retry that changes the business idempotency key is not a retry in the safety sense. It is another request for another effect.
Responsibility belongs to several layers
No single gateway, contract, or business API can solve the whole failure mode.
| Layer | Responsibility |
|---|---|
| Capability declaration | Describe portable hints such as readonly, idempotency, timeout, risk, subject requirement, and audit sensitivity |
| Caller or agent runtime | Preserve task and invocation identity, consume typed failures, and prevent model-driven blind retries |
| Gateway or execution control | Record pre-dispatch intent, distinguish lifecycle stages, preserve redacted evidence, and surface evidence degradation |
| Tool adapter | Carry stable invocation and idempotency context without letting model output rewrite trusted governance data |
| Business system | Decide final authorization, own commit truth, enforce idempotency atomically, and expose result lookup where possible |
| Reconciler or operator | Determine the original outcome and complete or repair the evidence chain without creating a second effect |
This separation explains why a gateway should not claim more than it observes.
It also explains why the business system should not be asked to own the entire agent trace.
Each layer should provide the evidence it can authoritatively produce.
What ACC does and does not say here
The Agent Capability Contract is a portable declaration contract, not a transaction coordinator or audit database.
ACC v1 can declare:
- whether an operation is readonly;
- whether it claims idempotent retry semantics;
- timeout and rate-limit hints;
- risk and approval intent;
- whether a trusted acting subject is required;
- whether arguments or results require sensitive audit handling.
An ACC-compatible runtime should preserve result or failure information and relevant audit context.
ACC v1 does not standardize:
- a business idempotency ledger;
- downstream commit observation;
- workflow recovery or compensation;
- audit storage topology;
- a universal
committed_but_evidence_degradedstate; - a guarantee that a runtime-authored log is independent proof.
That boundary matters.
This failure class is useful implementation evidence. It is not, by itself, a reason to add a new ACC core field.
Before any portable semantic is standardized, independent runtimes would need to agree on an observable state, a safe fallback, and a conformance test that does not require access to one product's private storage model.
The test that matters
A useful fault-injection test is concrete:
1. Persist the invocation intent.
2. Let the tool create an observable business effect.
3. Force the outcome-evidence append to fail.
4. Verify that the caller receives a distinct, non-retryable failure.
5. Verify that no second business dispatch occurs automatically.
6. Reconcile the original invocation using its stable identity.
7. Repair or complete the evidence without creating another effect.
The test should prove more than “an error was logged.”
It should prove:
one business intent
-> at most one business effect
-> an explainable terminal or reconciliation state
That is the reliability property an agent system needs when tools can change money, permissions, inventory, customer records, infrastructure, or external communications.
A better question than “did the tool fail?”
For read-only tools, repeating a call may only add cost and latency.
For business writes, a retry policy must answer three separate questions:
Did dispatch occur?
Did the business effect occur?
Was durable evidence preserved?
Sometimes the runtime can answer all three. Sometimes it can answer only the first. The API should communicate that uncertainty rather than hide it behind a generic error.
The goal is not to make every distributed action perfectly atomic. The goal is to prevent uncertainty from becoming a duplicate real-world consequence.
How does your runtime represent this case?
The tool may have run, but the outcome evidence did not persist. Should the caller retry, reconcile, or stop?
Top comments (0)