Tool Execution Needs Idempotency and Audit Trails

Before an AI tool performs a consequential action, commit the worker’s claim, its attempt ID, and the required audit event in a single transaction. After a crash, recovery can review these records before any further send attempts.

Human Approval as a System Boundary defines when permission is granted to dispatch a fixed support reply. Retries Are Not a Recovery Strategy explains why a missing response does not indicate whether the dispatch occurred. This article implements the records and transactions that support these rules.

The example uses EF Core with a file-backed SQLite database. It tracks one admitted reply through competing workers, a lost provider response, and reconciliation. The provider is a test double. No real email API or provider idempotency guarantee is involved.

The companion example on GitHub provides the complete implementation and automated failure checks. No model or email credentials are required.

Give each dispatch its own intent

In this example, the operation ID is an admitted DispatchIntentId. In a full workflow, the application allocates this ID for an explicit send request and binds it to a single authorization decision. The sample begins after that decision. The application must reuse the intent for repeated approval actions or tool calls.

The ticket and reply revision identify the payload but do not uniquely identify a send. A user can intentionally send the same stored reminder twice. These are two separate authorized intents, each with its own operation ID and provider key, even if their command fingerprints are identical.

Identity	What it identifies
Operation ID, the admitted DispatchIntentId	One intended send, stable across model turns and recovery
Reply revision and command fingerprint	The fixed recipient and body, plus the handler contract marker
Attempt ID	One worker’s submission attempt under that intent
Provider account, endpoint, and idempotency key	The destination scope in which replay may be safe

This fixture does not support attachments. ReplyCommand includes only the recipient and body, and Fingerprint hashes their serialized values along with the fixed reply-v1 handler contract marker. Add attachment versions and any other execution-relevant fields to the stored command and fingerprint before extending this tool.

Changing a command under an existing intent creates a conflict. If two admission requests try to insert the same intent at the same time, AdmitAsync reloads the winning record and checks its requester, revision, and fingerprint. Matching requests resolve to the same stored intent. A conflicting request still fails. The sample makes at most three storage attempts when handling SQLite contention, then propagates the error.

The stored reply remains available for any explicitly authorized future submission. Recovery uses the persisted provider key for lookup and never regenerates the reply. Store the payload in protected, versioned storage. A fingerprint cannot reconstruct a missing reply. Trace IDs and model tool-call IDs serve as correlation data.

Commit the claim, attempt, and audit event together

After admission, the operation starts in the Ready state. Its current state, attempt records, and audit events are stored in a shared database. The EF model assigns the operation a composite key and an application-managed concurrency token:

model.Entity<DispatchOperation>().HasKey(o => new { o.TenantId, o.OperationId });
model.Entity<DispatchOperation>().Property(o => o.Version).IsConcurrencyToken();
model.Entity<DispatchAudit>().HasKey(e => e.EventId);
model.Entity<DispatchAudit>().HasIndex(e => new { e.TenantId, e.OperationId, e.Sequence }).IsUnique();
model.Entity<DispatchAudit>().HasOne<DispatchAttempt>().WithMany()
    .HasForeignKey(e => new { e.TenantId, e.OperationId, e.AttemptId })
    .HasPrincipalKey(a => new { a.TenantId, a.OperationId, a.AttemptId });

Attempts also use a tenant-scoped primary key and a foreign key to their operation. The audit foreign key includes the tenant, operation, and attempt, supported by an alternate key on the attempt. The database rejects any event that names one operation but references an attempt from another operation. The Version field advances with each local audit transition and supplies the event sequence in this example. A production schema might keep these counters separate.

TryPrepareAttemptAsync runs this transaction. Authorized filters use the trusted tenant, operation ID, and requester. Authentication and policy decisions occur before this context reaches the store:

public async Task<PreparedAttempt?> TryPrepareAttemptAsync(
    DispatchContext context, Guid attemptId, CancellationToken token)
{
    await using var db = Open();
    var operation = await Authorized(db, context).AsNoTracking().SingleAsync(token);
    if (operation.State != DispatchState.Ready) return null;
    var command = new ReplyCommand(operation.Recipient, operation.Body);
    if (operation.Fingerprint != Fingerprint(command)
        || string.IsNullOrEmpty(operation.AdmissionReference))
        throw new InvalidOperationException("Stored command or admission is invalid.");

    await using var transaction = await db.Database.BeginTransactionAsync(token);
    int claimed = await Authorized(db, context)
        .Where(o => o.State == DispatchState.Ready && o.Version == operation.Version)
        .ExecuteUpdateAsync(setters => setters
            .SetProperty(o => o.State, DispatchState.InProgress)
            .SetProperty(o => o.ActiveAttemptId, attemptId)
            .SetProperty(o => o.Version, o => o.Version + 1), token);
    if (claimed != 1) return null;

    DateTime now = DateTime.UtcNow;
    db.Attempts.Add(new DispatchAttempt
    {
        TenantId = context.TenantId, OperationId = context.OperationId,
        AttemptId = attemptId, ExecutorId = "dispatch-worker",
        ProviderAccount = operation.ProviderAccount, ProviderEndpoint = operation.ProviderEndpoint,
        ProviderKey = operation.ProviderKey,
        PreparedUtc = now, RecoverAfterUtc = now.AddSeconds(30)
    });
    var audit = NewAudit(context, attemptId, operation.Version + 1,
        "dispatch-worker", "AttemptPrepared", null);
    db.Audit.Add(audit);
    await db.SaveChangesAsync(token);
    await transaction.CommitAsync(token);
    return new PreparedAttempt(attemptId, command, operation.ProviderKey,
        operation.ProviderAccount, operation.ProviderEndpoint);
}

The conditional update checks that the operation is Ready and matches the version that was read. Two workers might read the same version, but only one will be able to update it. If the update affects no rows, the worker reloads the existing status instead of sending. Disposing of the uncommitted transaction rolls it back.

ExecuteUpdateAsync runs immediately and bypasses change tracking. It does not wait for SaveChangesAsync or automatically include concurrency-token checks. This is why the version predicate and explicit transaction are used. The attempt and required audit event are included in the same transaction before commit. If either insert fails, the claim is rolled back as well.

NewAudit generates a unique event ID and stores the tenant, operation ID, attempt ID, sequence, actor, recording time, and optional evidence reference. AttemptPrepared records who took responsibility. It does not prove provider submission or acceptance.

Only a return after a confirmed commit allows the executor to submit. If a commit acknowledgement is lost, check the preallocated attempt ID before determining what persisted. If the lookup is inconclusive, do not proceed with dispatch. Database retry policies must handle uncertain commits, and provider calls must always occur outside any transaction-retry delegate.

SQLite serializes writers. Contention may cause waits, affect no rows, or raise a busy or locked exception when the configured timeout expires. The safety guarantee is a single committed claim, not an exception-free losing caller. Handle transient storage failures with bounded retries or by reloading the status later, always verifying any uncertain commit before dispatch. This fixture demonstrates a local transactional boundary. Apply the same contention checks to the production database and isolation settings.

Decide when permission becomes a dispatch commitment

The complete workflow applies the CommitForDispatch rule described in the approval article. That boundary must coordinate the specific revision, approval expiry, relevant resource versions, and policy before admitting an operation.

This example assumes the decision has already been made and stores a synthetic admission reference. It does not implement the approval service or verify a live authorization decision. The nonempty AdmissionReference check only rejects missing fixture data. The worker claim occurs after admission.

In this contract, approval expiry limits admission. It does not cancel an already admitted send while a worker is waiting. If a user withdraws after admission, it cannot be reported as a successful cancellation. The UI must indicate that dispatch has been committed and may already have reached the provider.

If the product instead promises cancellation or freshness until the provider handoff, add a coordinated gate between cancellation and dispatch commitment. Checking a flag before claiming, or again immediately before the HTTP call, still leaves a pause-and-revoke race condition. The persistence fixture below does not implement this stronger contract or a live authorization service.

Hard denials and service authority must still be enforced at the workflow’s defined gates, especially before any recovery submission. A denial can stop future attempts but cannot guarantee that an earlier uncertain call had no effect.

Keep the model-facing tool small

Tools and Dependency Injection in Microsoft Agent Framework covers registration and service wiring. Keep the tool bound to a single trusted dispatch context:

public sealed class SupportReplyTools(
    ReplyExecutor executor,
    DispatchContext context)
{
    [Description("Dispatches the stored support reply authorized for this operation.")]
    public Task<DispatchResult> SendApprovedReplyAsync(
        CancellationToken cancellationToken = default) =>
        executor.ExecuteAsync(context, cancellationToken);
}

The tool instance is scoped to the operation. The context includes the tenant, requester, and existing operation ID. The model cannot replace the recipient or generate another operation ID through this tool. The tool-validation tip explains how to validate new proposals.

The executor submits only the attempt returned after the database commit:

public sealed class ReplyExecutor(
    IReplyOperations operations,
    IReplyProvider provider)
{
    public async Task<DispatchResult> ExecuteAsync(
        DispatchContext context,
        CancellationToken cancellationToken)
    {
        Guid attemptId = Guid.NewGuid();
        PreparedAttempt? attempt = await operations.TryPrepareAttemptAsync(
            context, attemptId, cancellationToken);

        if (attempt is null)
        {
            return await operations.ReadStatusAsync(context, cancellationToken);
        }

        ProviderDestination.EnsureMatches(
            provider, attempt.ProviderAccount, attempt.ProviderEndpoint);

        SubmissionObservation observation;
        try
        {
            observation = await provider.SubmitAsync(
                attempt.Command, attempt.ProviderKey, cancellationToken);
        }
        catch (SubmissionOutcomeUnknownException)
        {
            observation = new(SubmissionOutcome.Unknown, null);
        }

        using var evidenceBudget = new CancellationTokenSource(
            TimeSpan.FromSeconds(2));

        return await operations.RecordObservationAsync(
            context, attemptId, observation, evidenceBudget.Token);
    }
}

The adapter exposes read-only AccountId and Endpoint values. These must describe its authenticated client, not labels loaded independently from mutable configuration. The application helper ProviderDestination.EnsureMatches compares both values with the persisted destination using exact, case-sensitive equality. A mismatch triggers an exception before submission. Recovery uses the same helper before lookup, so a configuration change to another account or endpoint will block both paths. Restore the original adapter binding or investigate, rather than substituting a fallback destination. The fixture uses a single synthetic account and endpoint.

IReplyOperations, IReplyProvider, and these result types are application code. The provider adapter classifies accepted submissions, definite non-acceptance, and unknown outcomes. It must treat cancellation after a possible submission as unknown. Catching every exception and treating it as a rejection would remove this distinction.

The separate evidence token allows the executor to try to save an observation after the caller cancels. The two-second window is illustrative and depends on the storage client’s timeout and cancellation behavior. It is not a strict process deadline. If the write fails, propagate the failure and recover the prepared attempt later.

A repeated tool call cannot create another attempt. Only the Ready state allows a claim. All other states return the recorded status, including a successful result where the caller never received the tool response.

These tests show that the executor does not resubmit a claimed operation. The simulator counts every submission and overwrites the response stored under that key. Safe provider-side replay is governed by a separate destination contract.

Separate attempt rejection from operation rejection

A rejected submission attempt does not always end the business operation. The adapter needs evidence that the provider did not accept the attempt before classifying a rejection as retryable.

Provider observation	Attempt outcome	Operation state
Confirmed acceptance	Accepted	Succeeded
Authoritative permanent rejection without acceptance	Permanent rejection	Rejected, terminal
Authoritative transient rejection without acceptance	Retryable rejection	RetryPending, still active
Submission may have reached the provider	Unknown	OutcomeUnknown

InProgress means a worker owns a prepared attempt and its final outcome has not been recorded. A crash may leave it in this state. Succeeded indicates provider acceptance of the fixed reply, not delivery or recipient reading.

The standard tool never transitions RetryPending back to Ready. An explicit recovery policy may authorize a new attempt under the same operation, depending on current permission, retry budgets, and the destination’s contract. A terminal Rejected operation remains terminal. If the user wants to send a corrected command, that requires a new admitted intent.

Save the result and evidence in the same transaction

RecordObservationAsync also starts a transaction. It loads the authorized operation as a tracked entity and loads the matching tenant-scoped attempt. The write portion is:

bool ownsState = operation.ActiveAttemptId == attemptId
    && operation.State is DispatchState.InProgress or DispatchState.OutcomeUnknown;
if (ownsState)
{
    operation.State = observation.Outcome switch
    {
        SubmissionOutcome.Accepted => DispatchState.Succeeded,
        SubmissionOutcome.PermanentRejection => DispatchState.Rejected,
        SubmissionOutcome.RetryableRejection => DispatchState.RetryPending,
        _ => DispatchState.OutcomeUnknown
    };
    operation.ProviderReference = observation.ProviderReference;
    attempt.Observation = observation.Outcome.ToString();
}
string eventType = ObservationEventType(observation.Outcome, ownsState, reconciled);
db.Audit.Add(NewAudit(context, attemptId, ++operation.Version,
    reconciled ? "recovery-worker" : "dispatch-worker",
    eventType, observation.EvidenceReference));
await db.SaveChangesAsync(token);
await transaction.CommitAsync(token);
return new DispatchResult(operation.OperationId, operation.State);

EF includes the original Version in the tracked operation update. A concurrent change makes SaveChangesAsync fail rather than silently overwrite it. The surrounding transaction rolls back both that update and the audit insert. Reload and reconcile the observation, rather than retrying a stale entity unchanged.

The active-attempt check prevents an old worker from replacing another attempt’s result. Recovery may already have resolved this very attempt. The state check preserves that result too. A late response still gets an audit event, such as LateUnknownObservation, without downgrading Succeeded to OutcomeUnknown.

ObservationEventType keeps event naming separate from the state update:

private static string ObservationEventType(
    SubmissionOutcome outcome, bool ownsState, bool reconciled)
{
    if (!ownsState) return $"Late{outcome}Observation";

    return outcome switch
    {
        SubmissionOutcome.Accepted when reconciled => "ProviderAcceptanceReconciled",
        SubmissionOutcome.Accepted => "ProviderAcceptanceConfirmed",
        SubmissionOutcome.Unknown => "SubmissionOutcomeUnknown",
        _ => outcome.ToString()
    };
}

Accepted observations require a provider reference and an evidence reference before this write executes. The adapter must obtain this evidence from a validated response, authenticated lookup, or verified provider event. A model’s claim that a send succeeded is not enough.

Recover by lookup before attempting another send

The recovery path can be smaller than the initial executor. This worker reads due InProgress or OutcomeUnknown attempts and performs a lookup using their persisted provider key:

public sealed class ReplyRecovery(SqliteReplyOperations operations, IReplyProvider provider)
{
    public async Task ReconcileOnceAsync(
        DispatchContext context, DateTime nowUtc, CancellationToken token)
    {
        DispatchAttempt? attempt = await operations.ReadRecoverableAsync(context, nowUtc, token);
        if (attempt is null) return;
        ProviderDestination.EnsureMatches(
            provider, attempt.ProviderAccount, attempt.ProviderEndpoint);

        if (!await operations.TryBeginReconciliationAsync(context, attempt.AttemptId, nowUtc, token))
            return;
        SubmissionObservation evidence = await provider.LookupAsync(attempt.ProviderKey, token);
        await operations.RecordAsync(context, attempt.AttemptId, evidence, true, token);
    }
}

TryBeginReconciliationAsync verifies the current operation state, active attempt, and due time within a transaction. It advances RecoverAfterUtc by thirty seconds and commits ReconciliationStarted with the version update before granting permission to look up the provider key. If another worker has resolved the operation or reserved the lookup slot, it returns false without adding an event.

An unknown result leaves the next check scheduled. A failed lookup or a crash after reservation also leaves that time unchanged, so the next scheduler pass cannot immediately repeat the work. RecordAsync uses the conditional result write shown above. The worker does not resubmit. An inconclusive lookup records OutcomeUnknown.

The thirty-second interval limits repeated lookups. It does not prevent a paused recovery worker from resuming after its slot expires. This sample does not implement a maximum lookup escalation policy.

Overlapping lookups can complete out of order. The provider adapter must classify responses based on the destination’s consistency guarantees and treat stale or inconclusive evidence as unknown. A lookup reservation alone cannot make an observation authoritative enough to move to a terminal state.

The scheduled recovery time is a prompt to investigate, not proof the original worker stopped. Even a worker whose local writes are fenced may resume and reach the provider later. Only the destination’s contract can make duplicate external submissions safe.

If automated replay is required, keep the original command, key, provider account, and destination. Persist a conservative replay deadline based on the provider’s documented guarantee, and never extend it through retries. Check the contract for the actual endpoint rather than assuming an idempotency key guarantees safe replay.

A lookup is useful only if the application stored something queryable before the response was lost. If the lookup requires a provider reference that was only present in the lost response, recovery may not be possible. A delayed “not found” result is also inconclusive. When safe replay is unavailable or its window has expired, stop automatic submission and assign manual review with an owner and a deadline.

An outbox can durably deliver a work item, but it cannot include the external provider in the local database transaction. The provider may still receive a request more than once, so duplicate prevention must be enforced at the destination boundary. The broader recovery tradeoffs are discussed in Retries Are Not a Recovery Strategy.

Inspect the stored audit history

The fixture produces this sequence when the provider accepts a reply but drops the response. Operation and attempt IDs are abbreviated as O1 and A1, and the provider key as K1. The simulator derives a distinct stable receipt from each accepted operation’s key. Each row also stores its own event ID and UTC recording time.

Sequence	Operation / attempt	Actor	Stored event	Evidence and meaning
1	O1 / A1	dispatch-worker	AttemptPrepared	Local claim and attempt committed, no proof of submission
2	O1 / A1	dispatch-worker	SubmissionOutcomeUnknown	Worker observed a missing result, acceptance remains unresolved
3	O1 / A1	recovery-worker	ReconciliationStarted	Lookup slot and next check committed, no proof of completion
4	O1 / A1	recovery-worker	ProviderAcceptanceReconciled	provider-lookup/receipt-K1, authoritative acceptance evidence from the test provider

The operation now has State = Succeeded, ActiveAttemptId = A1, ProviderReference = receipt-K1, and Version = 4. It retains the attempt ID for attribution. The attempt’s latest observation is Accepted. Earlier uncertainty is preserved in the audit rows. Repeated tool calls between rows 2 and 4 return OutcomeUnknown and do not create a new send attempt.

An investigator can see both what the worker observed and what the destination later confirmed. Sequence 2 is local evidence of uncertainty. Sequence 4 resolves acceptance for the fixed operation. Neither proves that the recipient received the message. Delivery and bounce events require their own lifecycle.

Keep required execution evidence in the operation store with durable export if investigators use a separate system. Sampled diagnostic traces might omit the request under investigation. A trace ID is useful for correlation but cannot replace the ledger.

Event identity does not deduplicate a logical observation. If a result transaction commits but its acknowledgement is lost, reprocessing that observation may append another event with a new ID. The fixture preserves state and records that replay, it does not suppress it. If observation deduplication is needed, persist a stable observation ID with a tenant-scoped uniqueness constraint covering the operation and observation. A unique sequence only protects ordering.

Deduplicate exported events by event ID and retain the local sequence. Record provider event time separately from local receipt time when integrating asynchronous events. Append corrections, restrict investigative access, and monitor export failures. The fixture’s tables are not tamper-resistant. Stronger preservation requires independently controlled retention or immutable storage.

Keep payloads and authorization decisions behind protected references. Exclude credentials and arbitrary exception text from audit events. Opaque identifiers and fingerprints can reveal correlations, so define their retention as well. Privacy Boundaries for AI Telemetry covers the diagnostic-capture policy.

Test the storage boundary without a model

The verification fixture uses .NET 10, Microsoft.Extensions.AI 10.9.0, and EF Core SQLite 10.0.0. It invokes the real AIFunction, opens independent database connections, and reads the stored rows after each scenario.

Follow the companion repository’s README to run the console demo and the xUnit tests. The C# excerpts in this article are from that implementation.

Verified area	What the tests demonstrate
Admission identity	Concurrent matching requests resolve to one intent, conflicting commands fail, storage retries are bounded
Atomic execution claims	Competing workers cannot commit two attempts for the same operation
Transactional audit integrity	Required audit failures roll back state changes, cross-operation references are rejected
Unknown outcomes	Lost responses, cancellation, and failed result writes retain recoverable state without triggering another submission
Recovery coordination	Competing workers reserve one lookup slot, unknown or failed lookups retain a durable next-check time
State preservation	Late or repeated observations cannot overwrite an already resolved operation
Execution boundaries	Distinct intents remain independent, provider destination mismatches and foreign tenant or requester contexts block execution

The companion example includes the individual test cases, such as process-exit recovery, SQLite contention, and provider receipt verification.

The tests exercise this file-backed implementation. They do not demonstrate a real provider’s idempotency, power-loss durability, or correctness under different database isolation rules. The child-process check exits after a committed claim and does not emulate every storage failure. Authorizing another send attempt, an outbox relay, live policy revocation, and provider-key expiry are beyond the scope of this fixture.

Before production, test for a sustained result-store outage, uncertain database commits, delayed old workers, and replay-window expiry against your chosen destination and database. Testing Microsoft Agent Framework Applications explains why these checks belong below the model boundary.

When to use this approach

Use durable operation and attempt records for consequential tools that may run again after queue redelivery, a timeout, or a restart. Start with one operation, commit its claim and required evidence together, and demonstrate recovery after losing the provider response.

Define what success proves and who investigates uncertainty before enabling automatic retries. The assistant should report the stored execution state, including unresolved outcomes.

When not to use this approach

Pure computation over public data usually does not need an execution ledger. Protected reads may need access auditing. A local write may require only one idempotent business transaction and an audit record, without a separate recovery worker.

For an external action without safe replay or reliable lookup, keep recovery manual. A local ledger cannot enforce exactly-once execution on an external service that does not provide such a guarantee.

Related reading

• Human Approval as a System Boundary
• Tools and Dependency Injection in Microsoft Agent Framework
• Testing Microsoft Agent Framework Applications

Sources

• Microsoft Learn: ExecuteUpdate, explicit concurrency checks, and transactions
• Microsoft Learn: Transactions in EF Core
• Microsoft Learn: Composite foreign keys and principal keys in EF Core
• Microsoft Learn: Connection resiliency and uncertain transaction commits
• Microsoft Learn: Transactions and concurrency in Microsoft.Data.Sqlite
• Microsoft Learn: SQLite busy and locked errors, retries, and timeouts
• Microsoft Learn: Access application context in AI function delegates
• Amazon Builders’ Library: Making retries safe with idempotent APIs
• Azure Architecture Center: Transactional Outbox pattern
• IETF: RFC 5321, reliable delivery and email replies
• OpenTelemetry: Sampling
• OWASP: Logging Cheat Sheet