Replayable AI Tool Calls Need Run Receipts

The missing artifact after a tool call

An AI workflow can say that it called a tool successfully, while leaving you with almost nothing useful when the next step breaks. A green log line is not the same thing as evidence. When a scheduled job runs at 3 AM, I want to know what the model asked for, which tool version answered, and what the workflow actually used.

My practical fix is a small run receipt: one structured record written after every tool call. It makes automation less mysterious and gives the next person a short path from failure to replay. This is especially handy when a job touches signup checks, disposable email, or other inputs that are difficult to reproduce later. A loose note like fake e mail com in a test case may be enough for a human, but it is a poor debugging contract.

What a run receipt contains

A receipt should be boring. Boring records are easy to diff, store, and search. Mine normally includes:

• a run ID and parent workflow ID
• the tool name and contract version
• a redacted input hash, rather than private input values
• start and finish timestamps
• status: ok, retryable, or failed
• the output location and a compact error code
• whether the result is safe to replay

That last field matters. A read-only lookup can often be replayed. Sending an email, charging a card, or creating an account cannot be repeated casually. Treat replayability as a property of the operation, not as a hopeful button in the dashboard.

For related thinking, an inspectable rollout record is a useful reminder that operational context should travel with the action. And if your workflow validates addresses, add explicit review windows for disposable email checks instead of hiding that decision in a boolean.

A small implementation

Here is a deliberately plain Python shape. The payload is already redacted before it reaches this function:

from dataclasses import asdict, dataclass
from datetime import datetime, timezone
import hashlib
import json

@dataclass
class Receipt:
    run_id: str
    tool: str
    contract: str
    input_hash: str
    status: str
    replayable: bool
    error_code: str | None = None

def make_receipt(run_id, tool, contract, redacted_input, status,
                 replayable, error_code=None):
    digest = hashlib.sha256(
        json.dumps(redacted_input, sort_keys=True).encode()
    ).hexdigest()[:16]
    receipt = Receipt(run_id, tool, contract, digest, status,
                      replayable, error_code)
    return {**asdict(receipt), "finished_at": datetime.now(timezone.utc).isoformat()}

The hash is for correlation, not security. Do not put tokens, full email addresses, or model prompts containing personal data into receipts just because the log store feels private. Keep the original input in a controlled, expiring store only when an actual replay needs it.

Replay rules that keep jobs safe

First, pin the contract version. A receipt from v1 should not silently be replayed against a v3 tool with different fields. Second, require an idempotency key for side effects. Third, make replay a new run with a link to the old receipt; editing history makes incidents very confusing.

I also separate replay from retry. A retry is the system continuing an operation after a transient error. A replay is a deliberate human or operator action using recorded context. They need different permissions and different audit language. This distinction sounds fussy at first, but it save time when a queue has several near-identical failures.

Checklist for the next workflow

Before shipping an AI-assisted scheduled job, ask:

1. Can I identify every tool call belonging to one run?
2. Can I tell a model decision from a tool result?
3. Are private inputs redacted before logging?
4. Is the contract version visible?
5. Are side effects protected by idempotency?
6. Can an operator reproduce a failure without guessing?

The goal is not a giant observability platform. Start with one JSON receipt per call, store it beside the run manifest, and make the failure path readable. Once the workflow has that small paper trail, AI tool calls feel much more like normal developer tools: understandable, testable, and safe enough to improve.