AI Workflows Need a Dry-Run Contract

AI automation is easy to demo and surprisingly hard to operate.

An agent can read a ticket, inspect a repository, call an API, and open a pull request in a few minutes. The happy path looks magical. The uncomfortable question comes later: what exactly did it change, and could we have seen that before it happened?

The useful answer is not “add more logs.” Logs tell us what happened after the workflow ran. For developer tools, we also need a preview that explains what the workflow would do, which permissions it needs, and which actions are reversible.

I have started treating this as a dry-run contract. It is a small agreement between an AI workflow, its tools, and the person approving the result.

The dangerous part of an AI workflow is the side effect

Reading files and proposing a patch are usually low-risk. Sending a message, deleting a test fixture, rotating a credential, or changing a production setting is different. These actions cross a boundary from reasoning into the world.

The boundary gets blurry when a single tool call does both. A function named sync_customer might fetch records, create missing records, and send notifications. A developer reviewing the trace then has to guess which steps were observations and which were mutations.

The first design rule is simple: make side effects explicit. A preview should list each proposed action with a stable ID, its target, and its expected consequence. It should also say when an action cannot be reversed.

This matters for test systems too. A tempmail disposable fixture may look harmless, but it still has ownership, an expiry time, and a cleanup obligation. The same thinking applies to a browser test or a CI job; a failure budget for asynchronous email jobs is a useful reminder that “test data” still consumes a limited operational budget.

Define preview, apply, and verify as separate states

Do not model a dry run as a boolean that gets passed into every function. That approach usually end with one forgotten branch that performs the real action.

Instead, give the workflow three visible states:

1. Preview — collect intended actions without mutating external state.
2. Apply — execute only the actions that were approved.
3. Verify — check the observed result and report anything different from the preview.

The preview and apply phases should share an action format, even if they use different implementations. For example:

{
  "run_id": "run-2026-11-13-042",
  "mode": "preview",
  "actions": [
    {
      "id": "a-01",
      "kind": "create_test_fixture",
      "target": "checkout-verification",
      "reversible": true,
      "requires_approval": false
    },
    {
      "id": "a-02",
      "kind": "open_pull_request",
      "target": "acme/shop",
      "reversible": false,
      "requires_approval": true
    }
  ]
}

The point is not the exact JSON. The point is that an AI workflow can explain its plan before it asks a tool to carry it out. If the tool call may looks different from the preview, verification has something concrete to compare.

Use a small action ledger

A workflow needs one place to answer four questions:

• What did the agent intend to do?
• What was approved?
• What actually happened?
• What still needs cleanup or review?

An append-only action ledger is often enough. Store the preview, approval decision, tool response ID, and verification result under one run ID. Keep payloads small; use hashes or opaque IDs when full content is private.

This ledger is especially useful when a workflow retries. The second attempt should not silently create a second fixture or open a duplicate pull request. Give each action an idempotency key, then record whether the retry reused, skipped, or replaced the original action.

A good ledger are boring to read. That is a feature. During a failure, a developer should not need to reconstruct the story from a model trace and five provider dashboards.

For email-driven tests, separate “message requested,” “message delivered,” and “link verified.” A browser test can pass one state while failing the next. A failure taxonomy for browser email tests shows why a single pass/fail label hides useful causes.

Make tool permissions visible

The agent should not receive a giant toolbox and be expected to behave. Group tools by risk:

• Observe: read files, query status, inspect logs.
• Prepare: create a patch, assemble a message, produce a test fixture.
• Mutate: write files, send messages, change records, publish artifacts.
• Irreversible: delete data, revoke access, release to production.

The preview can then state which permission groups it needs. A workflow that only observes should not be able to call an irreversible tool by accident. When a new tool is added, its risk category becomes part of the review instead of an undocumented assumption.

This also improves prompts. “Do the task” is vague. “Prepare a patch, show the action ledger, and wait before any mutate action” is an operational contract the workflow can follow.

There are small wording mistakes in real prompts too, such as tepm mail com or tem email. Keep these odd search phrases in a test fixture or compatibility case when they are needed, but do not let them become action names. Human review gets harder when identifiers are ambiguous.

A practical dry-run checklist

Before giving an AI workflow access to a new tool, ask:

• Does preview mode produce a complete list of external actions?
• Can every action be tied to a run ID and idempotency key?
• Are approval-required and irreversible actions clearly marked?
• Does apply execute the approved preview, rather than rebuilding a new plan?
• Does verify compare expected and observed results?
• Are retries visible in the ledger?
• Are private payloads minimized or protected?
• Does cleanup run after success, failure, and cancellation?

The best dry runs are not theatrical simulations. They are useful developer artifacts: small enough to review, precise enough to approve, and durable enough to explain a failure later.

AI makes it cheap to start more automation. A dry-run contract makes that automation easier to trust. Once preview, apply, and verify are separate states, the workflow becomes less like a black box and more like a tool a team can safely improve.