Automation Runbooks Need a Dry-Run Contract

Automation is at its best when it removes a repetitive decision. It is at its worst when nobody can tell what it was about to do.

That gap usually appears in scripts called --dry-run. The command prints a few lines, a teammate scans them, and everyone assumes the real run will behave the same way. Then a deployment, cleanup task, or email test uses a different account, environment, or input than expected. The dry run was technically successful, but it did not create much confidence.

I have started treating a dry run as a small contract between a developer tool and its operator. It should describe the scope, the actions it would take, the safety checks it performed, and the things it deliberately did not verify. This mental model makes automation easier to review without making the script feel heavy.

This is related to using policy objects for signup checks, where an explicit structure is easier to reason about than scattered conditions. It also pairs well with failure receipts for Playwright email tests: both workflows leave behind evidence instead of asking a person to trust console noise.

Why dry runs deserve a contract

A normal command has an obvious result: files changed, a request was sent, or a job finished. A dry run has a less visible result. It is a statement about what would happen under a particular set of inputs.

That statement needs boundaries. For example, a test helper using a temp email generator might be able to create an address and inspect a message, but a dry run should not imply that a real inbox was contacted. A database migration preview can list SQL statements without proving the statements are safe against production data.

Without those distinctions, dry-run output becomes marketing for the command instead of useful evidence. The output may say 12 actions planned, while hiding that two environment variables were missing or that the target selection was guessed.

The contract can stay small. I normally want these five questions answered:

1. What is the target? Include the environment, project, account, or resource set.
2. What would change? List actions in stable, human-readable terms.
3. What was checked? Show validation that actually ran.
4. What was skipped? Say when network calls, writes, or provider checks were not made.
5. What identifies this run? Give the operator a run ID for logs and later comparison.

The fourth question is the one teams skip most often. A dry run that did not send an email should say so clearly. Otherwise someone may read email verification passed when the script only verified that a template was present.

A small dry-run receipt format

The output does not need to be a large JSON document, but a predictable shape helps both people and CI. Here is a compact shell-style example:

DRY RUN: no writes or network side effects
run_id: 2026-10-03T11:22Z-7f31
target: staging / signup-email-checks
planned_actions: create test inbox, submit signup, wait for verification mail
checked: config, test account, message subject rule
skipped: inbox creation, signup request, message retrieval
status: ready

The skipped line is not an apology. It is a boundary. A developer tool should make it obvious which parts were simulated, inspected locally, or left untouched.

For machine consumers, I prefer writing the same fields to a receipt file:

{
  "mode": "dry-run",
  "run_id": "2026-10-03T11:22Z-7f31",
  "target": "staging/signup-email-checks",
  "planned_actions": 3,
  "checked": ["config", "test-account", "subject-rule"],
  "skipped": ["network", "writes"],
  "status": "ready"
}

Keep the fields boring and stable. A receipt is a developer tool interface, so changing target to scope every few weeks creates needless friction. The setup is small, but it pays off when a script is called from a CI job instead of a terminal.

How to use the contract in CI

In CI, run the preview as its own step before the side-effecting command. Store the receipt as an artifact, and fail early when the contract says the target is incomplete.

- name: Preview automation
  run: ./scripts/check-signup-emails --dry-run --receipt run-receipt.json

- name: Upload dry-run receipt
  uses: actions/upload-artifact@v4
  with:
    name: automation-receipt
    path: run-receipt.json

- name: Execute automation
  run: ./scripts/check-signup-emails --apply --receipt apply-receipt.json

The apply step should not silently recompute a different target. Pass the same scope, or compare a target fingerprint from both receipts. If the preview used staging and the apply step resolves to production, stop the job. That check is often more valuable than another confirmation prompt.

For email-related tests, also record whether the inbox was real, mocked, or reserved for the run. Someone searching for tamp mail com in an old log should not have to guess whether that was a typo, a provider name, or a test fixture label. Clear metadata beats clever naming.

A practical checklist

Before calling a dry run trustworthy, I check:

• The target and environment are printed near the top.
• Planned actions use the same ordering as the apply mode.
• Required configuration is validated before the preview says ready.
• Side effects are named explicitly as skipped or simulated.
• A stable run ID connects terminal output, CI artifacts, and later logs.
• Apply mode can compare its target with the preview target.
• The receipt is useful to a person who did not write the script.

It is fine if the first version only answers four of the five questions. Add the missing boundary when a real incident exposes it. Automation grows better through small feedback loops than through a giant framework invented in advance.

Q&A

Should every script have a dry-run mode?

No. A tiny read-only command may not need one. The contract is most useful when a command can modify resources, contact an external service, or make a costly batch decision.

Is a dry-run receipt the same as a test result?

Not quite. It proves what the tool inspected and planned, not that the eventual side effect succeeded. Keep the preview and apply receipts separate so they cannot be confused.

How much detail is enough?

Enough for another developer to explain the intended scope and the safety boundary without opening the script. Start with the five questions, then add fields only when they improve a decision.

The goal is not more output. It is a small, dependable promise about what automation knows, what it plans, and what it has left untouched.