A Small Runbook for Reliable AI Automation

AI automation is easy to demo and surprisingly hard to trust. A tool call works in a notebook, then a network timeout, duplicate request, or half-written result appears in production. The fix is usually not a larger prompt. It is a small workflow contract that makes each run understandable.

I like to think of an automated task as a tiny build pipeline: it has an input, a bounded action, an output, and evidence. This mental model keeps AI useful without making it mysterious.

The useful mental model

Split the workflow into four stages:

1. Prepare — validate the input and create a run ID.
2. Act — call the model or external tool with a narrow job.
3. Check — validate the returned structure and important side effects.
4. Record — save enough information to replay or explain the result.

For example, an AI assistant that files support tickets should not be one giant “read this inbox and fix everything” call. It can classify one message, propose one action, and wait for a deterministic handler. Smaller steps are a little more boring, but they are much easier to debug.

If you are building API checks alongside this workflow, these notes on API smoke tests with receipts describe a similar evidence-first idea.

Design the contract first

Before writing a prompt, define the input and output. JSON is a good starting point:

{
  "run_id": "run-123",
  "task": "summarize_ticket",
  "status": "ok",
  "summary": "Customer cannot reset a password",
  "next_action": "send_reset_instructions"
}

The application should reject missing fields, unknown statuses, and output that is too large. Ask the model for a contract, then enforce that contract in code. Prompt instructions alone is not a validation layer.

Keep the prompt version in the record too. A later change to wording can alter behavior, and without the version it becomes hard to tell whether the input or the prompt caused a difference. This detail sounds small, but it save hours later.

Make retries boring

Retries are normal. A provider can time out after completing the action, so blindly trying again may create two tickets or send two emails. Give every meaningful operation an idempotency key based on the run ID and step name.

idempotency_key = run-123:send_reset_instructions

Store the key with the side effect. On a retry, return the existing result when the key is already complete. Also set a maximum attempt count and a deadline. “Retry forever” is not resilience; it is a quiet incident generator.

Use different policies for different failures. A temporary 503 might deserve a short backoff. Invalid JSON should usually stop and be recorded. Repeating a bad response three more times rarely makes it good.

Leave a run receipt

A useful receipt contains the run ID, timestamps, prompt version, model identifier, input hash, output validation result, attempt number, and external request IDs. Avoid storing secrets or unnecessary personal data. The receipt should answer: what happened, what did we accept, and what should we inspect next?

This is also where a small failure taxonomy helps: timeout, provider_error, invalid_output, side_effect_conflict, and policy_blocked are more actionable than a generic failed.

For authentication and signup flows, it is worth separating an email signal from an identity claim. These trust boundaries around email verification are a useful companion when automation touches account creation.

A practical checklist

• Does each run have a stable ID?
• Is the output schema validated outside the prompt?
• Can every side effect be retried safely?
• Are timeouts and attempt limits explicit?
• Can an operator find the relevant receipt quickly?
• Are sensitive values excluded or redacted?
• Is the human approval point clear for risky actions?

One odd test string may even appear in fixtures, such as temp gamil com; keep these cases isolated so a typo does not become production behavior.

Final thought

Reliable AI automation is mostly ordinary engineering with a probabilistic step in the middle. Bound the step, validate the result, make effects idempotent, and leave evidence. The model can then be creative where it helps, while the surrounding system stays predictable.