Six failure cases to test before shipping an AI workflow

Dong Pang — Wed, 22 Jul 2026 03:50:16 +0000

An AI workflow is easy to demonstrate and harder to finish. The happy path can look convincing while retries duplicate side effects, approvals are skipped, or completion is reported without evidence.

This tutorial turns a workflow idea into a small contract and six acceptance tests. The examples are deliberately model-agnostic: they apply to an OCR review flow, a support-ticket agent, an API automation, or a research assistant.

Start with an observable contract

Before choosing tools, write five fields:

Workflow name — one bounded process, not an entire department.
Desired outcome — the state that must be observable when the run succeeds.
Human approval point — the exact irreversible or external action a person controls.
Side effect — the record, message, payment, or system change the workflow can create.
Evidence reference — the artifact that proves what happened.

A useful completion rule is:

complete = outcome observed
        AND evidence reference exists
        AND required approval is granted
        AND duplicate side effects = 0

This is stricter than “the model returned an answer.” It makes the surrounding automation testable.

Test 1: Happy path

Provide a representative input and assert the intended outcome, evidence reference, approval state, and side-effect count. A natural-language quality claim is not enough; the result needs machine-readable evidence.

Test 2: Malformed input

Remove a required field or provide an invalid type. The workflow should reject the input before calling a model or creating a side effect. The failure must be visible to an operator.

Test 3: Duplicate event

Deliver the same event twice. The second delivery should return the existing result or perform a safe no-op. If two notifications, tickets, or payments can be created, the workflow is not retry-safe.

Test 4: Dependency failure

Simulate a timeout or a downstream 5xx response. The run should record the dependency failure, preserve its evidence, and stop or retry within a fixed policy. Silent fallback is not a pass.

Test 5: Missing approval

Remove the required human decision. The workflow must stop before the external or irreversible action. “Approval is recommended” is weaker than an enforced gate.

Test 6: Evidence missing

Return a successful status without an artifact or evidence reference. The validator should reject completion. This prevents a green dashboard from becoming the only proof that work happened.

A local browser implementation

Acceptance Workbench provides an editable contract, deterministic four-file ZIP export, and a JSON/NDJSON run-log validator. It has no login, upload, tracking script, customer data, or model call. The source and verified Starter are public.

The browser tool is intentionally small. It does not claim to validate model accuracy, security, or regulatory compliance. Its job is to turn vague workflow language into observable pass/fail conditions before implementation gets expensive.

Example run-log shape

{
  "runId": "run_001",
  "status": "success",
  "outcomeEvidence": "ticket_1842 routed to escalation_queue",
  "validations": {
    "malformedInputRejected": true,
    "dependencyFailureRecorded": true
  },
  "metrics": { "duplicateSideEffects": 0 },
  "approval": { "required": true, "granted": true },
  "artifacts": ["audit_ticket_1842.json"],
  "evidenceRef": "evidence/run_001.json"
}

The important part is not the exact field names. It is that each acceptance test can point to observable evidence rather than an optimistic completion message.

Disclosure: this draft was created with AI assistance. It is intentionally unpublished and requires account-owner review before any external publication.

DEV Community: Dong Pang