A minimal QA checklist before shipping AI app output changes

AI app changes can fail quietly. A prompt tweak, model swap, retrieval change, or schema update can still return plausible answers while breaking required fields, citations, length limits, or safety wording.

Here is the smallest clean-room routine I use before release:

1. Use synthetic fixtures only. Do not use customer logs, secrets, support tickets, private documents, or production prompts.
2. Define expected output checks in plain rules: required fields, forbidden claims, required citations, length bounds, and valid JSON when needed.
3. Run the same scenarios before every prompt/model/RAG change.
4. Generate one pass/fail release note so the team can see exactly what changed.
5. Keep one human review step for edge cases deterministic checks cannot judge.

The first three checks to start with

• required fields exist
• forbidden wording is absent
• output length stays inside expected bounds

If useful, I packaged a tiny starter kit with synthetic examples, templates, and a local runner:

https://cleanfixture-kit.kevinskysunny.workers.dev

It is intentionally clean-room: no internal company data, no customer examples, and no claim that it replaces compliance or safety review.