When to Write Tests for AI Features (Almost Never)

• Non-deterministic outputs make unit tests useless. Model responses shift with every update or temperature setting, so your assertions fail randomly.
• Focus effort on the deterministic wrapper instead. The code that calls the API, handles errors, and formats results stays stable and deserves the tests.
• Prompt changes break tests faster than code does. Tweaking instructions happens daily, turning test maintenance into busywork.
• Run manual checks on high-stakes features only. For an AI product photography tool, I review a few dozen outputs by eye before shipping.
• Evals beat classic tests for quality. Compare outputs against a small golden set instead of writing brittle assertions.
• Skip tests when the fallback is cheap. If bad AI output just shows a default creative, monitoring in production costs less than upfront test suites.
• Write tests once the AI layer is swapped for rules. The moment behavior becomes predictable, the tests finally make sense.

Most AI features in early products improve faster through iteration and logs than through test coverage. I learned this building ad tools where speed matters more than perfect coverage.