Honest Fallbacks: Deterministic Classification When the Model Is Down

Why this matters

AI-powered SaaS products face an unglamorous reality: models time out, APIs rate-limit, and providers have bad days. The product still has to respond. The question is what it says when the smart path is unavailable.

Deterministic classification as a fallback

A pattern we keep returning to is keeping a deterministic classifier alongside the model-backed one. Rules-based keyword scoring, exact-match routing, and finite-state pipelines can cover a meaningful slice of queries without any inference at all.

• Keyword + regex routing covers the most common intents cheaply.
• Finite-state pipelines give auditable, repeatable decisions.
• Cached embeddings let you keep nearest-neighbor lookups alive during provider outages.

The point is not that rules beat models. It is that a predictable degraded mode beats a 500 error.

Design principles for fallbacks

1. Degrade, do not crash. Every model-backed endpoint should have a defined non-model path, even if it is narrower.
2. Label the mode. Tell users when they are getting the deterministic path - honesty builds more trust than pretending the slow answer was fast.
3. Log the delta. Track how often the fallback fires and how its output differs. That is your roadmap for which rules to improve.
4. Keep it boring. Fallback code should be the most boring, most tested code in the repo. It only runs when things are already going wrong.

What we have learned

Deterministic classification will not match model quality on ambiguous input. But for structured, high-volume request types - triage, routing, tagging - it covers enough ground to keep the product useful during outages, and the audit trail it produces is often better than model output when someone asks why a decision was made.

Honest fallbacks are not a stopgap. They are a product feature: a system that behaves predictably under pressure is easier to trust than one that only works when the stars align.

---

Disclosure: this article was written with AI assistance and describes the engineering approach of our own product. We build ChangelogForge, an AI-assisted changelog and release-notes tool.