If you've built or integrated an AI SQL assistant into your stack, you've likely hit this wall: it works great in the demo, it works great in week one, and then usage scales and the model API bill scales right alongside it. The default fix — swap in a cheaper model everywhere — usually just trades your cost problem for a quality problem.
The better fix is architectural: route each query to the model tier its actual complexity requires.
Why one model tier doesn't work
SQL queries vary wildly in the reasoning they require. SELECT * FROM users WHERE id = 4471 and a cross-schema retention analysis using window functions are both "SQL," but they're not remotely equivalent workloads. Routing both through a frontier model is expensive overkill for the first one.
A practical tiering scheme:
| Tier | Description | Examples | Model needed |
|---|---|---|---|
| 1 — Routine | Simple, well-defined | SELECTs, lookups, basic CRUD, syntax fixes | Fast, low-cost model |
| 2 — Moderate | Multi-step reasoning | Joins, subqueries, aggregations, optimization hints | Mid-tier model |
| 3 — Complex | Deep schema reasoning | Cross-DB queries, window functions, execution-plan tuning, schema refactoring | Frontier model |
Benchmarked figures put Tier 1 around $0.001/query versus roughly $0.03/query for a frontier model — a gap that scales linearly with volume. Tier 3 queries also need injected context (table relationships, foreign keys, indexes, dialect-specific syntax), which is expensive to carry through every request regardless of tier.
The pipeline: classify → route → execute → validate
Classification
This is the stage that determines whether the whole system works. Three implementation options:
Rule-based (regex/AST): Detect structural signals — table count, join depth, presence of window functions or subqueries. Fast, deterministic, zero model overhead. Handles the obvious cases well.
Lightweight classifier model: A small model trained specifically to estimate SQL complexity. Costs a fraction of a cent per call, which easily justifies itself by avoiding unnecessary frontier-model invocations. Can often run locally. Also useful for classifying natural-language prompts before SQL generation even happens.
Hybrid: Rules catch the clear cases for free; the classifier handles the ambiguous middle where structure alone doesn't tell you enough. This is the practical sweet spot for most teams.
Routing
Beyond tier, routing decisions should also account for:
- Schema context requirements — queries needing foreign key/index awareness typically need a higher-capability model regardless of surface complexity.
- Latency tolerance — autocomplete and inline suggestions have tight budgets; background jobs don't.
- Classifier confidence — low confidence should bias toward routing up. A bad downgrade often triggers a retry, and retries are more expensive than getting it right the first time.
Validation
Post-execution checks confirm syntax correctness, sane result shapes, and schema consistency. Failures trigger escalation and a rerun at a higher tier.
A real implementation detail worth knowing
While building schema-aware capabilities into Devart's dbForge AI Assistant, the team found that classification accuracy depended heavily on schema context — not just query structure. Queries with ambiguous table names or implicit relationships were reliably misclassified as simple and sent to models that couldn't actually resolve them correctly. The fix: feed the classifier schema metadata alongside the query itself, not just the syntax tree.
Metrics that tell you if the routing is actually working
Don't just track average cost — it hides problems.
- Cost per query, by tier. A blended average can look fine while masking a system that's routing 50% of queries to the wrong tier.
- Escalation rate. The percentage of Tier 1/2 outputs that fail validation and need rerouting. Keep it under 5%; above that, retrain the classifier or give it more schema context.
- Latency impact. Classification and routing overhead should add no more than 50–100ms.
Watch for the escalation tax: a misrouted query means a classifier call + initial model call + failed validation + reroute + second model call. Stack enough of those and you can end up paying more than if you'd just routed to the frontier model directly. Track escalation rate alongside cost per call, not in isolation.
Where the ROI ceiling sits
Well-tuned routing reportedly delivers 40–60% inference cost reduction while keeping escalation under 5% and preserving quality on complex queries. Pushing past that generally requires self-hosting smaller models for Tier 1 traffic — workable, but it adds real operational overhead (infra, monitoring, model lifecycle) that not every team needs to take on.
TL;DR for implementation
- Build the classifier before you finalize the model lineup — it's the highest-leverage piece.
- A hybrid classifier (rules + lightweight model) gets you most of the savings without excess complexity.
- Feed the classifier schema metadata, not just query syntax — this matters more than it looks like it should.
- Design validation logic before you lock in classification thresholds.
- Track escalation rate as your primary quality signal.
- As local inference gets cheaper, the payoff from correct tiering only grows — the cost gap between tiers widens, not shrinks.
This piece draws on an original analysis published on Unite.AI by Victor Horlenko, Head of AI Innovations at Devart.
Top comments (1)
Low-confidence routing up is sensible, but validation needs to measure semantic correctness, not just executable SQL and sane result shapes. A cheap model can produce valid SQL for the wrong revenue definition. I would require the route decision to carry the resolved schema objects and business metric version, then validate those before spending on query generation.