How We Route 6 AI Models in Production to Serve 19 Industries

The Problem: One Model Doesn't Fit All

When we started building SCALA — an AI operating system for service businesses — we made the classic mistake: we picked one LLM and tried to use it for everything.

Customer support in Italian? Same model. Financial report generation? Same model. WhatsApp message routing? Same model.

The result was predictable: mediocre at everything, great at nothing, and burning through tokens like they were free.

The Architecture We Landed On

After 18 months of iteration, we now route requests across 6 different models based on task type, latency requirements, and cost constraints. Here's the actual decision tree:

┌─────────────────────────────────────────────────┐
│              Request Router                       │
├─────────────────────────────────────────────────┤
│ Customer-facing chat (SARA)                      │
│   → Groq (LLaMA 3.3 70B) — 200ms p95           │
│   → Fallback: Cerebras → Mistral                │
├─────────────────────────────────────────────────┤
│ Long-form content / analysis                     │
│   → Claude Opus — accuracy over speed            │
├─────────────────────────────────────────────────┤
│ Embeddings (RAG, 57K chunks)                     │
│   → mxbai-embed-large (1024d, self-hosted)      │
│   → Jina v3 for external ingest                 │
├─────────────────────────────────────────────────┤
│ Code generation / frontend                       │
│   → Claude Sonnet — best speed/quality ratio     │
├─────────────────────────────────────────────────┤
│ Vision / document parsing                        │
│   → Multimodal model, task-dependent             │
├─────────────────────────────────────────────────┤
│ Bulk content (translations, social)              │
│   → Cheapest available, quality-gated            │
└─────────────────────────────────────────────────┘

Why This Matters for 19 Industries

SCALA serves hospitality, real estate, beauty, automotive, healthcare, and 14 other verticals. Each has different:

• Latency tolerance: A hotel guest on WhatsApp expects <3s. A quarterly report can take 30s.
• Accuracy requirements: Medical intake forms need 99%+ accuracy. Social media drafts need creativity.
• Language mix: Italian restaurant, German property manager, Brazilian beauty salon — all on the same platform.
• Cost sensitivity: An SMB paying €97/month can't subsidize enterprise-grade inference costs.

The Fallback Chain Pattern

The most important pattern we use is the fallback chain. Instead of one model with one API key, every inference path has 2-3 alternatives:

async function inference(prompt: string, config: RouteConfig) {
  for (const provider of config.chain) {
    try {
      const result = await provider.complete(prompt, {
        timeout: config.maxLatency,
        maxTokens: config.maxTokens,
      });
      if (result.quality >= config.minQuality) return result;
    } catch (e) {
      metrics.increment('fallback_triggered', { provider: provider.name });
      continue;
    }
  }
  return config.staticFallback;
}

This gives us 99.7% uptime across all inference paths — without paying for enterprise SLAs from any single provider.

Embedding Dimension: The Silent Killer

One lesson we learned the hard way: embedding dimension mismatches break RAG silently.

We have 57,000+ chunks in our knowledge base (pgvector). The embedding column is vector(1024). Everything — ingest and query — must use the same model at the same dimension.

When someone accidentally switched to a 768-dimensional model, nothing threw an error. Queries just returned garbage results. The symptom was "the AI answers badly" — not "embedding error."

Rule: If you run RAG in production, your embedding model is a contract, not a config option. Treat it like a database schema migration.

Cost Breakdown (Real Numbers)

For ~50K monthly inferences across all verticals:

Category	Monthly Cost	% of Revenue
Chat inference (Groq free tier + fallbacks)	~€0	0%
Embeddings (self-hosted Ollama)	~€0	0%
Heavy analysis (Claude)	~€80	<5%
Content generation (mixed)	~€40	<3%
Total AI cost	~€120	<8%

Our margin target is >85% on enterprise accounts, >90% on SMB. The multi-model approach makes this possible.

The 131 Free Tools Strategy

We also built 131 free business tools — invoice generators, SWOT analyzers, ROI calculators, industry benchmarks. Each one:

1. Drives organic traffic (programmatic SEO)
2. Demonstrates platform capability
3. Costs nearly nothing to run (static + edge compute)
4. Links back to the full platform

This is the distribution moat. The AI is the product, but the free tools are the acquisition channel.

What I'd Do Differently

1. Start multi-model from day 1. Don't marry a provider.
2. Build the fallback chain before you need it. The first outage is always a surprise.
3. Monitor embedding health obsessively. SELECT count(*) WHERE embedding IS NULL should be a cron job.
4. Don't self-host what you can't maintain. We self-host embeddings (stable, small model). We don't self-host chat inference (evolving fast, needs scale).

Try It

SCALA is live at get-scala.com with a 14-day free trial. The AI assistant (SARA) handles WhatsApp in 6 languages, and the platform covers CRM + BI + automation for any service business.

If you're building something similar, I'm happy to share more architecture details in the comments.

---

I'm Alessandro — solo founder building SCALA, an AI OS for service businesses. Previously 18 years in enterprise tech (Siemens, Norgine, Allianz). Ask me anything about multi-model architectures or vertical SaaS.