Originally published on PrepStack.
Every enterprise has now shipped "an AI feature." Almost none have shipped an AI-native architecture. The difference decides whether your AI roadmap survives 2027 — or quietly gets ripped out after the third incident.
We learned this the expensive way on Mattrx, a multi-tenant marketing-analytics SaaS (Angular 19 + .NET 9 / ASP.NET Core + Azure SQL, with a Python FastAPI AI service; 110k MAU, ~3,200 req/sec peak). The first "Mattrx Help" was a single MVC action calling a frontier model inline. It demoed beautifully. In production it leaked context across tenants, hallucinated 18% of the time, cost $0.021/query, and one tenant's runaway retry loop billed the entire fleet for an afternoon. The fix was not a better prompt — it was an architecture.
TL;DR
| Dimension | Bolt-on AI (before) | AI-native (after) |
|---|---|---|
| Model access | SDK called inline in controllers | Single governed AI gateway |
| Tenant isolation | "Please don't leak" in the prompt | Filters pushed into the data layer |
| Context size | ~14,000 tokens, stuff everything | ~3,500 tokens, assembled + ranked |
| Memory | Stateless, cold every turn | Short-term buffer + semantic long-term |
| Retrieval | Naive top-k cosine | Hybrid recall + cross-encoder rerank |
| Reasoning | One mega-prompt | Orchestrator + specialists + eval gate |
| Model choice | frontier everywhere | Routed by task complexity, with fallback |
| Actions | Agent had raw DB access | Typed, authorized tool contracts |
Production results after the rebuild:
- Hallucination rate 18% -> 3% (hybrid retrieval + rerank).
- Faithfulness 0.96, answer-relevance 0.91 (offline eval set).
- Context tokens/call 14k -> 3.5k — same answers, a quarter of the spend.
- Cost/query $0.021 -> $0.008 (mostly model routing).
- Agentic p95 latency 4.2s -> 1.8s.
- ~40 prompt-injection attempts blocked/week at gateway + identity.
- Eval gate threshold 0.90 — answers below it never reach a user.
- Zero cross-tenant leaks in the six months since.
- Help now deflects ~520 support tickets/month.
The one mental shift: stop treating the model as a feature you call, and start treating it as a tier you operate — with its own gateway, identity, memory, and governance, exactly like your database.
The nine layers (each added after a real failure)
1 → 2. The API gateway becomes an AI gateway. The first version put the model call inside a controller — no per-tenant rate limit, no token accounting, no PII redaction, no audit. Now every model call (Help, Insights, internal classification) passes through one gateway that reserves a per-tenant token budget, redacts PII before anything leaves the boundary, routes to a model, and writes an audit row. Runaway loops became a 429 for one tenant instead of a fleet-wide bill — and ~40 injection attempts/week get blocked here.
3. Identity stops meaning "who logged in." "Tell the model not to leak" is not a security control. Identity became a first-class AiPrincipal (tenant, user, scopes) that propagates into retrieval and every tool call, with the tenant filter pushed down into the vector store — data-layer enforcement, not a polite request. Zero cross-tenant leaks since.
4. The context layer (the one most teams skip). We used to concatenate everything and hope the window was big enough (~14k tokens). But bigger context isn't better — past a point recall drops and the model misses the middle. A context assembler now treats the prompt like a budget to pack: select, rank, compress to a hard token ceiling. Result: 14k -> 3.5k tokens, faithfulness 0.96.
5. Memory. Two tiers: a short-term conversation buffer (TTL-bounded) and a long-term semantic memory that persists only salient turns (decisions, preferences, corrections). It stops re-interrogating returning users — a big part of deflecting ~520 tickets/month.
6. The knowledge base — RAG done properly. Top-k cosine returns chunks that look relevant and are often subtly wrong — the worst failure mode because it sounds confident. We moved retrieval to Python: hybrid recall (BM25 + vector), then a cross-encoder rerank (the step most RAG skips), with the rule that returning nothing beats returning something wrong. This single change drove hallucination 18% -> 3%.
7. Agents. One mega-prompt that planned, queried, forecasted, and wrote up in a single shot drifted as a whole when any step drifted. Now an orchestrator plans, dispatches to specialist agents (sql / forecast / summarize), and nothing reaches the user until a 0.90 eval gate scores it. Agentic p95 4.2s -> 1.8s.
8. The LLM layer is a router, not a model. We used the frontier model for everything — including "is this about billing or analytics?" Instrumented, ~70% of calls were trivial. A router now picks the cheapest model that can do the job, with fallback. Cost/query $0.021 -> $0.008.
9. Business APIs become governed tools. The agent used to hold a database connection — functionally an unaudited admin user. Now every action is a typed, scope-checked tool: the model proposes, the tool layer decides, and the tenant is bound in code, never from model-supplied arguments. A model can hallucinate an action; it cannot hallucinate a scope check.
When NOT to build this
It earns its complexity at scale and under real risk — and is overkill otherwise: a single AI feature with no multi-tenancy, low traffic, no regulated/cross-tenant data, no v1 shipped yet, or no labeled eval set (the gate is theater without one). Smaller teams should adopt the gateway and the rerank first and stop there. We didn't build all nine at once either — each layer was a response to a specific failure, which is exactly how you should adopt it.
The model to carry forward
Treat AI as a tier, not a feature. Your database has a gateway, identity, isolation, and an audit trail because direct access doesn't stay safe at scale. The model is no different. Three habits: measure before you add a layer (no bad number, no layer); make governance structural, not textual (if a guarantee lives in a prompt, it isn't one); default to the cheapest path that passes eval.
👉 The full article — all nine layers with the complete C# (.NET 9) and Python code, the end-to-end request flow, the adoption checklist, and the full "when not to do this" — is on PrepStack:
AI-Native Architecture: The 9-Layer Blueprint
Top comments (0)