Every "build a RAG chatbot in 30 minutes" tutorial ends at the demo. Then you ship to production and discover the index has no security trimming, hallucinations are 22% because there's no grounding instruction, the model burns tokens with no cache, partner A can read partner B's docs, and latency is 4 seconds because chunks are too big.
A production RAG chatbot in C# has ten more pieces. This is the full build, with real metrics from Mattrx Help — a multi-tenant marketing analytics SaaS (Angular 19 + .NET 9, 110k MAU). The in-app docs assistant indexes ~3,200 docs (~24k chunks), serves ~8,400 MAU, and runs at 4% hallucination, p95 retrieval 95ms, $0.004/query.
The stack
| Layer | Choice | Why |
|---|---|---|
| Orchestrator | Semantic Kernel | First-class .NET, plugins, telemetry |
| Vector index | Azure AI Search | Hybrid BM25+vector, security trimming, reranker |
| LLM | Azure OpenAI (gpt-4o-mini) | Same tenant, managed identity |
| Embeddings | text-embedding-3-small (1536d) | Best price/quality in 2026 |
| Chunking | 400-600 tokens / 80 overlap | Sweet spot for docs |
| Retrieval | Hybrid + semantic reranker | Best recall at scale |
| Grounding | System prompt + cited chunks | Cuts hallucination 22% -> 4% |
| Streaming | Server-Sent Events | TTFT < 500ms |
| Cache | Redis (question hash, 60s TTL) | 34% hit rate |
| Eval | 80-question golden set, nightly | Catches regressions before users |
Azure AI Search vs pgvector — the honest call
The deciding factor was the semantic reranker. Hybrid search alone got 84% top-5 recall. Adding the L2 cross-encoder reranker pushed it to 91%. For a customer-facing assistant, that 7-point bump was worth $248/month.
If you're not on Azure or don't need the reranker, pgvector is still the right answer. Don't pick AI Search because it sounds enterprise — pick it if the constraints match: multi-tenant security + reranking + already on Azure.
The five things that actually moved the numbers
1. The system prompt does most of the work. Three rules dropped hallucination 22% -> 4%: "Use ONLY the excerpts" (grounding), "If not in excerpts, say so" (an escape hatch so the model isn't pressured to confabulate), and "Cite [n]" (makes hallucinations visible and reviewable in eval).
2. Security trimming is part of the query, not a post-filter. In a multi-tenant app the most common production bug is showing tenant A's data to tenant B. The Azure AI Search filter (partnerId eq '*' or partnerId eq '{partnerId}') is enforced server-side — even a manipulated query can't get rows outside the partner's scope. Write an integration test that asserts it, every release.
3. The reranker is the most underrated feature. A 7-point recall bump for one config flag + a tier upgrade.
4. Streaming is product UX. A 2-second response feels broken; one that starts streaming in 380ms feels instant. SSE, end-to-end.
5. Tenant-scoped cache key. 34% hit rate cuts cost by a third. NEVER a global key.
Production metrics (after 4-week build + 2-week tuning)
| Metric | Value |
|---|---|
| Top-5 retrieval recall | 91% (84% hybrid -> 91% reranker) |
| Answer cites a source | 96% |
| Hallucination rate | 4% (22% before grounded prompt) |
| Query latency p95 (Azure Search) | 95 ms |
| TTFT p95 | 380 ms |
| End-to-end p95 (cache miss) | 2.1 s |
| End-to-end p95 (cache hit) | 120 ms |
| Cache hit rate | 34% |
| Cost per query (cache incl.) | $0.004 |
| Total monthly infra | ~$1,688 |
| Tickets deflected / month | ~520 (~$13,000 saved) |
| User-rated "helpful" | 84% (vs 61% human first-response) |
The chatbot pays for itself ~8x in deflected support tickets. The architecture is mostly Microsoft pieces glued together with ~600 lines of careful C#.
The right mental model
A production RAG chatbot is four boring pieces composed correctly: a vector store, a retriever, a grounded prompt, and a streaming UI. Three habits that make it pay off:
- Treat the system prompt + eval set as a unit — never change the prompt without running the eval.
- Tenant-scope every cache key and security filter.
- Streaming + grounded prompt + cache + cost cap on day one — none are optimization, they're the minimum.
The full guide has the complete Semantic Kernel C# — index schema, batched ingestion, the hybrid retriever, the grounded prompt, SSE streaming, tool-calling plugins, the OpenTelemetry trace, the eval harness, and per-partner cost monitoring:
Originally published on PrepStack.
Top comments (0)