What's the first thing that breaks when a RAG system leaves the notebook?

I've been studying and building RAG systems for a while, and I've noticed a pattern.

Most tutorials stop once the demo works.
But production introduces a completely different set of problems.
In my experience, the first issues usually aren't model-related.
They're system-related.

Examples:

• Retrieval returns technically relevant but practically useless context
• Costs grow much faster than expected
• Evaluation becomes difficult and inconsistent
• Model updates introduce subtle behavior changes
• Latency becomes a real constraint

The more I worked on RAG systems, the more I realized that many failures happen between components rather than inside the model itself.

That shifted how I think about AI engineering.

Instead of asking:
"Can this work?"

I started asking:
"What happens when it fails?"

I've been organizing everything I've learned into an open-source project called AI Model Atlas, focused on AI system design rather than specific frameworks.

GitHub:

Hao610 / AI-Model-Atlas

Bilingual open-source AI learning map: RAG, agents, fine-tuning & system design (EN/ZH) | 中英双语开源 AI 学习路线图：RAG、智能体、微调及系统设计

🗺️ AI Model Atlas

Open-Source AI Learning Map — From Zero to RAG, Agents & Fine-Tuning

📖 Bilingual docs (EN/ZH) · 36 curriculum modules · 17 deep-dive chapters · Runnable RAG sandbox
🎯 A learning-focused architecture simulator — not a production framework, not a live model catalog.

---

📌 Reading guide

• ✅ Great for: learning AI concepts, RAG system design, and hands-on experimentation
• ⚠️ Model names & API prices may change — always verify against official docs
• 🚫 Not intended for: production deployment or real-time benchmarking

---

🧭 Start Here

I want to…	Go to
📚 Learn from scratch (step-by-step)	CURRICULUM.md — 36 modules, Phase 1→5
🧬 Understand the math & internals	DEEP_DIVES.md — 17 chapters
📐 See the system architecture	ARCHITECTURE.md
▶️ Run the demo in 5 min	Quick Start ↓
🗺️ Pick a learning track	Getting Started Guide

---

📦 What's Inside

Content	Count	Description
Curriculum	36 modules	Prompt → RAG

…

View on GitHub

For people who have deployed RAG systems:
What was the first production issue that surprised you?