I've been working on Cairnly, an open-source AI engineering learning platform that also serves as a reference implementation for building RAG applications.
GitHub: https://github.com/CherukuriSiva/Cairnly
Live demo: https://cairnly-blond.vercel.app
Why I built it
There are many tutorials explaining individual AI engineering concepts, such as embeddings, vector databases, retrieval, and LLM inference.
I wanted to bring these concepts together into one application where learners can study AI engineering and developers can explore how the underlying system works.
Cairnly currently includes 126 lessons across six tracks, along with an AI tutor that answers questions using the learning content.
The architecture
The RAG pipeline follows these stages:
- Convert MDX lesson content into chunks with section-level anchors.
- Generate embeddings and store the chunks in PostgreSQL with pgvector.
- Retrieve relevant content using keyword and vector search.
- Combine the retrieval results using Reciprocal Rank Fusion.
- Rerank candidates with a cross-encoder.
- Apply a confidence threshold before generation.
- Stream the answer with citations pointing to the original lesson sections.
Measuring retrieval quality
One of the most interesting parts of the project was evaluating retrieval rather than assuming it worked well.
On a 627-question evaluation set, Cairnly reports:
- Recall@5: 0.971
- Mean Reciprocal Rank: 0.914
But retrieval accuracy is only part of the problem.
The confidence threshold problem
A RAG application should not confidently answer questions that its knowledge base doesn't cover.
During evaluation, I found an important tradeoff.
At a confidence threshold of 0.70, the system refused 96 legitimate questions but answered none of the 150 off-topic questions.
At 0.66, it refused 32 legitimate questions but answered four off-topic questions.
This showed me that choosing a confidence threshold isn't simply about maximizing one metric.
What else is included?
Cairnly also demonstrates:
- Local LLM inference with Ollama
- FastAPI and Next.js
- Streaming responses with citations
- Langfuse observability
- Prompt-injection testing
- Voice interaction using Whisper and Kokoro
- Deployment configurations for Modal, Vercel, Neon, Kubernetes, and AWS
What's next?
I'm planning to extend Cairnly with agent-based study planning, tool integration, and more AI engineering examples.
The complete source code, evaluation approach, architectural decisions, and learning materials are available on GitHub.
Repository: https://github.com/CherukuriSiva/Cairnly
I'd love to hear how other developers approach retrieval evaluation, confidence thresholds, and RAG reliability.
Top comments (0)