DEV Community

Siva Cherukuri
Siva Cherukuri

Posted on

Building an Open-Source RAG Application: What I Learned from Cairnly

I've been working on Cairnly, an open-source AI engineering learning platform that also serves as a reference implementation for building RAG applications.

GitHub: https://github.com/CherukuriSiva/Cairnly

Live demo: https://cairnly-blond.vercel.app

Why I built it

There are many tutorials explaining individual AI engineering concepts, such as embeddings, vector databases, retrieval, and LLM inference.

I wanted to bring these concepts together into one application where learners can study AI engineering and developers can explore how the underlying system works.

Cairnly currently includes 126 lessons across six tracks, along with an AI tutor that answers questions using the learning content.

The architecture

The RAG pipeline follows these stages:

  1. Convert MDX lesson content into chunks with section-level anchors.
  2. Generate embeddings and store the chunks in PostgreSQL with pgvector.
  3. Retrieve relevant content using keyword and vector search.
  4. Combine the retrieval results using Reciprocal Rank Fusion.
  5. Rerank candidates with a cross-encoder.
  6. Apply a confidence threshold before generation.
  7. Stream the answer with citations pointing to the original lesson sections.

Measuring retrieval quality

One of the most interesting parts of the project was evaluating retrieval rather than assuming it worked well.

On a 627-question evaluation set, Cairnly reports:

  • Recall@5: 0.971
  • Mean Reciprocal Rank: 0.914

But retrieval accuracy is only part of the problem.

The confidence threshold problem

A RAG application should not confidently answer questions that its knowledge base doesn't cover.

During evaluation, I found an important tradeoff.

At a confidence threshold of 0.70, the system refused 96 legitimate questions but answered none of the 150 off-topic questions.

At 0.66, it refused 32 legitimate questions but answered four off-topic questions.

This showed me that choosing a confidence threshold isn't simply about maximizing one metric.

What else is included?

Cairnly also demonstrates:

  • Local LLM inference with Ollama
  • FastAPI and Next.js
  • Streaming responses with citations
  • Langfuse observability
  • Prompt-injection testing
  • Voice interaction using Whisper and Kokoro
  • Deployment configurations for Modal, Vercel, Neon, Kubernetes, and AWS

What's next?

I'm planning to extend Cairnly with agent-based study planning, tool integration, and more AI engineering examples.

The complete source code, evaluation approach, architectural decisions, and learning materials are available on GitHub.

Repository: https://github.com/CherukuriSiva/Cairnly

I'd love to hear how other developers approach retrieval evaluation, confidence thresholds, and RAG reliability.

Top comments (0)