<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Siva Cherukuri</title>
    <description>The latest articles on DEV Community by Siva Cherukuri (@siva_cherukuri_7da89760f3).</description>
    <link>https://dev.to/siva_cherukuri_7da89760f3</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4250400%2F2713c278-16ad-4b14-a6a5-417d2e2052bd.png</url>
      <title>DEV Community: Siva Cherukuri</title>
      <link>https://dev.to/siva_cherukuri_7da89760f3</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/siva_cherukuri_7da89760f3"/>
    <language>en</language>
    <item>
      <title>Building an Open-Source RAG Application: What I Learned from Cairnly</title>
      <dc:creator>Siva Cherukuri</dc:creator>
      <pubDate>Tue, 17 Nov 2026 18:19:28 +0000</pubDate>
      <link>https://dev.to/siva_cherukuri_7da89760f3/building-an-open-source-rag-application-what-i-learned-from-cairnly-4n06</link>
      <guid>https://dev.to/siva_cherukuri_7da89760f3/building-an-open-source-rag-application-what-i-learned-from-cairnly-4n06</guid>
      <description>&lt;p&gt;I've been working on Cairnly, an open-source AI engineering learning platform that also serves as a reference implementation for building RAG applications.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GitHub:&lt;/strong&gt; &lt;a href="https://github.com/CherukuriSiva/Cairnly" rel="noopener noreferrer"&gt;https://github.com/CherukuriSiva/Cairnly&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://cairnly-blond.vercel.app" rel="noopener noreferrer"&gt;https://cairnly-blond.vercel.app&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I built it
&lt;/h2&gt;

&lt;p&gt;There are many tutorials explaining individual AI engineering concepts, such as embeddings, vector databases, retrieval, and LLM inference.&lt;/p&gt;

&lt;p&gt;I wanted to bring these concepts together into one application where learners can study AI engineering and developers can explore how the underlying system works.&lt;/p&gt;

&lt;p&gt;Cairnly currently includes 126 lessons across six tracks, along with an AI tutor that answers questions using the learning content.&lt;/p&gt;

&lt;h2&gt;
  
  
  The architecture
&lt;/h2&gt;

&lt;p&gt;The RAG pipeline follows these stages:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Convert MDX lesson content into chunks with section-level anchors.&lt;/li&gt;
&lt;li&gt;Generate embeddings and store the chunks in PostgreSQL with pgvector.&lt;/li&gt;
&lt;li&gt;Retrieve relevant content using keyword and vector search.&lt;/li&gt;
&lt;li&gt;Combine the retrieval results using Reciprocal Rank Fusion.&lt;/li&gt;
&lt;li&gt;Rerank candidates with a cross-encoder.&lt;/li&gt;
&lt;li&gt;Apply a confidence threshold before generation.&lt;/li&gt;
&lt;li&gt;Stream the answer with citations pointing to the original lesson sections.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Measuring retrieval quality
&lt;/h2&gt;

&lt;p&gt;One of the most interesting parts of the project was evaluating retrieval rather than assuming it worked well.&lt;/p&gt;

&lt;p&gt;On a 627-question evaluation set, Cairnly reports:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Recall@5: 0.971&lt;/li&gt;
&lt;li&gt;Mean Reciprocal Rank: 0.914&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;But retrieval accuracy is only part of the problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  The confidence threshold problem
&lt;/h2&gt;

&lt;p&gt;A RAG application should not confidently answer questions that its knowledge base doesn't cover.&lt;/p&gt;

&lt;p&gt;During evaluation, I found an important tradeoff.&lt;/p&gt;

&lt;p&gt;At a confidence threshold of 0.70, the system refused 96 legitimate questions but answered none of the 150 off-topic questions.&lt;/p&gt;

&lt;p&gt;At 0.66, it refused 32 legitimate questions but answered four off-topic questions.&lt;/p&gt;

&lt;p&gt;This showed me that choosing a confidence threshold isn't simply about maximizing one metric.&lt;/p&gt;

&lt;h2&gt;
  
  
  What else is included?
&lt;/h2&gt;

&lt;p&gt;Cairnly also demonstrates:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Local LLM inference with Ollama&lt;/li&gt;
&lt;li&gt;FastAPI and Next.js&lt;/li&gt;
&lt;li&gt;Streaming responses with citations&lt;/li&gt;
&lt;li&gt;Langfuse observability&lt;/li&gt;
&lt;li&gt;Prompt-injection testing&lt;/li&gt;
&lt;li&gt;Voice interaction using Whisper and Kokoro&lt;/li&gt;
&lt;li&gt;Deployment configurations for Modal, Vercel, Neon, Kubernetes, and AWS&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What's next?
&lt;/h2&gt;

&lt;p&gt;I'm planning to extend Cairnly with agent-based study planning, tool integration, and more AI engineering examples.&lt;/p&gt;

&lt;p&gt;The complete source code, evaluation approach, architectural decisions, and learning materials are available on GitHub.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/CherukuriSiva/Cairnly" rel="noopener noreferrer"&gt;https://github.com/CherukuriSiva/Cairnly&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I'd love to hear how other developers approach retrieval evaluation, confidence thresholds, and RAG reliability.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>machinelearning</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
