DEV Community: Isaac Natarajan

Building my-assistant: teaching an AI agent to find my own files

Isaac Natarajan — Mon, 03 Aug 2026 07:39:00 +0000

Note: this post covers V1 of the project. It's a real, working assistant I use daily — not a finished product. There's a V2 in the works, and I'd genuinely love your ideas on it (more on that at the end).

Why I built this

I started learning agentic AI the way most people probably do — watching a course, working through LangChain and LangGraph fundamentals one concept at a time: state, nodes, edges, tools, memory, human-in-the-loop, guardrails. It's a lot of scaffolding before you get to build anything real, and at some point the tutorial-project itch wears off. I didn't want to build another "spec-to-API agent" demo that I'd never open again. I wanted something I'd actually use.

So I picked a problem I genuinely have: too many files, scattered across Downloads, Desktop, and Documents, and no memory of where anything is. The result is my-assistant — an always-on-top desktop widget where I can just ask, in plain English, "what did I write about TaskFlow's approval flow?" or "open my timesheet from last week," and it finds it.

What it actually does

Under the hood, it's one AI agent — not a swarm of them, more on that below — built with LangChain and LangGraph, backed by a local vector database (Qdrant) for semantic search, and a Groq-hosted model for the actual reasoning. It has six tools: search by content, search by filename, open a file or folder, count indexed files, list indexed folders, and index new files. A Flet-based UI wraps all of it into a small, always-on-top chat widget that sits on my desktop.

The part I'm most proud of isn't the search — it's that the index stays alive on its own. A background file-system watcher notices when I download something new, edit an existing document, or delete a file, and updates the index automatically, without me ever running a manual command. Ask it "index my new PDFs" and it'll do that too, or just let it work silently in the background.

One agent, not many — and why that took me a while to actually understand

Early on, I assumed a "long process with lots of steps" — scan folders, extract text, chunk it, embed it, store it — meant I needed multiple agents working together. It doesn't, and figuring out why it doesn't was one of the more useful lessons of this whole project.

The thing that actually determines whether you need an agent isn't how many steps a process has — it's whether any step requires judgment on ambiguous input. Extracting text from a PDF has exactly one correct way to do it, every time. So does chunking, embedding, and writing to a vector store. None of that needs an LLM's reasoning — it needs a deterministic pipeline. The only place genuine ambiguity shows up is right at the front: interpreting what I actually meant when I typed something into the chat box. That's the entire agentic surface area. Everything after it is plumbing.

That reframing changed how I think about when multi-agent design is actually worth it — not "this has many steps," but "does some sub-task need a genuinely different kind of judgment or persona than the rest." More on where that might actually apply in V2.

The debugging stories that taught me the most

A few things went wrong along the way that ended up teaching me more than the parts that worked first try.

The index that quietly ate my own virtual environments. My exclusion logic checked for folders literally named venv or .venv — until I found one named venv_rag, and hundreds of site-packages files from unrelated projects showed up in my search results. The fix wasn't adding more exact name checks; it was detecting virtual environments structurally, by the presence of pyvenv.cfg, so it doesn't matter what anyone names them.

"Storage folder is already accessed by another instance." Once I added a background watcher running independently of the chat agent, both started occasionally trying to open their own separate connections to the same local Qdrant database at the same moment — something that's fine for a single sequential agent, but breaks the instant you have two independent threads touching the same embedded database. The fix was a shared singleton connection instead of a fresh one per call.

The agent that wouldn't stop flailing. I asked about a table buried inside a Word document, and the agent, unsatisfied with its first search, started calling unrelated tools — trying to re-index files, trying to physically open the document in Word — cycling between them instead of just telling me it couldn't find something. The actual root cause, once I dug in with a raw database query, was almost funny: the content was there all along. It just wasn't ranking in the top 3 search results for that particular phrasing. The real fixes were tightening the system prompt so the agent stops and tells me instead of flailing, and widening how many results it pulls per search.

A spreadsheet answer that quietly wasn't stale, but the reasoning behind it was invented. Asking about a folder's contents, the agent gave me the right file count, but with a made-up explanation for why it couldn't show subfolders — a limitation that didn't actually exist. It's a good reminder that a technically correct answer can still come with confidently wrong reasoning attached, and it's worth checking both.

Windows Explorer creating a "deleted" file that was never indexed. Creating a new blank text file somehow triggered a "removed from index" message before the file even had content. Turned out Explorer's file-creation flow fires as a rename event under the hood, and my delete-handler was printing success messages unconditionally, regardless of whether anything was actually removed. A small bug, but the kind that erodes trust in a tool's logs if you don't catch it.

None of these were exotic problems. They were the ordinary friction of building something real instead of following a tutorial — and that's exactly why they were worth having.

What's next (and where I'd like your ideas)

This is V1. It works, and I use it daily, but it's not the end state. A few directions I'm considering for V2, specifically where I think multiple specialized agents would actually earn their keep rather than just be complexity for its own sake:

A writing agent that can draft summaries or emails from documents the file agent retrieves — a genuinely different skill and voice than "find and open files."
A vision agent for screenshots and scanned documents, if reasoning about images turns out to need real judgment rather than a single API call.
A proactive mode that notices clutter or duplicate files and suggests cleanup, instead of only reacting to what I ask.
Packaging it as a real standalone Windows app, so it doesn't need a terminal window open in the background.

If you've built something like this, or have a feature you'd genuinely want out of a personal file assistant, I'd like to hear it — drop a comment or reach out. Still very much a work in progress, and that's kind of the point.

GitHub :- https://github.com/IsaacNatarajan/My-Assistant/tree/main

I Built an AI That Interviews You Like a Real Mentor — Here's What I Learned About Multi-Agent Systems

Isaac Natarajan — Fri, 03 Jul 2026 18:28:27 +0000

A few weeks ago I set out to learn LangGraph. I ended up building something I actually use.

Here's the origin story, the architecture, and the mistakes that taught me the most.

The Problem With Interview Prep Tools

Most interview prep apps are the same thing wearing different skins: a list of questions, a text box, maybe a model answer if you're lucky. They don't react to you. They don't know if you're stuck, confused, or just had a brain fart and typed something completely unrelated. They grade you like a quiz, not coach you like a mentor.

I wanted to build something different — a system that feels less like a form and more like sitting across from someone who actually wants to see you improve.

And since I'd already worked with LangChain and LangSmith, this felt like the perfect excuse to finally learn LangGraph properly, by building something with real stakes: multiple agents, real conditional logic, and a genuinely useful end product.

Starting With the Wrong Architecture (On Purpose, Sort Of)

My first instinct was the "trendy" AI-app checklist: RAG, a vector database, the works. I was building this in the middle of a big shift in the ecosystem too — mid-project, Groq announced they were deprecating the exact models I'd been using (llama-3.3-70b-versatile and llama-3.1-8b-instant), pushing everyone toward their newer gpt-oss models. A good reminder that in this space, the ground moves under you constantly, and your architecture needs to be flexible enough to swap a model string without a rewrite.

But early on, a simpler realization changed the whole design: this system doesn't need a knowledge base. The questions aren't retrieved from documents — they're generated live, tailored to a role, an experience level, and an optional job description. Qdrant would have added infrastructure that solved a problem I didn't have.

So I cut it. MongoDB became the single source of truth for everything — sessions, users, performance history. One database, one mental model, zero unnecessary complexity. Sometimes the best architectural decision is the thing you don't build.

The Part That Actually Made It "Agentic"

Here's the distinction that took me the longest to internalize: using multiple LLM calls doesn't make a system agentic. Deciding which call to make, dynamically, based on context — that's what does.

My first version had a rigid pipeline: user answers → evaluator scores it → maybe a hint → done. It worked, but it was fragile. The moment a user typed something like "can we have more questions?" instead of an actual answer, the system would dutifully score it as a wrong answer. Because as far as the code was concerned, every input was "an answer."

That was the wake-up call. I built an Intent Classifier Agent that runs in front of everything else, reading the user's message and routing it to the right specialist:

Genuinely answering the question? → Evaluator Agent
Stuck and want help? → Hint Generator
Don't understand the question itself? → Question Clarifier (a new agent I hadn't planned for)
Want more detail on feedback you already got? → Answer Elaborator (another agent born from user testing, not the original design)
Talking about the session, not the question? → Off-Topic Handler
Want to skip? → Skip logic, no scoring, no judgment

Nine agents now, each with one job, coordinated through LangGraph's conditional edges instead of a fixed sequence. That's the difference between a chatbot with extra steps and an actual multi-agent system.

The Bugs That Taught Me the Most

A few humbling moments, because no build log is honest without them:

The Python scoping bug that looked like a networking issue. I had a function reassign a variable (state = hint_generator_agent(state)) inside what I thought was a safe scope. Python quietly decided the whole variable was local to the function from that point on — including before the reassignment — and threw an UnboundLocalError that manifested downstream as a browser CORS error. I spent twenty minutes checking my CORS config before realizing the backend had crashed entirely.

The evaluator that graded a session it never saw. Reload a chat mid-session, and the system would ask the exact same question the user had just answered — because the "current question" was only updated when explicitly requested, not the moment an answer came in. Small state-management oversight, big trust-breaking bug if left in.

The tone problem. My evaluator originally wrote feedback like a report: "The candidate demonstrated..." Clinical, third-person, cold. The fix wasn't more intelligence — it was a prompt rewrite asking the model to speak in first person, use the user's name occasionally, and react to what they specifically said rather than generic praise. One prompt change transformed the entire feel of the product from "quiz" to "conversation."

What "Agentic" Actually Buys You

By the end, the system could handle things I never explicitly coded for. Say "I don't get it" and it rephrases the question in its own words — as the same interviewer, not a third party summarizing someone else's question. Ask "can you elaborate?" after getting feedback, and it digs deeper into that specific model answer, not a generic explanation.

None of that was hardcoded as a feature. It emerged from giving the system the ability to classify intent and route accordingly. That's the part of agentic design that's easy to describe in a sentence and genuinely hard to appreciate until you've built the alternative first and watched it fail.

What's Next

Docker's wired up so anyone can clone the repo, drop in an API key, and run the whole stack — frontend, backend, and MongoDB — with one command. Langfuse traces every agent call, which turned out to be invaluable for actually seeing the routing decisions happen in real time, not just trust that they were happening.

From here: a proper deployment, resume-based personalization, and maybe a voice mode, because reading and typing answers isn't quite the same pressure as saying them out loud.

If you're thinking about learning LangGraph, my honest advice: don't build a toy example. Build something with a real, opinionated use case, let real usage expose the gaps in your first design, and be ready to throw away your first architecture when a simpler one reveals itself.

Check out the full project, architecture, and setup instructions on GitHub and feedback welcome.

I Built a Mental Health Journal Using MCP, Claude Desktop and NVIDIA NIM — Here's How

Isaac Natarajan — Sun, 28 Jun 2026 16:24:57 +0000

I've been exploring Model Context Protocol (MCP) recently and wanted to build something that was actually useful in my day-to-day life — not just another todo app or weather tool.

Mental health journaling is something I've always wanted to be consistent with, but the friction of opening an app, typing out my thoughts in a structured way, and then getting no real feedback always stopped me.

So I built a Mental Health Journal MCP — where I just talk to Claude naturally about my day and it handles everything else automatically. Logging, detecting cognitive distortions, tracking mood trends, suggesting coping strategies — all from a single conversation.
Here's the full story of how I built it.

What is MCP?
Before diving in, let me quickly explain Model Context Protocol (MCP) for those who haven't heard of it.

MCP is an open protocol by Anthropic that lets you connect AI models like Claude to external tools and data sources. Think of it like giving Claude the ability to call functions — read from a database, call an API, write files — anything you want.

You build an MCP server that exposes a set of tools, and an MCP client (like Claude Desktop) connects to it and uses those tools automatically during conversation.

It's a game changer for building AI-powered personal tools.

The Problem I Wanted to Solve
Traditional journaling apps require:

Opening an app
Filling structured forms (date, mood, text)
Getting no intelligent feedback

I wanted to just say:

I had a really stressful day today, kept worrying about my project deadlines and felt like everything was falling apart.

And have the system:

Log the entry automatically
Detect thinking patterns (cognitive distortions)
Suggest coping strategies
Track my mood over time

That's exactly what this project does.

Architecture

Claude Desktop (MCP Client)
        ↓
MCP Server (Python + FastMCP)
        ↓
NVIDIA NIM — meta/llama-4-maverick-17b-128e-instruct (AI Judge)
        ↓
SQLite (Local Storage)

The key design decision here is the dual-model pattern:

Claude handles the conversational layer — empathy, responses, deciding which tool to call
NVIDIA NIM (Llama 4 Maverick) acts as the AI judge — focused tasks like detecting cognitive distortions and generating reports

This keeps each model doing what it's best at.

Tools I Built

Tool	What it does
log_journal_entry	Logs date, text, and mood score to SQLite
detect_cognitive_distortions	Uses Llama 4 Maverick to analyze thinking patterns
analyze_mood_trends	Queries database and calculates mood statistics over a date range
get_stress_triggers	Identifies low mood entries and uses AI to detect patterns
suggest_coping_strategy	Generates personalized coping advice using AI
generate_weekly_report	Creates a full weekly mental health summary
set_intention	Sets a daily intention
check_intention_reflection	Reflects on how well the intention was followed

Setting Up the Project

Dependencies

mkdir mental-health-journal-mcp
cd mental-health-journal-mcp
python -m venv venv
venv\Scripts\activate
pip install uv
uv pip install mcp httpx python-dotenv

Database Setup (db.py)

import sqlite3
import os

DB_PATH = os.path.join(os.path.dirname(__file__), "data", "journal.db")

def get_connection():
    os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    return conn

def init_db():
    conn = get_connection()
    cursor = conn.cursor()

    cursor.executescript("""
        CREATE TABLE IF NOT EXISTS journal_entries (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            date TEXT NOT NULL,
            text TEXT NOT NULL,
            mood_score INTEGER NOT NULL,
            distortions TEXT,
            triggers TEXT,
            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );

        CREATE TABLE IF NOT EXISTS intentions (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            date TEXT NOT NULL,
            intention TEXT NOT NULL,
            reflection TEXT,
            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );
    """)

    conn.commit()
    conn.close()

Building the MCP Server

Server Skeleton (server.py)

from mcp.server.fastmcp import FastMCP
from db import init_db, get_connection
from dotenv import load_dotenv
import httpx
import os

load_dotenv()
init_db()

mcp = FastMCP("Mental Health Journal")

Logging Journal Entries

@mcp.tool()
def log_journal_entry(date: str, text: str, mood_score: int) -> str:
    """Log a journal entry with date, text and mood score (1-10)"""

    if not 1 <= mood_score <= 10:
        return "Mood score must be between 1 and 10"

    conn = get_connection()
    cursor = conn.cursor()

    cursor.execute("""
        INSERT INTO journal_entries (date, text, mood_score)
        VALUES (?, ?, ?)
    """, (date, text, mood_score))

    conn.commit()
    conn.close()

    return f"Journal entry logged for {date} with mood score {mood_score}/10"

Detecting Cognitive Distortions (The AI Judge)

This is where it gets interesting. I use NVIDIA NIM's Llama 4 Maverick as a focused AI judge to analyze journal text:

@mcp.tool()
def detect_cognitive_distortions(entry_text: str) -> str:
    """Analyze journal entry text and detect cognitive distortions using AI"""

    api_key = os.getenv("NVIDIA_API_KEY")

    prompt = f"""You are a cognitive behavioral therapy expert. Analyze this journal entry and detect any cognitive distortions present.

Journal entry: {entry_text}

List the cognitive distortions found (e.g. catastrophizing, all-or-nothing thinking, mind reading, overgeneralization etc.) and briefly explain why. Be concise and empathetic."""

    response = httpx.post(
        "https://integrate.api.nvidia.com/v1/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json={
            "model": "meta/llama-4-maverick-17b-128e-instruct",
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 300
        }
    )

    return response.json()["choices"][0]["message"]["content"]

Here's a real output I got:

I've identified two cognitive distortions: Catastrophizing — expecting the worst-case scenario without considering other outcomes. Fortune Telling — predicting future failure without concrete evidence.

That's genuinely useful feedback from a journal entry.

Making it Feel Natural

The real magic happens with a system prompt in Claude Desktop. I created a Project and added these instructions:

You are a compassionate mental health journal assistant.
When the user shares how they are feeling or talks about their day:

Always automatically log a journal entry using the log_journal_entry tool with today's date

If the user doesn't mention a mood score, estimate it from their tone (1-10)

After logging, automatically run detect_cognitive_distortions on their entry

Respond empathetically and suggest coping strategies if mood score is 4 or below When the user asks about their mood, week, patterns or triggers:

Use analyze_mood_trends, get_stress_triggers or generate_weekly_report automatically Always be warm, non-judgmental and supportive.

Now I just talk naturally and everything happens automatically. No commands, no forms, no friction.

Connecting to Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "mental-health-journal": {
      "command": "C:\\path\\to\\venv\\Scripts\\python.exe",
      "args": ["C:\\path\\to\\server.py"]
    }
  }
}

On Windows the config file is located at:

C:\Users\<username>\AppData\Local\Packages\Claude_pzs8sxrjxfjjc\LocalCache\Roaming\Claude\claude_desktop_config.json

Restart Claude Desktop and your tools show up automatically.

What I Learned

1. MCP is incredibly powerful for personal tools — the ability to give Claude access to your own data and functions opens up so many possibilities.
2. The dual-model pattern works well — using a lighter, focused model (Llama 4 Maverick) for specific analysis tasks keeps things efficient and consistent.
3. System prompts are the UX layer — the difference between a tool that feels robotic and one that feels natural is entirely in how you prompt the client model.
4. SQLite is underrated — for personal local tools, SQLite is perfect. Zero setup, fast, private.

Source Code

The full source code is available on GitHub: https://github.com/IsaacNatarajan/Mental-Health-Journal/

If you're exploring MCP and want to build something practical, I highly recommend starting with a personal tool like this. The feedback loop of actually using what you build makes learning so much faster.

Have questions or want to share what you're building with MCP? Drop a comment below! 👇

Building an Advanced RAG System with Multiple Chunking Strategies — A Practical Guide

Isaac Natarajan — Thu, 25 Jun 2026 19:32:15 +0000

I built an Advanced RAG system that compares 4 chunking strategies (fixed-size, recursive, semantic, hierarchical) on Apple's 10-K filings using NVIDIA NIM models, Qdrant, and custom evaluation metrics. Semantic chunking won with an overall score of 0.86. Here's everything I learned.

Introduction
Retrieval-Augmented Generation (RAG) is one of the most practical applications of LLMs today. Instead of relying on a model's training data, RAG retrieves relevant information from your own documents and uses it to generate accurate, grounded answers.

But here's what most RAG tutorials skip: how you chunk your documents matters enormously. The same pipeline with different chunking strategies can produce wildly different results. I wanted to test this properly, so I built a system that runs 4 chunking strategies side by side on the same corpus and evaluates them with real metrics.

In this post I'll walk through everything — data ingestion, chunking, vector storage, retrieval, generation, evaluation, and a Streamlit chatbot UI.

Tech Stack

Component	Tool / Technology
Embedding Model	NVIDIA llama-nemotron-embed-1b-v2
LLM	NVIDIA llama-3.3-nemotron-super-49b-v1.5
Vector Database	Qdrant (local via Docker)
Pipeline	LangChain
Tracing	LangSmith
Evaluation	Custom LLM-as-judge metrics
UI	Streamlit

All NVIDIA models are accessed via https://integrate.api.nvidia.com/v1 which is OpenAI-compatible, making integration straightforward.

One important quirk with llama-3.3-nemotron-super-49b-v1.5 — it has a thinking mode that needs to be explicitly disabled, and you need high max_tokens (8192+) otherwise the model spends all its tokens on internal reasoning and returns None as content:

response = client.chat.completions.create(
    model=LLM_MODEL,
    messages=[...],
    max_tokens=8192,
    extra_body={"chat_template_kwargs": {"thinking": False}}
)

Similarly, the embedding model is asymmetric and requires an input_type parameter:

# For document chunks
client.embeddings.create(model=EMBED_MODEL, input=text, extra_body={"input_type": "passage"})

# For queries
client.embeddings.create(model=EMBED_MODEL, input=query, extra_body={"input_type": "query"})

Data Ingestion
I used Apple's 10-K annual reports for 2022 and 2023, downloaded directly from Apple's investor relations page as PDFs. Financial documents are ideal for this kind of project because they have mixed content — dense paragraphs, tables, numbered sections, and boilerplate — which makes chunking strategy comparison genuinely meaningful.

Extraction with pdfplumber:

import pdfplumber

def load_pdfs():
    documents = []
    for filename in os.listdir("data/pdfs"):
        if filename.endswith(".pdf"):
            with pdfplumber.open(f"data/pdfs/{filename}") as pdf:
                full_text = ""
                for page in pdf.pages:
                    text = page.extract_text()
                    if text:
                        full_text += text + "\n"
            documents.append({"filename": filename, "text": clean_text(full_text)})
    return documents

After cleaning, I ended up with ~221k characters from the 2022 report and ~207k from 2023.

The 4 Chunking Strategies
This is the heart of the project. Each strategy produces a different number and quality of chunks from the same documents.

1. Fixed-size Chunking
The simplest approach — split every N characters with some overlap regardless of content boundaries.

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(chunk_size=512, chunk_overlap=50, separator="\n")
chunks = splitter.split_text(text)

Result: 951 chunks
Pros: Fast, simple, predictable
Cons: Cuts across sentences and paragraphs, losing context

2. Recursive Character Splitting
LangChain's default. Tries to split on paragraph breaks first, then sentences, then words — preserving semantic units as much as possible.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ".", " ", ""]
)
chunks = splitter.split_text(text)

Result: 954 chunks
Pros: Smarter splits, respects natural language boundaries
Cons: Still fixed-size, just more intelligent about where to cut

3. Semantic Chunking
Instead of splitting by size, this approach embeds every sentence and splits where the semantic similarity between adjacent sentences drops below a threshold. Topics stay together, topic boundaries become chunk boundaries.

def semantic_chunking(documents, threshold=0.6, min_chunk_size=200):
    # Embed every sentence
    # Split where cosine similarity drops below threshold
    # Merge tiny chunks to ensure minimum size

Key insight: The threshold matters enormously. At 0.8 I got 3281 tiny chunks that couldn't answer questions. Lowering to 0.6 produced 1123 meaningful chunks that performed much better.

Result: 1123 chunks (after tuning)
Pros: Topically coherent chunks, great for complex documents
Cons: Slow (embeds every sentence), sensitive to threshold choice

4. Hierarchical Chunking
Store small chunks for precise retrieval, but return their larger parent chunk to the LLM for rich context. Best of both worlds.

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2048, chunk_overlap=50)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)

for parent in parent_splitter.split_text(text):
    for child in child_splitter.split_text(parent):
        chunks.append({"text": child, "parent_text": parent, ...})

During retrieval, the child chunk is used to find the right section, but the parent text is returned to the LLM:

if strategy == "hierarchical" and "parent_text" in result.payload:
    text = result.payload["parent_text"]  # Return richer context
else:
    text = result.payload["text"]

Result: 1070 chunks
Pros: Precise retrieval + rich context, perfect faithfulness scores
Cons: More storage, context recall can suffer if parent chunks are too broad

Advanced RAG Techniques
Beyond chunking, I added three techniques to improve retrieval quality:

Query Rewriting
Before searching, the LLM generates 3 variations of the user's query to capture different aspects:

# Original: "What was Apple's total revenue in 2023?"
# Rewritten:
# 1. "What was Apple Inc.'s total revenue for fiscal year ending September 2023?"
# 2. "How much revenue did Apple generate during its 2023 fiscal period?"
# 3. "What is Apple's consolidated revenue for the twelve months ending 2023?"

Each variation searches the vector store independently, results are deduplicated and ranked by score.

Hybrid Search (Dense + BM25)
Combines dense vector search (semantic meaning) with BM25 keyword search (exact term matching). Financial documents have specific numbers and terminology where exact matching helps.

# Dense search score * 0.7 + BM25 score * 0.3
combined_score = dense_score * 0.7 + bm25_score * 0.3

Contextual Compression
Before passing chunks to the LLM, extract only the sentences relevant to the query. Reduces noise and token usage:

# From a 500-word chunk about Apple's products and revenue,
# extract only the 2 sentences about revenue figures

Vector Storage with Qdrant
I chose Qdrant over ChromaDB for its better performance, built-in hybrid search support, and production-readiness. Running locally via Docker:

docker run -p 6333:6333 qdrant/qdrant

Each chunking strategy gets its own collection (2048-dimensional vectors from the NVIDIA embedding model):

COLLECTION_NAMES = {
    "fixed_size": "fixed_size_collection",
    "recursive": "recursive_collection",
    "semantic": "semantic_collection",
    "hierarchical": "hierarchical_collection"
}

One lesson learned: upsert in batches of 100, not all at once. Sending 1000+ points in a single request causes connection timeouts.

Evaluation Framework
I originally planned to use RAGAS but ran into dependency conflicts with the latest version. Instead of spending hours fighting package versions, I built custom LLM-as-judge metrics — which actually gives more control and transparency.

The 4 Metrics
Faithfulness — Does the answer stick to the retrieved context, or does the model hallucinate?

Answer Relevance — Does the response actually address the question asked?

Context Precision — Of what was retrieved, how much was actually relevant?

Context Recall — Does the context contain enough information to answer the question?

Each metric prompts the LLM to return a score between 0.0 and 1.0:

def faithfulness(answer, contexts):
    context = "\n\n".join([c[:300] for c in contexts])
    prompt = f"""Given this context: {context}
And this answer: {answer}
Is the answer fully supported by the context? Reply with just a number: 1.0 for yes, 0.5 for partially, 0.0 for no."""
    return llm_score(prompt)

LangSmith Tracing
Every pipeline run — query, strategy, response, contexts, and all 4 metric scores — is logged to LangSmith automatically. This runs silently in the background and gives a full audit trail of every evaluation run.

Results
After evaluating all 4 strategies on 5 financial questions:

Strategy	Faithfulness	Ans. Relevance	Ctx. Precision	Ctx. Recall	Overall
Fixed-size	0.70	1.00	0.62	0.60	0.73
Recursive	0.70	1.00	0.89	0.60	0.80
Semantic	0.90	1.00	0.76	0.80	0.86 🏆
Hierarchical	1.00	1.00	0.69	0.54	0.81

Key Findings

Semantic chunking wins overall (0.86) — After tuning the threshold from 0.8 to 0.6, semantic chunking produced the best faithfulness (0.90) and context recall (0.80). Topically coherent chunks mean the LLM gets focused, relevant context.

Hierarchical has perfect faithfulness (1.00) — Returning parent text to the LLM means it always has rich, complete context to work with. No hallucination.

Recursive has best context precision (0.89) — Smart splitting means retrieved chunks are highly relevant to the query.

Fixed-size is weakest but simplest — Works fine as a baseline but leaves performance on the table.

Streamlit Chatbot
To make the project interactive, I built a Streamlit UI that lets you switch between chunking strategies in real time and see retrieved contexts:

strategy = st.selectbox("Chunking Strategy", 
    ["fixed_size", "recursive", "semantic", "hierarchical"])

if prompt := st.chat_input("Ask about Apple's financials..."):
    result = rag_pipeline(prompt, strategy, use_rewriting=True, use_compression=True)
    st.markdown(result["response"])

    with st.expander("Retrieved Contexts"):
        for ctx in result["contexts"]:
            st.markdown(ctx)

Run with streamlit run app.py. Try asking comparison questions like "How did iPhone revenue change between 2022 and 2023?" to see how different strategies handle multi-document retrieval.

Lessons Learned
1. The NVIDIA Nemotron model needs special handling
The model has a built-in thinking mode. Always set max_tokens=8192 and chat_template_kwargs: {"thinking": False} or you'll get None responses as the model exhausts its token budget on internal reasoning.

2. Semantic chunking threshold is critical
Threshold of 0.8 → 3281 tiny, useless chunks. Threshold of 0.6 → 1123 meaningful chunks. Always add a minimum chunk size as a guard.

3. Hierarchical chunking needs parent text for retrieval
If you retrieve child chunks but pass child text to the LLM, context recall suffers. Always return the parent text to the LLM while using the child for retrieval.

4. Batch your Qdrant upserts
Sending all vectors at once causes connection timeouts. Batch in groups of 100.

5. Build custom eval metrics when RAGAS doesn't cooperate
Dependency conflicts are real. Custom LLM-as-judge metrics are transparent, flexible, and work with any model.

6. Evaluation reveals what tuning hides
Without evaluation, I would never have caught that semantic was producing tiny useless chunks, or that hierarchical was ignoring parent text. Run eval early and often.

GitHub

Full source code available at: https://github.com/IsaacNatarajan/Advanced-RAG/

Built with NVIDIA NIM, Qdrant, LangChain, LangSmith, and Streamlit. If you found this useful, drop a ❤️ and feel free to ask questions in the comments.