DEV Community: Estevão Machado

Evolving My AI Journal: From Python MCPs to Rust Scripts and Claude Code

Estevão Machado — Sun, 17 Aug 2025 04:37:25 +0000

The Problem: Context Pollution and Portability

Over the past few months, I've been evolving my AI-powered journaling system from a simple experiment into a sophisticated tool for self-reflection and personal growth. This journey has taken me from Python-based MCP servers to Rust scripts, and from basic tool integration to leveraging the full Claude Code ecosystem. In this post, I'll share the technical evolution and the lessons learned along the way.

The Solution: Three Technical Migrations

Migration 1: Python MCPs to Rust Scripts

Finding a Real Rust Project

I've been studying Rust for a few months now, but I struggle with finding practical projects to solidify my learning. I created Trackwatch out of necessity after a friend shared the concept and my frustration with Tidal's web player limitations. However, relying heavily on AI assistance for that project left me feeling unsatisfied - I wasn't truly learning.

So I decided to rewrite my journal's markdown parsers in Rust, figuring out the indexing challenges with minimal AI help. My first professional programming work involved parsing HTML, XML, and RSS feeds to build a news database for mobile display. This felt like coming full circle - learning to parse Markdown 15 years later. The difference is stark though. Modern tooling makes it almost trivial compared to wrestling with Nokogiri back then.

You can find the complete Rust implementation here: markdown-journal-rust. For a detailed explanation of how these scripts work together with semantic search and structured metadata analysis, check out my previous post: An Architecture for Personal AI: Combining Semantic Search and Structured Metadata Analysis.

Lance DB and Embeddings

The vector database of choice was Lance DB. After researching vector databases in Rust, I found Lance DB offered the best combination of performance and ease of use. For embeddings, I upgraded from the older all-MiniLM-L6-v2 model to the newer BGE-base-en-v1.5, which provides much richer semantic representations with 768 dimensions instead of 384. Testing showed noticeably better retrieval quality - the larger model captures more nuanced relationships between concepts.

With over 5 months of journaling, I now have 150+ daily files plus countless topic-specific documents - ideas, study logs, personal reflections. Indexing is slower without GPU acceleration (haven't prioritized this yet), but queries return in under 20ms. I evolved the chunking strategy from fixed 500-character chunks with 50-character overlap to dynamic 200-2000 character chunks with no overlap. This preserves entire paragraphs, maintaining context integrity.

Migration 2: Eliminating MCP Context Pollution

When I was creating the first version of the journal, everything was novel to me. The concept of MCPs was just being understood by the public, Anthropic had just released it a few weeks prior. It felt like I was doing something incredibly smart by giving the model a focused tool it could use. And it was indeed very interesting to use for a while. But I soon started noticing some drawbacks.

First is that it's not that easy to install an MCP. Every time I wanted to change my editor, or the AI program I was using (VS Code, Cursor, Windsurf, Trae, Rovo Dev, Gemini CLI, Roo Code, Cline, Claude Code, Cursor CLI, Opencode), I had to change something about the configuration. Either the JSON fields were not the same, or it couldn't take an SSE connection and had to basically do a cold call to a python script which meant it took a very long time.

Then there's the fact that I'm writing a lot more code, not only I need the scripts to run the queries and index, but I also have to write and maintain the MCP to use those scripts. It felt unnecessary since it was a local script, and I soon realized it was.

The MCP also added a lot of pollution to the context, the calls (successful or not), the agent would sometimes fail to use it properly even with instructions in the prompt. I had the brilliant idea to simply create scripts and have the model run bash commands. This would allow me to trim the return of the scripts as much as I wanted and know exactly what was going into the context.

Migration 3: Adopting Claude Code's Ecosystem

After testing out every single model that was released on day 1, to see what better accommodates the purpose of the journal, I have settled on Claude Sonnet 4. I'm not going to go into a lot of details on its personality because you can change all that with a prompt, but the fact that it's great at tool calling is a plus. Another HUGE advantage is the Claude Code ecosystem which makes maintaining some details of the journaling a lot easier.

Claude Code Feature Integration

Slash Commands

The first Claude Code tool that made my life easier was the slash command. With it you can create a file in the commands/ folder that will be executed every time you run /nameofthefile. So I created two very basic commands /start and /commit.

The first asks Claude to check the time (as it doesn't have an internal clock) and then follow the agent rules, which is to read the previous 2 days files, my profile, and then create today's file if it doesn't exist. The second will verify the previous commits messages, run git add ., commit and push (I keep my files in a private repo to avoid losing everything like in April).

Sub Agents

When this feature was released, it opened up new possibilities for handling complex queries...

journal-rag-search

I no longer need to ask my running agent to query the database, which doesn't always return exactly what I want, still pollutes the context and sometimes needs multiple queries to get the whole story. For this I created the rag-search-agent. A sub agent that is in charge of calling the rust script, as many times as it needs to create a story about what it found and then return to my main agent exactly what it needs to know to keep the conversation going. This gives me the best results from the query return, which started with a simple MCP in the past, then evolved into the running agent calling the scripts, finally left to background agents that can run multiple queries, create a proper story and return just that, perfect context for the running agent.

journal-field-completer

My daily file is very complex with frontmatter fields at the top of the markdown, multiple fixed fields across the file that have to be filled by the model during the day as the conversation happens. Many times the model doesn't understand it's supposed to do that, and at the end of the day a lot of these fields would still be blank even though there was enough data in the file to populate everything. So I created a sub agent whose only task is to read the whole file and fill the fields with real and relevant data. When this worked, I added a line to the /commit slash command, to check the time, and if it's night time, it should run the field completer before pushing. Works every time and my files are always complete now.

weekly-retro-analyzer

This is basically a template that runs once a week and creates a retrospective of my journey in the past 7 days, which I'll discuss more below.

Status Line

This is their last released feature, it hasn't been a week since it went live. You can add a simple line under your chat box with whatever data you want. It should be a bash command, so people usually add the model they're using, the folder they're in, the branch, etc. For me, the most important thing is knowing the current context. I strongly dislike the auto-compact feature because it's a waste of tokens, and I also don't like being caught by surprise with the dreaded system message (You have x% of context left before auto-compact). This allows me to always know more or less the size of my prompts, how far along the conversation I am and plan for calling /commit or restarting the conversation.

Hooks

Hooks are a very powerful feature where you can tell Claude to do something based on a given event. I'm definitely not using its full potential. The only use case of hooks for me so far is to inject the time into every prompt, so the model no longer hallucinates how long things take. It loves adding timestamps to my conversations, even if I have a strict rule in my prompt to NOT do that. It just can't avoid... So to stop having wrong time added to each paragraph, I found out how to inject the correct time every time.

The Weekly Retrospective System

This was the latest addition to the journal. Weekly retrospective.

Thoughts Were Getting Lost

As I continue to write down my thoughts every day, it's still not possible to have the model read every single file in my project before talking to me because of context size limitations. One rule I have in my prompt is that the model should read the past two days and then we can start our conversation. This makes it so it doesn't know things that happened a few days ago. The weekly retro fixes that issue by creating a one-file snapshot of the previous week, that can be read every day in the initial prompt.

Creating Accountability

The journal has evolved from being simply a snapshot of my mind, to be something a bit more complex akin to a therapist based on my own rules. That means I have to keep track of all the things I've decided to do (studying, going to the gym, maintaining boundaries with people, saving money, staying sober...). The weekly retro will read the past week's file, the daily files for the past 7 days and write a comprehensible summary and then check if I've done everything I said I would, and what new things came up. It's easier for me too, to keep track of my own progress. I'm gamifying my existence and becoming the main character of my life by tracking the evolution.

Verifying Results

The goal of doing this is of course checking that I am on track, that no matter what happens on a day-to-day basis, I'm still doing what I said I was going to do. We can only achieve great things with repetition, creating plans is great but it only makes me anxious if I don't follow them. Now I get excited to do exactly what I said I would do because it will give me that feeling of reward at the end of the "sprint".

Real-World Application: Engineering Management

As an Engineering Manager, the weekly retro has become essential for juggling multiple responsibilities. Here's how I use it to maintain performance across different areas:

Team Leadership Tracking:

Document key points from 1-on-1s with team members
Track action items and follow-ups from previous weeks
Monitor team morale and individual developer growth
Ensure I'm providing consistent support without micromanaging

Technical Contribution Balance:

Verify I'm still contributing code despite management duties
Track my PR velocity to remain one of the top contributors in the company
Document technical decisions and architectural choices
Review code quality against my personal museum worthy standard: "Would Sandi Metz be proud of this?"

Quality Metrics:

Number of PRs submitted and merged
Code review feedback received and given
Refactoring improvements implemented
Test coverage maintained or improved

The weekly retro aggregates all this data, showing me patterns like:

Am I spending enough time coding vs. meetings?
Are my 1-on-1s actually helping my team members grow?
Is my code quality suffering when I'm context-switching too much?
Which days/times am I most productive for deep work?

This systematic review ensures I'm not just busy, but effective in both leadership and technical contribution. The journal becomes my accountability partner, reminding me when I've neglected either my team or my craft.

Conclusion

This evolution from MCPs to Rust scripts and Claude Code's ecosystem represents more than just technical improvements - it's about finding the right balance between automation and control. The journal has transformed from a simple markdown collection into an intelligent system that actively helps me maintain accountability and track progress.

The key insight? Sometimes simpler is better. Direct bash scripts provide cleaner context than MCPs, sub-agents handle complex queries elegantly, and weekly retros create the continuity that daily snapshots miss. Most importantly, building these tools in Rust gave me the practical experience I was seeking while creating something genuinely useful.

As I continue this journey, the system keeps evolving with my needs. That's the beauty of building your own tools - they grow with you.

If you're interested in building something similar or want to explore the code, check out the markdown-journal-rust repository on GitHub.

An Architecture for Personal AI: Combining Semantic Search and Structured Metadata Analysis

Estevão Machado — Sat, 28 Jun 2025 11:51:39 +0000

Abstract

Digital knowledge bases, from personal journals to enterprise wikis, often suffer from a common flaw: they become write-only archives where information is difficult to retrieve and impossible to analyze systematically. Standard keyword search fails to capture context, while valuable metadata remains siloed and unqueried. This article presents a dual-system architecture that addresses these challenges by combining Retrieval-Augmented Generation (RAG) for semantic search with structured metadata analysis. We detail this architecture, present a reference implementation built with open-source tools, and explore its broader implications for the future of local-first, personal AI systems.

The Challenge: From Data Graveyard to Dynamic Intelligence

For decades, the promise of the "personal knowledge base" has been to create a second brain—a perfect, searchable repository of our thoughts, learnings, and experiences. In reality, most attempts result in a data graveyard. The core problem is twofold:

Unstructured Data: The narrative content of our notes is rich with context, but traditional search tools are poor at understanding intent. A search for "productivity" might miss a key entry about "deep work" or "flow states."
Structured Data: Metrics we might track—mood, project status, sleep quality, code commits—are often embedded in templates or YAML frontmatter. This data is highly valuable for trend analysis but is typically invisible to the primary search function.

A truly intelligent system must be able to query both. It needs to understand the meaning of the text and analyze the metrics surrounding it.

The Dual-System Architecture

The solution lies in a composable architecture that treats unstructured and structured data as two equally important pillars. Our proposed system consists of two core components, served via the Model Context Protocol (MCP) that allows an AI model to intelligently choose the right tool for a given query.

Component 1: Semantic Search via RAG

This component addresses the unstructured data problem using vector embeddings to create a mathematical representation of the meaning of each journal entry or document.

Function: When a user asks "What have I learned about project management?", the system doesn't just look for keywords. It searches for documents that are semantically related to the query's concept.

Technology: This is achieved using a Retrieval-Augmented Generation (RAG) pipeline, which finds relevant text chunks and provides them as context to a large language model (LLM).

Component 2: Structured Data Analysis via Frontmatter

This component unlocks the potential of metadata by enforcing a simple schema within each document's YAML frontmatter, creating a queryable time-series database.

---
date: 2025-06-29
mood: 8
anxiety: 3
weight_kg: 70
sleep_hours: 7.5
exercise_minutes: 30
---

Function: This enables queries like "Chart my mood trends over the last quarter" or "Identify correlations between my sleep duration and anxiety levels."

Technology: A dedicated MCP server parses all frontmatter blocks, aggregates the data, and performs statistical analysis or generates visualizations.

Technical Architecture

Here's how the complete system works together:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│                 │    │   MCP Servers    │    │  Your Journal   │
│   Rovo Dev CLI  │◄──►│  - RAG Search    │◄──►│  - .md files    │
│                 │    │  - Frontmatter   │    │  - Frontmatter  │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Conversations  │    │  Vector Database │    │   Local Files   │
│  (Cloud/Claude) │    │    (ChromaDB)    │    │    (Private)    │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Privacy Model:

✅ Local: Journal files, search index, frontmatter analysis
☁️ Cloud: Only your conversations with the AI

This architecture provides a robust balance of privacy and performance. User data remains on the local machine by default, with only the relevant context and queries sent to cloud APIs.

Case Study: A Reference Implementation

To validate this architectural pattern, we developed a reference implementation using the following stack:

CLI Interface: Atlassian's Rovo Dev, selected for its native MCP support and access to state-of-the-art Claude models
LLM: The Claude Sonnet 4, which exhibits strong performance in conversational analysis, complex reasoning, and tool use
Open-Source Backend: Our MD RAG MCP repository, featuring two Python-based MCP servers for RAG (ChromaDB) and frontmatter analysis (Pandas)

The free tier offered by Rovo Dev (20 million tokens daily) significantly lowers the barrier to entry for developers wishing to replicate this system.

System in Action: Practical Examples

The following are real queries executed on the reference implementation, demonstrating the synergy of the dual-system architecture.

Example 1: Semantic Query

Query: "What programming languages am I studying?"

AI Response:

> what programming languages am I studying?
  └── Calling query_journal: {"query": "programming languages studying learning", "n_results": 10}

  [{'source': 'topics/tech/2025_06_06_rust_wasm_browser_game_plan.md', 'date': 'Unknown date', 'text': '## Learning...


╭─ Response ───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Based on your journal entries, you're currently studying two main programming languages:                             │
│                                                                                                                      │
│                                                                                                                      │
│                                                  Go (Primary Focus)                                                  │
│                                                                                                                      │
│  • Current Status: Actively learning through a 4-hour tutorial (completed nearly an hour)                            │
│  • Motivation: "I really like the lack of classes and hierarchy. I love Structs and composition. I really like       │
│    interfaces. I also like the := syntax"                                                                            │
│  • Learning Plan: learn Go → replace frontmatter MCP → replace personal website backend from Rails to Go             │
│  • Recent Progress: Working on slices, discovered Go's minimalist philosophy (only for loops, even for infinite      │
│    loops!)                                                                                                           │
│  • Practical Project Ideas: Blog with Go backend + SQLite, Go+htmx or Go+Svelte combinations                         │
│                                                                                                                      │
│                                                                                                                      │
│                                           Rust (Secondary/Evening Project)                                           │
│                                                                                                                      │
│  • Status: Started studying, has structured 16-week learning plan                                                    │
│  • Goal: Learn Rust → WASM → Guardian Heroes-style browser game                                                      │
│  • Learning Plan: Phase 1 (Weeks 1-3) covering fundamentals, ownership, borrowing                                    │
│  • Intermediate Project: Dashboard for frontmatter data analysis (before jumping to game development)                │
│  • Setup Complete: rustc 1.87.0, cargo 1.87.0, rustup 1.28.2 installed                                               │
│                                                                                                                      │
│ Current Challenge: You mentioned feeling like you've learned "everything I wanted to learn" recently, and these      │
│ languages feel "secondary" without urgent use cases. The main motivation now is avoiding doomscrolling during free   │
│ time since you don't play games anymore.                                                                             │
│                                                                                                                      │
│ Perfect example for the blog post! 🎯                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
  └── Calling find_and_replace_code: {"file_path": "journal/2025/06/28.md", "find": "**RAG Query Test (9:00 PM):** Te...

Example 2: Structured Data Visualization

Query: "Create an ASCII graph of my sleep hours in June"

AI Response:

Sleep Hours Per Night - June 2025
Average: 9.0 hours | Min: 6.0 hours | Max: 12.5 hours

12.5h ┤                                        ●                    
12h   ┤      ●                       ●                             
11h   ┤                        ●                                      
10.5h ┤          ●                           ●                       
10h   ┤                                          ●   ●   ● ●     
9.5h  ┤●                                                             
9h    ┤        ●           ● ●                               ●              
8.5h  ┤                                    ●       ●    ●  
8h    ┤  ●                         ●     ●                             
7.5h  ┤                          ●     ●                           
7h    ┤            ● ● ● ●                                           
6h    ┤    ●                                                         
      └┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─
       1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930

Example 3: Graphical Data Analysis

The system can also generate and save graphical plots for more complex data analysis.

Query: "Create a plot graph of my historical weight loss"

AI Response:
(Generates and saves the following image)

Implementation Guide

The open-source repository includes automated setup scripts for multiple platforms:

Step 1: Install Rovo Dev

Follow the official Atlassian Community guide for your operating system.

Step 2: Clone and Setup

git clone https://github.com/estevaom/md-rag-mcp.git
cd md-rag-mcp

# Choose your platform:
./setup_mac_environment.sh        # macOS
./setup_ubuntu_environment.sh     # Ubuntu/Debian/WSL  
./setup_arch_environment.sh       # Arch Linux

Step 3: Configure MCP Integration

acli rovodev mcp  # Opens configuration file
# Copy contents from mcp.json.template
# Replace ${PROJECT_ROOT} with your absolute project path

Step 4: Create AI Persona

Create a .agent.md file defining how the AI should interact with your journal. The repository includes guidance for creating effective therapeutic companion personas.

Step 5: Begin Journaling

acli rovodev

Your AI companion now has full access to both semantic search and structured data analysis capabilities.

Architectural Flexibility and Alternative Tooling

The power of this approach lies in the architecture, not specific tools. The MCP protocol acts as a standardized interface, allowing components to be swapped:

Alternative AI Platforms

Google Gemini CLI: Free alternative with direct Gemini model access (though weaker tool-use capabilities)
Local LLMs: Complete privacy via Ollama + Cline/Roo Code (requires significant compute resources)
Cursor IDE: Excellent for development workflows, includes Claude Sonnet 4 access with subscription

Extensibility

The MCP architecture enables easy addition of new capabilities:

Web browsing during journaling sessions
Integration with external APIs
Custom analysis tools for specific domains

Real-World Impact: Measured Results

Since implementing this system, quantifiable improvements in journaling practice include:

Consistency Metrics

Daily engagement: From sporadic entries to consistent daily journaling
Entry depth: Average entry length increased 3x due to AI-guided reflection

Pattern Recognition Capabilities

Trigger identification: Automated detection of anxiety correlations previously unnoticed
Progress tracking: Data-driven insights into personal goal advancement
Historical context: AI connects current challenges with past solutions

Actionable Intelligence

Mood correlation analysis: "Your mood tends to drop when you skip exercise"
Pattern alerts: "You've written about this challenge before - here's what worked"
Predictive insights: "Your anxiety patterns suggest you might benefit from..."

Broader Implications: The Future of Personal AI

This system exemplifies several critical trends in applied AI:

Local-First Architecture

User data sovereignty is maintained while leveraging cloud AI capabilities. This hybrid approach provides optimal balance of privacy, performance, and functionality.

Composable Intelligence

Rather than monolithic AI solutions, the future lies in specialized, interoperable tools orchestrated to solve complex problems. Each component excels in its domain while contributing to emergent system capabilities.

Democratized Access

Open-source components combined with free-tier APIs enable individual developers to build sophisticated AI systems previously available only to large organizations.

Purpose-Built Solutions

Domain-specific AI applications outperform generic chatbots by incorporating specialized knowledge, custom workflows, and targeted optimization.

Conclusion

The dual-system architecture of combined semantic search and structured metadata analysis represents a significant advancement beyond simple note-taking or basic RAG implementations. It transforms static knowledge bases into dynamic, queryable analytics engines capable of surfacing insights invisible to traditional approaches.

By understanding both narrative content and quantitative patterns within our records, we unlock deeper self-awareness, identify hidden correlations, and build truly personal AI companions for knowledge work and self-improvement.

The open-source system detailed here serves as a practical blueprint for developers and organizations pioneering this next generation of intelligent applications. The architecture is proven, the tools are accessible, and the potential for personal transformation is profound.

The future of personal AI is not about replacing human insight—it's about augmenting our capacity for self-reflection, pattern recognition, and growth. This system provides the foundation for that future, available today.

Implementation Resources:

Tags: #AI #Architecture #PersonalAI #RAG #MCP #OpenSource #LocalFirst #KnowledgeManagement

Building a Local RAG System with MCP for VS Code AI Agents: A Technical Deep Dive

Estevão Machado — Fri, 02 May 2025 08:26:38 +0000

In my previous post, I shared how I supercharged my VS Code AI agent with a local RAG (Retrieval-Augmented Generation) system using MCP (Model Context Protocol). Today, I'm following up with a detailed technical tutorial on how I built this system from scratch.

This tutorial will walk you through creating a local RAG system that indexes your Markdown journal entries and exposes semantic search capabilities to VS Code AI agents through an MCP server. By the end, you'll have a powerful system that gives your AI assistant "memory" of your past writings.

The complete code for this project is available on GitHub: https://github.com/estevaom/md-rag-mcp

Prerequisites

Python 3.8+ installed
NVIDIA GPU with CUDA support (optional but recommended for faster embeddings)
VS Code with an AI agent that supports MCP (like Cursor, Cline or Roo Code)
Basic understanding of Python and RAG concepts For Windows users, detailed instructions on setting up the environment, including WSL and CUDA, can be found in the install_instructions.md file.

Project Structure

Before diving into the code, let's understand the project structure:

/
├── README.md                 # Project overview
├── journal/                  # Journal entries
│   ├── 2025/                 # Organized by year
│   │   └── 04/               # Month
│   │       ├── 18.md         # Daily entries
│   │       └── 19.md
│   └── topics/               # Topic-based entries (future use)
├── code/                     # Code and scripts
│   ├── mcp/                  # MCP server code
│   │   └── journal_rag_mcp.py
│   ├── scripts/              # Python scripts
│   │   └── rag_search.py
│   └── data/                 # Data storage (e.g., vector DB)
│       └── chroma_db/        # Vector database (example)
└── .venv/                    # Python virtual environment

Step 1: Setting Up the Environment

First, let's create a virtual environment and install the necessary packages:

# Create a virtual environment
python -m venv .venv

# Activate the virtual environment
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install required packages
pip install sentence-transformers chromadb langchain-text-splitters python-frontmatter rich torch

If you have an NVIDIA GPU, ensure you have the appropriate CUDA toolkit installed to leverage GPU acceleration for generating embeddings.

Step 2: Building the RAG System

Let's start by implementing the core RAG functionality in code/scripts/rag_search.py.

2.1 Importing Dependencies

import os
import frontmatter  # For parsing Markdown YAML front matter
from langchain_text_splitters import RecursiveCharacterTextSplitter
from sentence_transformers import SentenceTransformer
import chromadb
import torch  # For GPU detection
from rich.console import Console  # For nice printing

# Initialize console for rich printing
console = Console()

2.2 Configuration

# Get the project root directory
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) # 3 levels up from code/scripts/rag_search.py
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "journal")
CHROMA_DB_PATH = os.path.join(PROJECT_ROOT, "data", "chroma_db")
CHROMA_COLLECTION_NAME = "life_journal_collection"
EMBEDDING_MODEL_NAME = 'all-MiniLM-L6-v2'  # Efficient, high-quality embedding model
CHUNK_SIZE = 500  # Max characters per chunk
CHUNK_OVERLAP = 50  # Characters overlap between chunks

2.3 Loading Documents

def load_markdown_docs(directory_path):
    """
    Recursively loads all Markdown files from the specified directory and its subdirectories,
    parsing YAML front matter.
    """
    documents = []
    if not os.path.isdir(directory_path):
        console.print(f"[bold red]Error: Directory not found:[/bold red] {directory_path}")
        return documents

    console.print(f"Scanning directory: [cyan]{directory_path}[/cyan]")
    found_files = False

    # Walk through directory and subdirectories
    for root, dirs, files in os.walk(directory_path):
        for filename in files:
            if filename.endswith(".md"):
                found_files = True
                filepath = os.path.join(root, filename)
                # Get relative path from the base directory for better source identification
                rel_path = os.path.relpath(filepath, directory_path)

                try:
                    post = frontmatter.load(filepath)
                    # Add filename and path to metadata for later reference
                    if 'source' not in post.metadata:
                        post.metadata['source'] = rel_path
                    documents.append(post)
                    console.print(f"  [green]Loaded:[/green] {rel_path}")
                except Exception as e:
                    console.print(f"  [bold red]Error loading {rel_path}:[/bold red] {e}")

    if not found_files:
        console.print(f"[yellow]Warning: No .md files found in {directory_path} or its subdirectories[/yellow]")

    return documents

2.4 Splitting Documents into Chunks

def split_docs(documents):
    """
    Splits the content of loaded documents into smaller chunks.
    """
    console.print("\nSplitting documents into chunks...")
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
        length_function=len,
        is_separator_regex=False,
    )

    all_chunks = []
    for doc in documents:
        chunks = text_splitter.split_text(doc.content)
        for i, chunk_text in enumerate(chunks):
            # Create a unique ID for the chunk based on source and index
            chunk_id = f"{doc.metadata.get('source', 'unknown')}_{i}"
            chunk_metadata = doc.metadata.copy()  # Start with original metadata
            chunk_metadata['chunk_index'] = i
            chunk_metadata['chunk_id'] = chunk_id  # Store ID in metadata too

            all_chunks.append({
                "id": chunk_id,  # ID for ChromaDB
                "text": chunk_text,
                "metadata": chunk_metadata
            })

    console.print(f"  Split into {len(all_chunks)} chunks.")
    return all_chunks

2.5 Initializing the Embedding Model

def initialize_embedding_model(model_name):
    """Initializes and returns the Sentence Transformer model with GPU support if available."""
    console.print(f"\nInitializing embedding model: [cyan]{model_name}[/cyan]")
    try:
        # Check if CUDA is available
        device = "cuda" if torch.cuda.is_available() else "cpu"
        if device == "cuda":
            console.print(f"  [green]CUDA is available! Using GPU: {torch.cuda.get_device_name(0)}[/green]")
        else:
            console.print("  [yellow]CUDA not available. Using CPU for embeddings (slower).[/yellow]")

        # Load model with device specification
        model = SentenceTransformer(model_name, device=device)
        console.print("  [green]Embedding model loaded successfully.[/green]")
        return model
    except Exception as e:
        console.print(f"[bold red]Error initializing embedding model:[/bold red] {e}")
        return None

2.6 Setting Up the Vector Database

def initialize_vector_store(db_path, collection_name):
    """Initializes and returns the ChromaDB client and collection."""
    console.print(f"\nInitializing vector store at: [cyan]{db_path}[/cyan]")
    try:
        chroma_client = chromadb.PersistentClient(path=db_path)

        # Get the collection if it exists, or create it if it doesn't
        collection = chroma_client.get_or_create_collection(name=collection_name)
        console.print(f"  [green]Ensured vector store collection '{collection_name}' exists.[/green]")
        return collection
    except Exception as e:
        console.print(f"[bold red]Error initializing vector store:[/bold red] {e}")
        return None

2.7 Indexing Chunks

def index_chunks(collection, chunks, embedding_model):
    """Generates embeddings and indexes chunks in the vector store."""
    if not collection or not chunks or not embedding_model:
        console.print("[bold red]Error: Cannot index chunks due to missing components.[/bold red]")
        return False

    console.print(f"\nIndexing {len(chunks)} chunks...")
    # Prepare data for ChromaDB batch insertion
    ids = [chunk['id'] for chunk in chunks]
    documents = [chunk['text'] for chunk in chunks]

    # Process metadata to ensure all values are of compatible types (str, int, float, bool)
    processed_metadatas = []
    for chunk in chunks:
        processed_metadata = {}
        for key, value in chunk['metadata'].items():
            # Convert any non-compatible types to strings
            if isinstance(value, (str, int, float, bool)):
                processed_metadata[key] = value
            else:
                processed_metadata[key] = str(value)
        processed_metadatas.append(processed_metadata)

    metadatas = processed_metadatas

    try:
        console.print("  Generating embeddings (this may take a moment)...")
        embeddings = embedding_model.encode(documents, show_progress_bar=True)
        console.print("  Embeddings generated.")

        console.print(f"  Adding {len(ids)} items to collection '{collection.name}'...")
        # Use upsert to add new or update existing chunks by ID
        collection.upsert(
            ids=ids,
            embeddings=embeddings.tolist(),  # Convert numpy array to list
            documents=documents,
            metadatas=metadatas
        )
        console.print("  [green]Chunks indexed successfully.[/green]")
        return True
    except Exception as e:
        console.print(f"[bold red]Error during indexing:[/bold red] {e}")
        return False

2.8 Querying the Vector Database

def query_rag(collection, embedding_model, query_text, n_results=3):
    """
    Queries the vector store for chunks similar to the query text.
    """
    if not collection or not embedding_model or not query_text:
        console.print("[bold red]Error: Cannot query due to missing components.[/bold red]")
        return None

    try:
        console.print(f"\n  Generating embedding for query: '{query_text}'")
        # Ensure query_text is handled correctly if empty or invalid
        if not isinstance(query_text, str) or not query_text.strip():
             console.print("[yellow]Warning: Empty query provided.[/yellow]")
             return None
        query_embedding = embedding_model.encode([query_text.strip()])  # Encode expects a list

        console.print(f"  Querying collection '{collection.name}' for {n_results} results...")
        results = collection.query(
            query_embeddings=query_embedding.tolist(),  # Convert numpy array to list
            n_results=n_results,
            include=['documents', 'metadatas', 'distances']  # Include distances for relevance check
        )
        console.print("  [green]Query successful.[/green]")
        return results
    except Exception as e:
        # Log the full exception for debugging
        import traceback
        console.print(f"[bold red]Error during query:[/bold red]\n{traceback.format_exc()}")
        return None

2.9 Main Execution

if __name__ == "__main__":
    console.print("[bold blue]=== Starting Journal RAG Search ===[/bold blue]")

    # 1. Load Documents
    loaded_docs = load_markdown_docs(DATA_DIRECTORY)
    if not loaded_docs:
        console.print("[bold red]No documents loaded. Exiting.[/bold red]")
        exit()  # Exit if loading failed

    console.print(f"\n[bold green]Successfully loaded {len(loaded_docs)} documents.[/bold green]")

    # 2. Split Documents
    doc_chunks = split_docs(loaded_docs)
    if not doc_chunks:
        console.print("[bold red]Failed to split documents into chunks. Exiting.[/bold red]")
        exit()  # Exit if splitting failed

    # 3. Initialize Models & Vector Store
    embed_model = initialize_embedding_model(EMBEDDING_MODEL_NAME)
    chroma_collection = initialize_vector_store(CHROMA_DB_PATH, CHROMA_COLLECTION_NAME)

    if not embed_model or not chroma_collection:
        console.print("[bold red]Failed to initialize models or vector store. Exiting.[/bold red]")
        exit()  # Exit if initialization failed

    # 4. Index Chunks
    indexing_successful = index_chunks(chroma_collection, doc_chunks, embed_model)

    if indexing_successful:
        console.print("\n[bold green]Indexing complete.[/bold green]")

        # 5. Interactive Querying
        console.print("\n--- Query Test ---")
        while True:
            query = input("Enter your query (or type 'quit' to exit): ")
            if query.lower() == 'quit':
                break
            if not query:
                continue

            results = query_rag(chroma_collection, embed_model, query, n_results=3)

            console.print("\n[bold magenta]Query Results:[/bold magenta]")
            if results and results.get('ids') and results['ids'] and results['ids'][0]:
                if not results.get('documents') or not results['documents'][0]:
                    console.print("  No relevant results found.")
                    continue

                for i, res_doc in enumerate(results['documents'][0]):
                    distance = results.get('distances', [[None]])[0][i]
                    metadata = results.get('metadatas', [[{}]])[0][i]
                    source = metadata.get('source', 'N/A')

                    distance_str = f"{distance:.4f}" if distance is not None else "N/A"
                    console.print(f"  [cyan]Result {i+1} (Distance: {distance_str}):[/cyan]")
                    console.print(f"    [dim]Source: {source}[/dim]")
                    console.print(f"    {res_doc}")
            elif results is None:
                console.print("  An error occurred during the query.")
            else:
                console.print("  No relevant results found.")

    else:
        console.print("[bold red]Indexing failed. Check errors above.[/bold red]")

    console.print("\n[bold blue]=== Journal RAG Search Finished ===[/bold blue]")

Step 3: Creating the MCP Server

Now, let's implement the MCP server that will expose our RAG system to VS Code AI agents. Create a file at code/mcp/journal_rag_mcp.py:

3.1 Imports and Configuration

#!/usr/bin/env python3
import sys
import json
import os
import time
import traceback
import frontmatter

# --- Early Startup Logging ---
print("[MCP Server Debug] Script starting up", file=sys.stderr, flush=True)
print(f"[MCP Server Debug] Python version: {sys.version}", file=sys.stderr, flush=True)
print(f"[MCP Server Debug] Current working directory: {os.getcwd()}", file=sys.stderr, flush=True)
print(f"[MCP Server Debug] Script directory: {os.path.dirname(os.path.abspath(__file__))}", file=sys.stderr, flush=True)

try:
    import chromadb
    from sentence_transformers import SentenceTransformer
    import torch
except ImportError as e:
    print(json.dumps({
        "jsonrpc": "2.0",
        "error": {
            "code": -32000,  # Server error
            "message": f"Missing required Python package: {e}. Please install chromadb, sentence-transformers, and torch.",
        }
    }), file=sys.stderr)
    sys.exit(1)

# --- Configuration ---
# Calculate PROJECT_ROOT based on this script's location (code/mcp/journal_rag_mcp.py)
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_ROOT = os.path.dirname(os.path.dirname(SCRIPT_DIR)) # 2 levels up

CHROMA_DB_PATH = os.path.join(PROJECT_ROOT, "data", "chroma_db")
CHROMA_COLLECTION_NAME = "life_journal_collection"
CHROMA_COLLECTION_NAME = "life_journal_collection"
EMBEDDING_MODEL_NAME = 'all-MiniLM-L6-v2'
DEFAULT_N_RESULTS = 3

# --- Global Variables ---
embedding_model = None
chroma_collection = None
is_initialized = False

3.2 Logging and Initialization

# --- Logging ---
def log_error(message):
    """Logs an error message to stderr."""
    print(f"[MCP Server Error] {message}", file=sys.stderr, flush=True)

def log_info(message):
    """Logs an info message to stderr."""
    print(f"[MCP Server Info] {message}", file=sys.stderr, flush=True)

# --- Initialization Function ---
def initialize_resources():
    """Loads the embedding model and connects to ChromaDB."""
    global embedding_model, chroma_collection, is_initialized
    if is_initialized:
        return True

    log_info("Initializing resources...")
    try:
        # 1. Initialize Embedding Model
        log_info(f"Loading embedding model: {EMBEDDING_MODEL_NAME}")
        device = "cuda" if torch.cuda.is_available() else "cpu"
        log_info(f"Using device: {device}")
        embedding_model = SentenceTransformer(EMBEDDING_MODEL_NAME, device=device)
        log_info("Embedding model loaded.")

        # 2. Initialize ChromaDB
        log_info(f"Connecting to ChromaDB at: {CHROMA_DB_PATH}")
        if not os.path.exists(CHROMA_DB_PATH):
             log_info(f"ChromaDB path does not exist: {CHROMA_DB_PATH}. Creating it now.")
             try:
                 os.makedirs(CHROMA_DB_PATH, exist_ok=True)
                 log_info(f"Created ChromaDB directory: {CHROMA_DB_PATH}")
             except Exception as e:
                 log_error(f"Failed to create ChromaDB directory: {e}")
                 log_error(traceback.format_exc())
                 return False  # Indicate initialization failure

        chroma_client = chromadb.PersistentClient(path=CHROMA_DB_PATH)
        log_info(f"Getting/Creating collection: {CHROMA_COLLECTION_NAME}")
        chroma_collection = chroma_client.get_or_create_collection(name=CHROMA_COLLECTION_NAME)
        log_info("ChromaDB collection ready.")

        is_initialized = True
        log_info("Resources initialized successfully.")
        return True

    except Exception as e:
        log_error(f"Initialization failed: {e}")
        log_error(traceback.format_exc())
        embedding_model = None
        chroma_collection = None
        is_initialized = False
        return False

3.3 Timestamp Tracking for Incremental Updates

# --- Timestamp Tracking Functions ---
def get_last_indexed_time():
    """Gets the timestamp of the last indexing operation."""
    timestamp_file = os.path.join(PROJECT_ROOT, "data", "last_indexed_time.txt")

    if not os.path.exists(timestamp_file):
        return 0  # Return 0 if file doesn't exist (never indexed)

    try:
        with open(timestamp_file, 'r') as f:
            timestamp = float(f.read().strip())
        return timestamp
    except:
        return 0  # Return 0 if there's an error reading the file

def update_last_indexed_time():
    """Updates the timestamp of the last indexing operation to current time."""
    timestamp_file = os.path.join(PROJECT_ROOT, "data", "last_indexed_time.txt")

    try:
        # Create directory if it doesn't exist
        timestamp_dir = os.path.dirname(timestamp_file)
        log_info(f"Ensuring directory exists: {timestamp_dir}")
        os.makedirs(timestamp_dir, exist_ok=True)

        # Write current timestamp
        current_time = time.time()
        log_info(f"Writing timestamp {current_time} to {timestamp_file}")
        with open(timestamp_file, 'w') as f:
            f.write(str(current_time))
        log_info(f"Timestamp updated successfully")
        return True
    except Exception as e:
        log_error(f"Error updating timestamp: {e}")
        log_error(traceback.format_exc())
        return False

def load_markdown_docs_since(directory_path, timestamp):
    """
    Loads only markdown files that have been modified since the given timestamp.
    """
    documents = []
    if not os.path.isdir(directory_path):
        log_error(f"Error: Directory not found: {directory_path}")
        return documents

    log_info(f"Scanning directory for modified files: {directory_path}")
    found_files = False

    # Walk through directory and subdirectories
    for root, dirs, files in os.walk(directory_path):
        for filename in files:
            if filename.endswith(".md"):
                filepath = os.path.join(root, filename)

                # Check if file was modified after the timestamp
                if os.path.getmtime(filepath) > timestamp:
                    found_files = True
                    # Get relative path from the base directory for better source identification
                    rel_path = os.path.relpath(filepath, directory_path)

                    try:
                        post = frontmatter.load(filepath)
                        # Add filename and path to metadata for later reference
                        if 'source' not in post.metadata:
                            post.metadata['source'] = rel_path
                        documents.append(post)
                        log_info(f"  Loaded modified file: {rel_path}")
                    except Exception as e:
                        log_error(f"  Error loading {rel_path}: {e}")

    if not found_files:
        log_info(f"No modified .md files found since {time.ctime(timestamp)}")

    return documents

3.4 Index Update Function

# --- Index Update Function ---
def update_journal_index(full_reindex=False):
    """
    Updates the journal index with new or modified entries.
    """
    try:
        # Import functions from rag_search.py
        sys.path.append(os.path.join(PROJECT_ROOT, "code", "scripts"))
        try:
            from rag_search import (
                load_markdown_docs, split_docs, initialize_embedding_model,
                initialize_vector_store, index_chunks
            )
        except ImportError as e:
            log_error(f"Error importing from rag_search.py: {e}")
            log_error(f"sys.path: {sys.path}")
            return False, f"Error importing from rag_search.py: {e}"

        # Define data directory (same as in rag_search.py)
        DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "journal")

        # Track stats for reporting
        files_processed = 0
        chunks_indexed = 0

        # Load documents (all or only new/modified)
        if full_reindex:
            # Process all documents
            log_info("Performing full reindex of all journal entries")
            documents = load_markdown_docs(DATA_DIRECTORY)
            files_processed = len(documents)
        else:
            # Get last indexing timestamp
            last_indexed_time = get_last_indexed_time()
            log_info(f"Performing incremental index update since {time.ctime(last_indexed_time)}")

            # Process only new or modified documents
            documents = load_markdown_docs_since(DATA_DIRECTORY, last_indexed_time)
            files_processed = len(documents)

            if not documents:
                return True, "No new or modified journal entries found."

        log_info(f"Found {files_processed} files to process")

        # Split documents into chunks
        chunks = split_docs(documents)
        chunks_indexed = len(chunks)

        if not chunks:
            return True, f"No chunks generated from {files_processed} files."

        log_info(f"Split into {chunks_indexed} chunks")

        # Initialize embedding model and vector store
        embed_model = initialize_embedding_model(EMBEDDING_MODEL_NAME)
        chroma_collection = initialize_vector_store(CHROMA_DB_PATH, CHROMA_COLLECTION_NAME)

        if not embed_model or not chroma_collection:
            return False, "Failed to initialize embedding model or vector store."

        # Index chunks
        log_info(f"Indexing {chunks_indexed} chunks")
        indexing_successful = index_chunks(chroma_collection, chunks, embed_model)

        if indexing_successful:
            # Update last indexed time
            update_last_indexed_time()
            return True, f"Successfully indexed {chunks_indexed} chunks from {files_processed} files."
        else:
            return False, "Indexing failed."

    except Exception as e:
        log_error(f"Error updating index: {str(e)}")
        log_error(traceback.format_exc())
        return False, f"Error updating index: {str(e)}"

3.5 RAG Query Function

# --- RAG Query Function ---
def perform_rag_query(query_text: str, n_results: int = DEFAULT_N_RESULTS):
    """Executes a semantic search query."""
    if not is_initialized or embedding_model is None or chroma_collection is None:
        log_error("Query attempted before resources were initialized.")
        return None, "Resources not initialized"

    try:
        log_info(f"Encoding query: '{query_text}'")
        query_embedding = embedding_model.encode([query_text.strip()])

        log_info(f"Querying collection '{chroma_collection.name}' for {n_results} results...")
        results = chroma_collection.query(
            query_embeddings=query_embedding.tolist(),
            n_results=n_results,
            include=['documents', 'metadatas', 'distances']
        )
        log_info("Query successful.")

        # Format results
        formatted_results = []
        if results and results.get('ids') and results['ids'][0]:
            for i, doc_text in enumerate(results['documents'][0]):
                metadata = results['metadatas'][0][i] if results['metadatas'] and results['metadatas'][0] else {}
                distance = results['distances'][0][i] if results['distances'] and results['distances'][0] else None
                formatted_results.append({
                    "source": metadata.get('source', 'N/A'),
                    "text": doc_text,
                    "distance": distance
                })
        return formatted_results, None  # Results, No error message

    except Exception as e:
        log_error(f"Error during query: {e}")
        log_error(traceback.format_exc())
        return None, str(e)  # No results, Error message

3.6 MCP Response Formatting

# --- MCP Response Formatting ---
def create_mcp_response(request_id, result=None, error=None):
    """Creates a JSON-RPC 2.0 response dictionary."""
    response = {"jsonrpc": "2.0", "id": request_id}
    if error:
        response["error"] = error
    else:
        response["result"] = result
    return response

def create_mcp_error(code, message):
    """Creates an MCP error object."""
    return {"code": code, "message": message}

3.7 Main Server Loop

# --- Main Server Loop ---
def main():
    """Reads MCP requests from stdin, processes them, writes responses to stdout."""
    if not initialize_resources():
        # Send an error response if init fails
        log_error("Exiting due to initialization failure.")
        sys.exit(1)  # Exit if resources can't be loaded

    log_info("MCP server started. Waiting for requests on stdin...")

    for line in sys.stdin:
        request_id = None  # Reset for each request
        try:
            log_info(f"Received request line: {line.strip()}")
            request = json.loads(line)
            request_id = request.get("id")
            method = request.get("method")
            params = request.get("params", {})
            log_info(f"Processing method: {method} for request ID: {request_id}")

            response = None

            if method == "initialize":
                log_info("Handling 'initialize' method.")
                # Handle the initial handshake from the AI agent
                response = create_mcp_response(request_id, result={
                    "protocolVersion": "2024-11-05",  # Specify MCP protocol version
                    "serverInfo": {
                        "name": "journal-rag-mcp-server",  # Required server name
                        "displayName": "Journal RAG Server",
                        "version": "0.1.0",
                    },
                    "capabilities": {
                        "tools": {
                            "listChanged": False  # We don't support notifications for tool list changes
                        }
                    }
                })
            elif method == "notifications/initialized":
                # This is just a notification, no response needed
                log_info("Received initialized notification")
                response = None  # No response needed for notifications

            elif method == "tools/list":
                log_info("Handling 'tools/list' method.")
                response = create_mcp_response(request_id, result={
                    "tools": [
                        {
                            "name": "query_journal",
                            "description": "Queries the personal journal entries using semantic search.",
                            "inputSchema": {
                                "type": "object",
                                "properties": {
                                    "query": {
                                        "type": "string",
                                        "description": "The search query text."
                                    },
                                    "n_results": {
                                        "type": "number",
                                        "description": f"Number of results to return (default: {DEFAULT_N_RESULTS}).",
                                        "minimum": 1
                                    }
                                },
                                "required": ["query"]
                            }
                        },
                        {
                            "name": "update_index",
                            "description": "Updates the journal index with new or modified entries",
                            "inputSchema": {
                                "type": "object",
                                "properties": {
                                    "full_reindex": {
                                        "type": "boolean",
                                        "description": "Whether to reindex all journal entries (true) or only new/modified ones (false)",
                                        "default": false
                                    }
                                }
                            }
                        }
                    ]
                })

            elif method == "resources/list":
                log_info("Handling 'resources/list' method.")
                # Return empty list if we don't have resources
                response = create_mcp_response(request_id, result={
                    "resources": []
                })

            elif method == "resources/templates/list":
                log_info("Handling 'resources/templates/list' method.")
                # Return empty list if we don't have resource templates
                response = create_mcp_response(request_id, result={
                    "templates": []
                })

            elif method == "tools/call":
                log_info("Handling 'tools/call' method.")
                tool_name = params.get("name")
                arguments = params.get("arguments", {})
                log_info(f"Tool name: {tool_name}, Arguments: {arguments}")

                if tool_name == "query_journal":
                    query = arguments.get("query")
                    n_results = arguments.get("n_results", DEFAULT_N_RESULTS)

                    if not query or not isinstance(query, str):
                        response = create_mcp_response(request_id, error=create_mcp_error(-32602, "Invalid params: 'query' argument is missing or not a string."))
                    elif not isinstance(n_results, int) or n_results < 1:
                         response = create_mcp_response(request_id, error=create_mcp_error(-32602, f"Invalid params: 'n_results' must be a positive integer (default: {DEFAULT_N_RESULTS})."))
                    else:
                        results, error_msg = perform_rag_query(query, n_results)
                        if error_msg:
                            response = create_mcp_response(request_id, error=create_mcp_error(-32000, f"Query execution failed: {error_msg}"))
                        else:
                            # Embed the list of results as a JSON string within the text content
                            response = create_mcp_response(request_id, result={
                                "content": [{
                                    "type": "text",
                                    "text": json.dumps(results, indent=2) # Pretty print for readability if needed
                                }]
                            })
                elif tool_name == "update_index":
                    full_reindex = arguments.get("full_reindex", False)
                    log_info(f"Updating index with full_reindex={full_reindex}")

                    success, message = update_journal_index(full_reindex)

                    if success:
                        response = create_mcp_response(request_id, result={
                            "content": [{
                                "type": "text",
                                "text": message
                            }]
                        })
                    else:
                        response = create_mcp_response(
                            request_id, 
                            error=create_mcp_error(-32000, f"Index update failed: {message}")
                        )
                else:
                    log_error(f"Unknown tool called: {tool_name}")
                    response = create_mcp_response(request_id, error=create_mcp_error(-32601, f"Method not found: Unknown tool '{tool_name}'"))

            else:
                 log_error(f"Unknown method received: {method}")
                 response = create_mcp_response(request_id, error=create_mcp_error(-32601, f"Method not found: Unknown method '{method}'"))

            if response:
                response_json = json.dumps(response)
                log_info(f"Prepared response for ID {request_id}: {response_json[:200]}...") # Log truncated response
                print(response_json, flush=True)
                log_info(f"Successfully sent response for request ID: {request_id}")
            elif method == "notifications/initialized":
                # This is expected for notifications that don't require responses
                log_info(f"No response needed for notification: {method}")
            else:
                log_error(f"No response generated for request ID: {request_id}, method: {method}")

        except json.JSONDecodeError:
            log_error(f"JSONDecodeError: Failed to decode JSON request: {line.strip()}")
            error_response = create_mcp_response(request_id, error=create_mcp_error(-32700, "Parse error: Invalid JSON received."))
            print(json.dumps(error_response), flush=True)
        except Exception as e:
            log_error(f"Unexpected error processing request: {e}")
            log_error(traceback.format_exc())
            error_response = create_mcp_response(request_id, error=create_mcp_error(-32000, f"Internal server error: {e}"))
            print(json.dumps(error_response), flush=True)

# --- Script Execution Guard ---
if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        log_info("Server stopped by KeyboardInterrupt.")
    except Exception as e:
        log_error(f"Unhandled exception in main execution: {e}")
        log_error(traceback.format_exc())
        sys.exit(1) # Ensure non-zero exit code on unhandled error
    finally:
        log_info("Server process ending.")

Step 4: Integrating with VS Code

Now that we have our RAG system and MCP server implemented, we need to integrate it with VS Code. This involves configuring VS Code to recognize and connect to our MCP server.

4.1 Configure the MCP Server in VS Code Settings

To connect VS Code to your Journal RAG MCP server, you need to add its configuration to your VS Code settings. Open your VS Code settings (File > Preferences > Settings) and search for "MCP Servers". Add the following JSON configuration:

{
  "mcpServers": {
    "journal-rag-mcp": {
      "command": "${workspaceFolder}/.venv/bin/python3",
      "args": [
        "./code/mcp/journal_rag_mcp.py" # Path relative to workspaceFolder
      ],
      "env": {},
      "disabled": false,
      "alwaysAllow": [],
      "autoApprove": [
        "query_journal"
      ]
    }
  }
}

This configuration tells VS Code how to start your MCP server. The ${workspaceFolder} variable is a built-in VS Code variable that resolves to the path of your currently open workspace folder (which should be the project root).

After adding this configuration, restart VS Code for the changes to take effect. Your AI agent should now be able to detect and interact with the "journal-rag-mcp" server.
This configuration tells VS Code's AI assistant (like Cline or Roo Code) to connect to our MCP server when it starts up.

Step 5: Testing and Usage

Now that everything is set up, let's test our RAG system and MCP server.

5.1 Running the RAG System Standalone

You can run the RAG system standalone to test indexing and querying:

# Activate the virtual environment
source .venv/bin/activate

# Run the RAG search script
python code/scripts/rag_search.py

This will index all your journal entries and allow you to interactively query them.

5.2 Using the MCP Server with VS Code

Open VS Code in your project directory
Start a conversation with your AI assistant (Cline or Roo Code)
The MCP server will automatically start and connect
You can now ask your AI assistant questions about your journal entries

Example queries:

"What did I write about anxiety last month?"
"Summarize my thoughts on my work project from last week"
"Find entries where I discussed my relationship with my mother"

The AI assistant will use the query_journal tool to retrieve relevant information from your journal entries and provide a response based on that information.

Step 6: Advanced Features

6.1 Incremental Indexing

Our implementation includes incremental indexing, which only processes files that have been modified since the last indexing operation. This makes the indexing process much faster when you only add or modify a few files.

To trigger an incremental update:

Update the rag-mcp index.

Okay, I will update the journal index now.

Cline wants to use a tool on the journal-rag-mcp MCP server:
update_index
Updates the journal index with new or modified entries
Arguments
{
  "full_reindex": false
}

6.2 GPU Acceleration

If you have an NVIDIA GPU, our implementation automatically uses it for generating embeddings, which significantly speeds up the indexing and querying process. The code detects if CUDA is available and uses it if possible.

Minimum Requirements

CPU: Any modern multi-core processor (Intel i5/i7 or AMD Ryzen 5/7 from the last few years)
RAM: 8GB minimum, 16GB recommended for smoother performance
Storage: ~500MB for the embedding model + additional space for the vector database (varies based on journal size, but typically a few hundred MB)
OS: Any OS that supports Python 3.8+ (Windows, macOS, Linux)

Optional but Recommended

GPU: NVIDIA GPU with CUDA support (GTX 1060 6GB or better)
- The system automatically detects and uses CUDA if available
- GPU acceleration significantly speeds up embedding generation during indexing and querying
- Without a GPU, the system will use CPU which works fine but is slower

Performance Notes

The system uses the 'all-MiniLM-L6-v2' embedding model, which is relatively lightweight
Indexing performance depends on:
- Number and size of journal entries
- Whether you're using GPU or CPU
- For reference, on my system with an RTX 4090, indexing 500+ chunks takes just a few seconds
- On CPU only, expect indexing to take longer (potentially minutes for large journals)
Query performance is generally fast even on CPU (typically under a second)

Scaling Considerations

For very large journals (thousands of entries), consider:
- More RAM (32GB+)
- Faster storage (SSD)
- More powerful GPU (if available)

The system is designed to be efficient and should run reasonably well even on modest hardware. The GPU is helpful but not required - it just makes the indexing and querying processes faster.

Conclusion

In this tutorial, we've built a powerful local RAG system that integrates with VS Code AI agents through the Model Context Protocol. This system allows your AI assistant to have "memory" of your past journal entries, making it much more effective as a reflective journaling partner.

The key components we've implemented are:

A RAG system that indexes Markdown journal entries and provides semantic search capabilities
An MCP server that exposes the RAG system to VS Code AI agents
Integration with VS Code to connect the AI agent to our MCP server

This implementation is just the beginning. You can extend it in many ways, such as:

Adding more sophisticated chunking strategies
Implementing metadata filtering (e.g., by date, tags, or mood)
Creating a dashboard to visualize journal metrics
Adding support for other file formats

I hope this tutorial helps you build your own local RAG system and supercharge your VS Code AI agent!

Supercharging My VS Code AI Agent with Local RAG

Estevão Machado — Sun, 20 Apr 2025 08:46:33 +0000

My AI Agent Needed a Better Memory

Working with AI assistants directly in my IDE (like Cline or Roo Code in VS Code) has become a core part of my workflow, especially for personal reflection and journaling – what I call "Project Life". The agent is great at capturing my thoughts in the moment, saving them neatly into dated Markdown files. But a crucial piece was missing: efficient access to the past.

When I asked my AI agent, "What did I write about managing anxiety last month?", it faced a dilemma:

Manual File Reading? It could try to list and read relevant files, but that's slow, clunky, and requires complex instructions.
Massive Context Window? We could try stuffing weeks or months of journal entries into the prompt, but LLM context windows have limits, costs escalate quickly, and performance degrades. Sending the entire history frequently is inefficient and costly.
Lost Context: Long-running conversations eventually lose track of earlier details.

I needed a way for my agent to instantly retrieve relevant historical context without these drawbacks. I wanted it to feel like it had a memory of my past entries, specifically tailored to my current query.

The Solution: Local RAG Accessed via MCP

The answer lay in combining two powerful concepts:

Retrieval-Augmented Generation (RAG): Instead of relying solely on the prompt, RAG retrieves relevant information from an external knowledge base (my journal files!) and adds it to the prompt before sending it to the LLM.
Model Context Protocol (MCP): A standardized way for my VS Code AI agent to communicate with other local tools and services.

The plan: Build a local RAG service that indexes my Markdown journal files and expose its querying capability as a tool via an MCP server. My VS Code agent could then call this tool whenever I asked about past entries.

The Workflow: Giving My Agent Instant Recall

Here’s how it works in practice:

Journaling: I talk to my AI agent in VS Code. It captures my thoughts and saves them to journal/YYYY/MM/DD.md.
Indexing: A local RAG service (journal-rag-mcp) monitors the journal directory (or runs on demand) and updates its index. It reads Markdown files, splits them into chunks, generates vector embeddings using sentence-transformers, and stores them in a local ChromaDB vector database.
Querying the Past: I ask the agent, "What were my main concerns during the project planning phase?"
Agent Uses Tool: The agent recognizes this requires historical context. Instead of reading files or using a huge prompt, it makes a single, targeted call to the journal-rag-mcp server using the query_journal tool via MCP, sending my query ("project planning phase concerns").
RAG Server Works: The MCP server receives the request, embeds the query, searches its ChromaDB index for the most semantically similar text chunks from my past entries, and gathers the top few results.
Context Returned: The MCP server sends only these relevant text snippets back to the AI agent as the tool's result.
Informed Response: The agent now has the specific, relevant context needed. It combines my original question and these retrieved snippets into a concise prompt for the LLM (e.g., Gemini). The LLM generates a response grounded in my actual past writings.

The key benefit: The agent gets the historical context it needs in one efficient request, without manual file handling or bloated prompts.

Building the RAG MCP Service

The core components are:

Markdown Files: The source of truth.
Python: The language for the server.
sentence-transformers: For creating text embeddings locally (using all-MiniLM-L6-v2 and GPU acceleration via PyTorch/CUDA).
ChromaDB: A local vector database for storing and querying embeddings.
python-frontmatter: To parse metadata if needed (though the focus here is the text content).
MCP Server Script (journal_rag_mcp.py): A Python script that:
- Initializes the embedding model and ChromaDB connection.
- Listens for incoming MCP requests via standard input/output (stdio).
- Defines tools like query_journal and update_index.
- Handles query_journal requests by embedding the query, searching ChromaDB, and returning results.
- Handles update_index requests to keep the vector store fresh.

Here's a conceptual look at the MCP tool definition:

# Conceptual MCP tool definition within the server script
QUERY_TOOL_SCHEMA = {
    "name": "query_journal",
    "description": "Queries personal journal entries using semantic search.",
    "inputSchema": {
        "type": "object",
        "properties": {
             "query": {"type": "string", "description": "The search query text."}
        },
        "required": ["query"]
    }
}
# Server logic maps requests matching this schema to the RAG query function.

The build involved setting up the Python environment (.venv), installing dependencies, and configuring the MCP server in VS Code's settings. Debugging involved fixing dependency issues (frontmatter), path misconfigurations, Python NameErrors (false vs False), and ensuring the index update mechanism worked correctly.

Architecture: Agent -> MCP -> RAG

This diagram shows the primary data flows:

+-----------------+                    +-----------------+
| Journal Files   | <--(Save Data)---- | VS Code AI Agent| -- User Interaction --> (LLM)
| (.md)           |                    | (e.g.,Cline/Roo)|
+-------+---------+                    +-------+---------+
        |                                        | Tool Call (query_journal)
        | Indexing                               v
        v                                +-----------------+
+-----------------+                      | journal-rag-mcp |
| Vector Store    | <-- Search Query --- | (MCP Server)    |
| (ChromaDB)      |                      +-----------------+
+-----------------+ --- Results ------->   (Agent uses Results)
                       (Chunks)

(Note: Indexing reads Journal Files to update the Vector Store, typically as a background or on-demand process)

Current Status & Future

The journal-rag-mcp server is running reliably. My VS Code AI agent can now use the query_journal tool to instantly retrieve relevant context from my past entries. This significantly enhances its usefulness as a reflective journaling partner.

Future plans include:

Building the Metrics Parser and Dashboard to visualize data from YAML front matter.
Creating a dedicated AI Chat Interface outside of the IDE.
Refining the RAG process (e.g., experimenting with different models or chunking strategies).

Conclusion: Efficient, Relevant AI Memory

Combining a capable AI agent with specialized local tools like this RAG server via MCP is incredibly powerful. It provides an efficient, relevant, and scalable way to give AI assistants access to personal knowledge bases, sending only pertinent snippets to the LLM instead of the entire history.

Is this the perfect solution? Honestly, no. In an ideal world, the AI would always have the full conversation history instantly available. But current technology has limitations. My earlier experiments involved conversations with staggering context lengths – think over 400 million input tokens and 800k context windows using experimental models. While powerful, the latency became unbearable (often over 30 seconds per response), and the cost implications were significant. RAG offers a pragmatic compromise today.

While the retrieved snippets are sent to the external LLM API for generation, the core index remains private and local, and the overall process is far more targeted and cost-effective than relying solely on massive context windows.

What truly excites me is how accessible building such tools has become. Creating an MCP server, especially with frameworks or assistance from IDE agents like Roo/Cline, is surprisingly straightforward. Debugging Python code with an LLM partner turns potential roadblocks into learning opportunities. Given this landscape, there's immense potential for developers to build personalized software that genuinely improves their lives.

For anyone building personal AI tools or struggling with context window limitations, I highly recommend exploring local RAG and protocols like MCP. It puts you in control of the retrieval process and unlocks new possibilities for how AI can interact with your data efficiently.