Chat with Your Docs: Working with LangChain, RAG, MCP, ChromaDB Made Simple

Have you ever wished you could ask questions from a pile of documents, spreadsheets, or PDFs sitting on your machine and magically get answered? It's not magic and in this guide, we will build exactly that: a local Q&A system that lets we chat with your documents using AI techniques and tools.

We will combine Retrieval Augmented Generation (RAG) with the Model Context Protocol (MCP) to create a modular, production-ready pipeline. By the end of this article, we will have a working system composed of three services: a document ingestion server, an MCP tool server, and a conversational AI LangChain-powered agent.

What We Will Build

• Build a powerful Q&A system from documents.
• Understand the core concepts of RAG architecture.
• Set up and interact with ChromaDB vector store.
• Implement data loading using LangChain document loaders.
• Stream responses for a great user experience.
• Integrate Retrieval Augmented Generation (RAG).

About Me

I am David Archanjo, a Full-Stack Engineer with 10+ years of experience in the technology industry. I am passionate about software development and AI Engineering, and I enjoy sharing practical knowledge through technical articles that help developers build real-world applications.

---

Table Of Contents

• Background
• Why RAG and MCP?
• Step 1: Setting Up the Environment
• Step 2: Loading and Indexing Documents into ChromaDB
• Step 3: Exposing MCP tools
• Step 4: Building the RAG chain
• Step 5: Running and Testing the Q&A Agent
• Wrapping Up

---

Background

Large Language Models (LLMs) excel at reasoning and text generation, but their knowledge is limited to what they were trained on. They cannot access our private documents, internal company data, or any information not included in their training, which makes them unable to answer questions about our knowledge base without additional context.

Retrieval-Augmented Generation (RAG) addresses this limitation by retrieving more relevant information, for instance, from our documents at query time and providing it to the LLM as context, enabling grounded responses. In this guide, we expose that retrieval layer as a Model Context Protocol (MCP) tool, creating a reusable service that any MCP-compatible agent or application can leverage.

---

Why RAG and MCP?

The Case for RAG

Without RAG, an LLM can only answer questions based on what it learned during training.

It cannot:

• Answer questions about our private or proprietary documents.
• Reflect updates that happened after its training cutoff.
• Cite specific sources or passages to back up its answers.

RAG solves these problems by combining semantic search with LLM reasoning. Our documents are first split into small, semantically meaningful chunks, converted into embeddings (vector representations of their meaning), and stored in a vector database. When a user submits a question, the query is embedded into the same vector space, allowing the system to retrieve the most relevant chunks based on semantic similarity rather than keyword matching. Those retrieved passages are then injected into the LLM's prompt as context, enabling the model to generate responses that are grounded in our data, more accurate, and less prone to hallucinations.

Key benefits of RAG:

• Accuracy: answers come from our actual documents.
• Transparency: we can trace every answer back to a specific source.
• Cost efficiency: we only process what is relevant to the question.
• Freshness: new documents can be added at any time without retraining the model.

The Case for MCP

The Model Context Protocol is an open standard that defines how AI agents discover and invoke external tools.

By exposing our document search capability as an MCP tool, we gain several architectural advantages:

• Modularity: the retrieval service is completely decoupled from the agent.
• Reusability: the same MCP server can serve multiple agents or applications simultaneously.
• Discoverability: agents can automatically discover available tools and their descriptions at runtime.
• Standardization: an open protocol decouples our integration from any single framework, reducing vendor lock-in and ensuring long-term interoperability.

Combining RAG with MCP gives the best of both worlds: the accuracy and contextual grounding of retrieval-augmented generation with the flexibility, interoperability, and maintainability of a standardized protocol for integrating tools and data sources.

---

Step 1: Setting Up the Environment

Before writing any code, let us get the project structure in place and install the required dependencies.

Project Structure

.
├── .env                        # Environment variables
├── .db/                        # ChromaDB persistent storage
├── agent.py                    # Conversational RAG agent
├── assets/                     # Sample documents for testing
│   ├── car_prices.csv
│   ├── company_credentials.txt
│   ├── meeting_notes.md
│   └── resume.pdf
├── config.py                   # Centralized configuration
├── doc_ingestion_server.py     # Document ingestion pipeline & API
├── doc_mcp_server.py           # MCP tool server
├── docs/                       # Watched document directory
└── requirements.txt            # Python dependencies

NOTE: You can find these sample documents in the accompanying GitHub repository.

Our project is organized around three separate services, each with a clear responsibility:

Service	File	Responsibility
Ingestion Server	doc_ingestion_server.py	Load, chunk, embed, and store documents
MCP Tool Server	doc_mcp_server.py	Expose documents' tools via MCP protocol
AI Agent	agent.py	Accept user queries and orchestrate retrieval

Installing Dependencies

1. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

1. Create a requirements.txt file and add the following:

fastapi>=0.139.2
fastmcp>=3.4.4
langchain>=1.3.14
langchain-chroma>=1.1.0
langchain-community>=0.4.2
langchain-mcp-adapters>=0.3.0
langchain-openai>=1.4.0
pypdf>=6.14.2
pydantic-settings
python-dotenv

1. Install the dependencies:

pip install -r requirements.txt

Environment Configuration

Create a .env at the root of the project with the following contents:

DOCS_DIR="./docs"
CHROMA_DIR="./.db"
CHROMA_COLLECTION_NAME="documents"
EMBEDDING_MODEL_NAME="mxbai-embed-large-v1"
EMBEDDING_MODEL_BASE_URL="http://localhost:8001/v1"
CHUNK_SIZE=500
CHUNK_OVERLAP=50
LLM_API_KEY="sk-xxxxx"
LLM_BASE_URL="http://localhost:8000/v1"
LLM_MODEL_NAME="gpt-4o"
LLM_MODEL_PROVIDER="openai"
LLM_TEMPERATURE=0.2
MCP_SERVER_NAME="document-search"
MCP_SERVER_URL="http://localhost:3001/mcp"

Breaking down each variable:

Variable	Purpose
DOCS_DIR	Directory where uploaded documents are stored
CHROMA_DIR	Persistent storage directory for ChromaDB
CHROMA_COLLECTION_NAME	Name of the ChromaDB collection
EMBEDDING_MODEL_NAME	Name of the embedding model
EMBEDDING_MODEL_BASE_URL	Base URL for the embedding model API
CHUNK_SIZE	Maximum number of characters per document chunk
CHUNK_OVERLAP	Overlap in characters between adjacent chunks
LLM_API_KEY	API key for the language model
LLM_BASE_URL	Base URL for the LLM API
LLM_MODEL_NAME	Name of the language model to use
LLM_MODEL_PROVIDER	Name of the language model's provider (e.g. openai, anthropic, bedrock)
LLM_TEMPERATURE	Controls the creativity of model responses
MCP_SERVER_NAME	Identifier for the MCP server
MCP_SERVER_URL	URL where the MCP server is reachable

Local OpenAI-Compatible API Service

In this article, we will interact with language and embedding models through an OpenAI-compatible API. LangChain's integrations can communicate with any server implementing this interface. Luckily this API does not require the official OpenAI service, so any server implementing the same HTTP interface can be used transparently.

We will run both the chat model and the embedding model locally. I recommend using llamafile, a standalone distribution of llama.cpp, as this approach keeps the entire stack running on our own machine while allowing the application code to remain identical to what we would write for the official OpenAI API.

The following llamafile models are recommended:

Purpose	Llamafile Model
Chat model	Qwen3.5-2B-Q8_0.llamafile
Embedding model	mxbai-embed-large-v1-f16.llamafile

Start the embedding model:

./mxbai-embed-large-v1-f16.llamafile \
  --server \
  --host 0.0.0.0 \
  --port 8001 \
  --nobrowser \
  --embedding \
  --alias mxbai-embed-large-v1

In another terminal, start the chat model :

./Qwen3.5-2B-Q8_0.llamafile \
  --server \
  --host 0.0.0.0 \
  --port 8000 \
  --alias gpt-4o

The --alias gpt-4o option exposes our local language model under the name gpt-4o. From LangChain's perspective, it behaves exactly like if it was connected to the OpenAI GPT-4o model, allowing us to work with the same LLM initialization setup.

NOTE: If you prefer using the official OpenAI service, simply replace LLM_API_KEY with your own API key, and update EMBEDDING_MODEL_BASE_URL if your embedding service is running elsewhere, or remove EMBEDDING_MODEL_BASE_URL if you're using OpenAI embeddings. No other change is required.

Why llamafile? I personally recommend llamafile because it packages llama.cpp, the model weights, and all required runtime components into a single executable. There is no installation, daemon service, package manager, or additional Python environment to maintain. We simply download the file, make it executable, and run it. Since it is built on top of llama.cpp, it inherits excellent performance while remaining portable across Linux, macOS and Windows.

The Configuration Class

# config.py
from pydantic_settings import BaseSettings

class EnvironmentConfiguration(BaseSettings):
    DOCS_DIR: str
    CHROMA_DIR: str
    CHROMA_COLLECTION_NAME: str
    EMBEDDING_MODEL_NAME: str
    EMBEDDING_MODEL_BASE_URL: str
    CHUNK_SIZE: int
    CHUNK_OVERLAP: int
    LLM_API_KEY: str
    LLM_BASE_URL: str
    LLM_MODEL_NAME: str
    LLM_MODEL_PROVIDER: str
    LLM_TEMPERATURE: float
    MCP_SERVER_NAME: str
    MCP_SERVER_URL: str

    class Config:
        env_file = ".env"

The config.py file extends BaseSettings class from pydantic-settings to load all environment variables automatically from the .env file. By centralizing configuration this way, every service (i.e. doc_ingestion_server.py, doc_mcp_server.py and agent.py) imports the same EnvironmentConfiguration class and receives consistent, validated settings.

---

Step 2: Loading and Indexing Documents into ChromaDB

The ingestion pipeline is the foundation of our RAG system. It is responsible for reading raw documents, splitting them into chunks, converting those chunks into vector embeddings, and persisting everything in ChromaDB so the MCP server can query it later.

The full implementation will live in doc_ingestion_server.py, which is built as a FastAPI web server so we can upload new documents via API request at runtime without having to restart the server.

"""
Document ingestion pipeline & server.

Start the server:
    uvicorn doc_ingestion_server:app --host 0.0.0.0 --port 3002

The ingestion pipeline is automatically executed once during server startup.

Upload new documents through:
    POST /documents
"""

import sys
import logging
import shutil
from config import EnvironmentConfiguration
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, UploadFile
from langchain_chroma import Chroma
from langchain_community.document_loaders import (
    CSVLoader,
    DirectoryLoader,
    TextLoader,
    PyPDFLoader
)
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from pathlib import Path


ALLOWED_FILE_EXTENSIONS = {".csv", ".txt", ".md", ".pdf"}

config = EnvironmentConfiguration()

logger = logging.getLogger("uvicorn.error")

embeddings = OpenAIEmbeddings(
    model=config.EMBEDDING_MODEL_NAME,
    base_url=config.EMBEDDING_MODEL_BASE_URL,
    check_embedding_ctx_length=False
)


def split_documents(loader):
    documents = loader.load()
    if not documents:
        return []

    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=config.CHUNK_SIZE,
        chunk_overlap=config.CHUNK_OVERLAP,
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    chunks = text_splitter.split_documents(documents)
    logger.info("Loaded %d document(s) → %d chunk(s).", len(documents), len(chunks))
    return chunks


def embed_and_write(chunks):
    if len(chunks) == 0:
        Chroma(
            collection_name=config.CHROMA_COLLECTION_NAME,
            persist_directory=config.CHROMA_DIR,
            embedding_function=embeddings
        )
        return

    Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        collection_name=config.CHROMA_COLLECTION_NAME,
        persist_directory=config.CHROMA_DIR,
    )
    logger.info("Ingestion complete. %d chunks indexed into ChromaDB.", len(chunks))


def ingest():
    logger.info("Starting ingestion pipeline.")

    loader = DirectoryLoader(
        config.DOCS_DIR,
        glob=["**/*.txt", "**/*.md"],
        loader_cls=TextLoader,
        loader_kwargs={"encoding": "utf-8"}
    )
    chunks = split_documents(loader)

    csv_loader = DirectoryLoader(
        config.DOCS_DIR,
        glob="**/*.csv",
        loader_cls=CSVLoader,
        loader_kwargs={"encoding": "utf-8"}
    )
    chunks.extend(split_documents(csv_loader))

    pdf_loader = DirectoryLoader(
        config.DOCS_DIR,
        glob="**/*.pdf",
        loader_cls=PyPDFLoader
    )
    chunks.extend(split_documents(pdf_loader))

    embed_and_write(chunks)


@asynccontextmanager
async def lifespan(app: FastAPI):
    try:
        Path(config.DOCS_DIR).mkdir(parents=True, exist_ok=True)
        ingest()
        yield
    except Exception as e:
        logger.critical("Failed to initialize: %s", e)
        sys.exit(1)


app = FastAPI(title="Document Ingestion Server", lifespan=lifespan)


@app.post("/documents")
async def upload_document(file: UploadFile):
    if not file.filename:
        raise HTTPException(status_code=400, detail="Filename is required.")
    extension = Path(file.filename).suffix.lower()
    if extension not in ALLOWED_FILE_EXTENSIONS:
        raise HTTPException(status_code=400, detail=f"Only '{ALLOWED_FILE_EXTENSIONS}' files are supported.")

    destination = Path(config.DOCS_DIR) / file.filename
    with destination.open("wb") as buffer:
        shutil.copyfileobj(file.file, buffer)

    try:
        if extension == ".csv":
            loader = CSVLoader(str(destination), encoding="utf-8")
        elif extension == ".pdf":
            loader = PyPDFLoader(str(destination))
        else:
            loader = TextLoader(str(destination), encoding="utf-8")
        chunks = split_documents(loader)
        embed_and_write(chunks)
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"File saved but ingestion failed: {e}")

    return {"message": "Document uploaded and ingested successfully.", "filename": file.filename}

Breaking Down the Code

1. Allowed File Types

ALLOWED_FILE_EXTENSIONS = {".csv", ".txt", ".md", ".pdf"}

We restrict ingestion to four file formats. Each format has a dedicated LangChain's community loader which understands its structure:

• CSVLoader parses rows and columns
• PyPDFLoader extracts text page by page
• TextLoader handles plain text and Markdown files

1. Embeddings Initialization

embeddings = OpenAIEmbeddings(
    model=config.EMBEDDING_MODEL_NAME,
    base_url=config.EMBEDDING_MODEL_BASE_URL,
    check_embedding_ctx_length=False
)

We initialize an OpenAIEmbeddings instance pointing to our locally hosted embedding model (configured via EMBEDDING_MODEL_BASE_URL). Notice we have check_embedding_ctx_length=False settings. This specific setting disables the token-length guard which is required when working with locally served models that may not report context limits the same way the OpenAI API does.

1. The split_documents Function

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=config.CHUNK_SIZE,
    chunk_overlap=config.CHUNK_OVERLAP,
    separators=["\n\n", "\n", ". ", " ", ""],
)

This is the heart of the chunking strategy. RecursiveCharacterTextSplitter tries to split text at natural boundaries, i.e. first at double newlines (paragraph breaks), then single newlines, then sentence boundaries, and so on, working down the list of separators until the resulting chunk is within the configured CHUNK_SIZE (500 by default).

The CHUNK_OVERLAP (50 characters by default) ensures that context is not lost at chunk boundaries. If a sentence spans two chunks, the tail of the first chunk appears at the head of the second, so retrieval can always return complete thoughts.

1. The embed_and_write Function

Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    collection_name=config.CHROMA_COLLECTION_NAME,
    persist_directory=config.CHROMA_DIR,
)

Chroma.from_documents takes the list of chunks, calls the embedding model to convert each chunk into a dense vector, and writes both the vectors and the original text to disk under CHROMA_DIR. ChromaDB automatically persists this data, so the MCP server can load it later without re-embedding anything.

When the chunk list is empty (no documents found during startup), we initialize an empty ChromaDB collection as fallback. This prevents errors downstream when the MCP server tries to connect before any documents have been uploaded.

1. The ingest Function

def ingest():
    # Text and Markdown
    loader = DirectoryLoader(config.DOCS_DIR, glob=["**/*.txt", "**/*.md"], ...)
    chunks = split_documents(loader)

    # CSV
    csv_loader = DirectoryLoader(config.DOCS_DIR, glob="**/*.csv", ...)
    chunks.extend(split_documents(csv_loader))

    # PDF
    pdf_loader = DirectoryLoader(config.DOCS_DIR, glob="**/*.pdf", ...)
    chunks.extend(split_documents(pdf_loader))

    embed_and_write(chunks)

The ingest function runs three separate DirectoryLoader passes: one per document family because each file type requires a different loader class. All resulting chunks are collected into a single list before being written to ChromaDB in one batch, which is more efficient than writing each file type separately, at least for the purpose of this guide.

1. The FastAPI Lifespan and Upload Endpoint

@asynccontextmanager
async def lifespan(app: FastAPI):
    Path(config.DOCS_DIR).mkdir(parents=True, exist_ok=True)
    ingest()
    yield

FastAPI's lifespan context manager runs ingest() exactly once at startup. This means that if documents already exist in DOCS_DIR from a previous run, they are re-indexed automatically. No manual intervention is required.

The POST /documents endpoint handles runtime uploads. A user can submit a new file and it will be chunked and embedded immediately, making it searchable without restarting either service. upload_document processes the file upload by selecting the appropriate loader class based on the file extension.

Ingestion Server Startup

uvicorn doc_ingestion_server:app --host 0.0.0.0 --port 3002

Once running, we can upload new documents like this:

curl -X POST http://localhost:3002/documents -F "file=@assets/meeting_notes.md"

---

Step 3: Exposing MCP Tools

Now with our documents properly indexed in ChromaDB, we need to equip our AI agent with tool calling capability. This is where the Model Context Protocol (MCP) comes in. Instead of hard-coding retrieval logic inside the agent (though this is valid in some context), we publish it as a discoverable, decoupled, protocol-compliant tool which any MCP-compatible client can use.

The full implementation will live in doc_mcp_server.py, which is also built as a FastAPI web server.

"""
FastMCP ASGI server exposing a local document search service.

Start the server:
    uvicorn doc_mcp_server:app --host 0.0.0.0 --port 3001
"""

import logging
from config import EnvironmentConfiguration
from pathlib import Path
from fastmcp import FastMCP
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from typing import Any

config = EnvironmentConfiguration()
logger = logging.getLogger("uvicorn.error")


def create_app():
    if not Path(config.CHROMA_DIR).exists():
        raise RuntimeError(
            f"Chroma database '{config.CHROMA_DIR}' does not exist.\n"
            "Run doc_ingestion_server.py first."
        )

    embeddings = OpenAIEmbeddings(
        model=config.EMBEDDING_MODEL_NAME,
        base_url=config.EMBEDDING_MODEL_BASE_URL,
        check_embedding_ctx_length=False,
    )

    vector_store = Chroma(
        collection_name=config.CHROMA_COLLECTION_NAME,
        persist_directory=config.CHROMA_DIR,
        embedding_function=embeddings,
    )

    mcp = FastMCP(config.MCP_SERVER_NAME)

    @mcp.tool
    def search_documents(query: str) -> list[dict[str, Any]]:
        """
        Search the indexed knowledge base using semantic similarity.
        ...
        """
        logger.info("Executing tool: search_documents ['%s']", query)
        try:
            retriever = vector_store.as_retriever(search_kwargs={"k": 3})
            documents = retriever.invoke(query)
            return [
                {"metadata": doc.metadata, "content": doc.page_content}
                for doc in documents
            ] if documents else []
        except Exception as ex:
            logger.error("search_documents failed: %s", ex)
            return []

    @mcp.tool
    def list_documents() -> list[str]:
        """
        Return the names of every indexed document.
        ...
        """
        logger.info("Executing tool: list_documents")
        data = vector_store.get(include=["metadatas"])
        sources = set()
        for metadata in (data.get("metadatas") or []):
            if metadata and "source" in metadata:
                sources.add(Path(metadata["source"]).name)
        return sorted(sources)

    return mcp.http_app()


app = create_app()

Breaking Down the Code

1. The Application Factory Pattern

def create_app():
    ...
    return mcp.http_app()

app = create_app()

We wrap the entire server setup in a create_app factory function rather than initializing everything at module level. This pattern makes it easy to test the server in isolation, swap out dependencies, or run multiple configurations side by side. The factory runs all initialization eagerly at import time, i.e. if ChromaDB is missing, the error is raised immediately during startup rather than at the first tool invocation.

1. Safety Guard at Startup

if not Path(config.CHROMA_DIR).exists():
    raise RuntimeError(
        f"Chroma database '{config.CHROMA_DIR}' does not exist.\n"
        "Run doc_ingestion.py first."
    )

This guard ensures the MCP server refuses to start if the vector database has not been created yet.

1. The search_documents Tool

@mcp.tool
def search_documents(query: str) -> list[dict[str, Any]]:
    retriever = vector_store.as_retriever(search_kwargs={"k": 3})
    documents = retriever.invoke(query)
    return [
        {"metadata": doc.metadata, "content": doc.page_content}
        for doc in documents
    ] if documents else []

The @mcp.tool decorator registers the function as a discoverable MCP tool. FastMCP automatically reads the function's docstring, type annotations, and parameter names to generate the tool's schema. Our AI agent uses this schema to understand when and how to call the tool.

Internally, as_retriever(search_kwargs={"k": 3}) converts the vector store into a LangChain retriever that returns the top 3 most semantically similar chunks for any query. The similarity is computed using cosine similarity between the query's embedding vector and all stored chunk vectors.

Each result is serialized as a plain dictionary containing two fields:

• metadata: information like the source file path and page number, which the agent can use to cite its sources.
• content: the actual text of the chunk, which becomes the context for the LLM's answer.

1. The list_documents Tool

@mcp.tool
def list_documents() -> list[str]:
    data = vector_store.get(include=["metadatas"])
    sources = set()
    for metadata in (data.get("metadatas") or []):
        if metadata and "source" in metadata:
            sources.add(Path(metadata["source"]).name)
    return sorted(sources)

This tool addresses a common user need: "What documents do you know about?" or "How many documents do I have in my knowledge base?". Rather than searching by content, it scans all stored metadata records, extracts the source field from each, and returns a deduplicated, sorted list of file names.

Using here a set for deduplication is important because every chunk from the same document carries the same source metadata. For example, A 50-page PDF might produce hundreds of chunks, but list_documents should report it as a single entry.

Running the MCP Server

uvicorn doc_mcp_server:app --host 0.0.0.0 --port 3001

The MCP server is now reachable at http://localhost:3001/mcp. Any MCP-compatible client — including our agent — can connect, discover the two tools, and start invoking them.

Testing the MCP Server

Before integrating our document MCP server with our AI agent, it's useful to verify that it is working correctly. The easiest way to do this is with the official MCP Inspector, which provides an web application for connecting to our server, discovering available tools, and invoking them manually.

• We can permanently install it globally:

npm install -g @modelcontextprotocol/inspector

• Or, we can run it directly without installing it:

npx @modelcontextprotocol/inspector

Once the Inspector opens in the default browser, connect it to the document MCP server. Now we can initialize a session, list the available tools as well as execute one of them (i.e. search_documents or list_documents).

---

Step 4: Building the RAG Chain

With both the ingestion server and the MCP tool server running, we can now build the agent that ties everything together. The agent accepts natural language questions, decides which MCP tools to call, interprets the retrieved documents, and produces grounded answers.

The full implementation is contained in agent.py, leveraging LangChain to orchestrate the workflow.

import asyncio
from config import EnvironmentConfiguration
from langchain.agents import create_agent
from langchain.chat_models import init_chat_model
from langchain_mcp_adapters.client import MultiServerMCPClient

config = EnvironmentConfiguration()


async def run_agent():
    print(f"📡 Connecting to the MCP Server [{config.MCP_SERVER_NAME}] at [{config.MCP_SERVER_URL}]...")
    client = MultiServerMCPClient(connections={
        config.MCP_SERVER_NAME: {"transport": "http", "url": config.MCP_SERVER_URL}
    })

    try:
        print("📥 Fetching and registering MCP tools...")
        discovered_tools = await client.get_tools()
        print(f"✅ Successfully registered {len(discovered_tools)} tools:")
        tool_map = {tool.name: tool for tool in discovered_tools}
        for name in tool_map.keys():
            print(f"   ➡️ [Discovered] Name: '{name}'")

        llm = init_chat_model(
            api_key=config.LLM_API_KEY
            base_url=config.LLM_BASE_URL,
            model=config.LLM_MODEL_NAME,
            model_provider=config.LLM_MODEL_PROVIDER,
            temperature=config.LLM_TEMPERATURE
        )

        system_prompt = """
        You are a document retrieval assistant.

        Whenever the user asks a question whose answer may exist in the document store:

        1. Call the document search tool.
        2. Examine the retrieved documents.
        3. Answer exclusively from the retrieved content.
        4. If no relevant documents are found, reply:
          "I couldn't find that information in your documents."

        Never skip the search step.
        Never answer using your own knowledge when the answer should come from the document store.
        Do not fabricate information.
        Keep responses concise and factual.
        """

        agent_executor = create_agent(model=llm, tools=discovered_tools, system_prompt=system_prompt)

        session_state = {"messages": []}
        print("-" * 50)

        while True:
            user_prompt = await asyncio.to_thread(input, "\n👨 User Query ('quit' to exit): ")

            if user_prompt.strip().lower() == "quit":
                print("\n👋 Terminating session. Goodbye!")
                break

            if not user_prompt.strip():
                continue

            session_state["messages"].append(("user", user_prompt))
            result = await agent_executor.ainvoke(session_state)
            session_state["messages"] = result["messages"]

            print(f"\n🤖 Agent Answer: {session_state['messages'][-1].content}")
            print("-" * 50)

    except Exception as e:
        print(f"❌ Error encountered during client execution: {e}")


if __name__ == "__main__":
    asyncio.run(run_agent())

Breaking Down the Code

1. Connecting to the MCP Server

client = MultiServerMCPClient(connections={
    config.MCP_SERVER_NAME: {"transport": "http", "url": config.MCP_SERVER_URL}
})
discovered_tools = await client.get_tools()

MultiServerMCPClient acts as a universal MCP client. It connects to one or more MCP servers, fetches their tool schemas, and wraps each tool as a native LangChain BaseTool object that the agent can invoke directly.

The transport: "http" setting tells the client to communicate over standard HTTP, which is the same transport our doc_mcp_server.py is getting served. This makes the integration completely network-transparent: the agent, the MCP server, and the ingestion server can all run on different machines/environments — that's the beauty of decoupled architecture and HTTP-based communication.

1. Configuring the Language Model

llm = init_chat_model(
    api_key=config.LLM_API_KEY
    base_url=config.LLM_BASE_URL,
    model=config.LLM_MODEL_NAME,
    model_provider=config.LLM_MODEL_PROVIDER,
    temperature=config.LLM_TEMPERATURE
)

We use LangChain's init_chat_model factory function to create the chat model. The LLM_API_KEY authenticates requests, while LLM_MODEL_PROVIDER, LLM_BASE_URL, and LLM_MODEL_NAME configure the provider, endpoint, and model to use. With LLM_TEMPERATURE=0.2, the model produces focused, deterministic responses, making it well suited for our RAG application.

1. The System Prompt

system_prompt = """
You are a document retrieval assistant.

Whenever the user asks a question whose answer may exist in the document store:
1. Call the document search tool.
2. Examine the retrieved documents.
3. Answer exclusively from the retrieved content.
4. If no relevant documents are found, reply:
  "I couldn't find that information in your documents."

Never skip the search step.
Never answer using your own knowledge when the answer should come from the document store.
Do not fabricate information.
"""

The system prompt is the most important part of an agent's configuration. Without explicit instructions, an LLM naturally relies on its pre-trained knowledge instead of the retrieved context. This completely defeats the purpose of RAG.

The four numbered rules enforce a strict retrieve-then-answer workflow. They require searching the document store before answering and prohibit relying on the model's internal knowledge. The fallback response prevents hallucinations when the requested information is not found.

1. The Agent Executor

agent_executor = create_agent(model=llm, tools=discovered_tools, system_prompt=system_prompt)

create_agent assembles a ReAct-style agent that can iteratively call tools and reason about the results before producing a final answer. The agent receives the full list of discovered MCP tools and uses the LLM to decide, at each step, whether to call a tool or respond directly.

1. Maintaining Conversation History

session_state = {"messages": []}

session_state["messages"].append(("user", user_prompt))
result = await agent_executor.ainvoke(session_state)
session_state["messages"] = result["messages"]

The agent maintains a running list of messages in session_state. After each turn, the full updated message history (including tool calls and their results) is stored back into session_state. This gives the agent memory across turns as it can answer follow-up questions that reference earlier exchanges without losing context.

1. Asynchronous Input Handling

user_prompt = await asyncio.to_thread(input, "\n👨 User Query ('quit' to exit): ")

Python's built-in input() function is blocking, i.e. it would freeze the entire async event loop while waiting for the user to type. asyncio.to_thread offloads the blocking call to a thread pool, keeping the event loop free to process ongoing async operations (like the agent's tool calls) concurrently.

---

Step 5: Running and Testing the Q&A Agent

Now that we walked through all three services implementation, it's time to start them up and put the system through its paces.

Starting All Services

Let's open three separate terminals and run the following commands, in this order:

1. Terminal 1 (Ingestion Server):

uvicorn doc_ingestion_server:app --host 0.0.0.0 --port 3002

1. Terminal 2 (MCP Tool Server):

uvicorn doc_mcp_server:app --host 0.0.0.0 --port 3001

1. Terminal 3 (Agent):

python agent.py

The order matters because the ingestion server must run first to create the ChromaDB database. The MCP tool server must start second because it reads from that database on startup. The agent can then connect to the live MCP tool server.

Uploading Sample Documents

Before querying, upload the sample files from the assets/ directory using the ingestion server's REST endpoint POST /documents:

curl -X POST http://localhost:3002/documents -F "file=@assets/meeting_notes.md"
curl -X POST http://localhost:3002/documents -F "file=@assets/car_prices.csv"
curl -X POST http://localhost:3002/documents -F "file=@assets/resume.pdf"
curl -X POST http://localhost:3002/documents -F "file=@assets/company_credentials.txt"

Each upload triggers the full pipeline: file saved → chunked → embedded → written to ChromaDB. You will see confirmation logs in Terminal 1 like this:

INFO:     New document upload: 'meeting_notes.md'.
INFO:     Loaded 1 document(s) → 5 chunk(s).
INFO:     Ingestion complete. 5 chunks indexed into ChromaDB.
INFO:     Success document upload: 'meeting_notes.md'.

Sample Questions for Testing the Knowledge Base

General questions

• How many documents do I have in my knowledge base?

assets/company_credentials.txt:

• What is my Jira login credential?
• How often should I change my passwords based on my company rules?
• What is the URL for my company's GitHub Enterprise instance?
• What is my personal AWS Account ID?

assets/meeting_notes.md

• What specific tasks was the QA team assigned across all the strategy meetings?
• How did the engineering team's database priorities change between July and September?
• What decisions were made at the Q1 strategy meeting?

assets/resume.pdf

• Do we have any candidates whose first name is John?
• What degree does John Doe hold?
• Which university did John Doe attend, and were there any academic honors or scholarships?
• Breakdown John Doe's experiences and see if he speaks Portuguese.

assets/car_prices.csv

• List the cheapest car by brands.
• What is the price of a red Honda with high horsepower?
• What is the most expensive Chevrolet option?
• How many white cars are there for each brand?

Agent Example Interaction

Once the agent is running, we can interact with the agent. A sample session looks like this:

📡 Connecting to the MCP Server [document-search] at [http://localhost:3001/mcp]...
📥 Fetching and registering MCP tools...
✅ Successfully registered 2 tools:
   ➡️ [Discovered] Name: 'search_documents'
   ➡️ [Discovered] Name: 'list_documents'

--------------------------------------------------

👨 User Query ('quit' to exit): What documents do I have indexed?

🤖 Agent Answer: You have the following documents indexed:

*   car_prices.csv
*   company_credentials.txt
*   meeting_notes.md
*   resume.pdf

--------------------------------------------------

👨 User Query ('quit' to exit): What were the key decisions made in Q2?

🤖 Agent Answer: Based on the meeting notes, here are the key decisions made in Q2:

1.  **Data Quality**: Added automated validation for ingested documents.
2.  **UX Improvements**: Improved search results by adding feedback when low-confidence results are returned.
3.  **Operations**: Deployed a new monitoring dashboard to track vector store health.

--------------------------------------------------

👨 User Query ('quit' to exit): quit

👋 Terminating session. Goodbye!

Verifying Retrieval Quality

If the agent's answers seem off or incomplete, here are the main levers to adjust:

Parameter	Location	Effect
CHUNK_SIZE	.env	Larger chunks preserve more context per result; smaller chunks improve precision
CHUNK_OVERLAP	.env	More overlap reduces information loss at boundaries
LLM_TEMPERATURE	.env	Lower values produce more conservative, factual answers
k in search_kwargs	doc_mcp_server.py	Retrieve more chunks for complex questions; fewer for focused ones

After changing chunk size or overlap, re-run the ingestion server to re-index all documents with the new settings.

The Full Execution Flow

Wrapping Up

In this guide, we built a complete document Q&A system powered by LangChain, RAG, ChromaDB, and the Model Context Protocol (MCP). By separating document ingestion, semantic retrieval, and conversational orchestration into independent services, we created a modular architecture that is easy to extend, maintain, and reuse.

Although the implementation runs entirely on local, open-source models through llamafile, the same application can target any OpenAI-compatible service by changing only the configuration. From here, you can continue expanding the system by adding more document loaders, improving retrieval strategies, or exposing additional capabilities as MCP tools.

BTW, this guide has an accompanying GitHub repository, making it easy to run, modify, and extend on your own.

Happy AI coding! 🚀