RAG as an MCP Server: How I Let Claude Code Search My Own Documents

This article is for AI developers who want Claude Code to answer questions from their own documents. Claude Code can already read files in your project, but it opens them one by one and reads them as a whole. If you have a folder with many long PDFs, for example reports or manuals, that costs a lot of tokens, and Claude has to guess which file to open first. A search over all documents that only returns the matching passages works much better.

I already had this search in my RAG chatbot. I mainly built the MCP server to understand how MCP works in practice, and I was surprised how little I had to change. Below, I show how I made the search of my RAG pipeline available to Claude Code with a small MCP server, how to connect it and how I test it.

What is MCP?

MCP (Model Context Protocol) is an open protocol that lets AI agents use tools from a server. The server describes its tools, and the client, for example Claude Code, decides when to call them. A good example is a Gmail MCP server. When Claude is connected to it, it can search your emails or create drafts in your Gmail account for you.

The big advantage over your own REST API is that you build the server once and every MCP client can use it, like Claude Code, Claude Desktop or Cursor. With a REST API, you would have to write a separate integration for every client.

My server uses the transport Streamable HTTP instead of stdio. With stdio, every client starts its own copy of the server. With HTTP, the server runs separately, and several clients can connect to the same server.

Design decisions

Before writing any code, I made four decisions.

The server calls no LLM. It only returns the passages it found. The client already has a model, so the user decides which model answers the question. This also means the server needs no API keys and costs nothing per request.

It has its own knowledge base. In my chatbot, users upload files for each chat. The MCP server should always be ready, so it reads a folder of documents once and stores them on disk with Chroma.

Two tools are enough. One tool searches the documents, and the other one shows which files are in the knowledge base. These are the two things a client really needs, so I didn't add more.

Tool	Returns
search_documents(query)	the most relevant passages with file name and page
list_documents()	the files in the knowledge base

The search code is the same as in the chatbot. The agent and the MCP server both call the same function, search_vectorstore. So both return the same results, and I only have to test the search once.

Step 1: Ingest a folder of documents

The ingest script reads all PDF, TXT and Markdown files in a folder, splits them into chunks, small pieces of about 500 characters, and stores them in the knowledge base.

def load_folder(folder: str) -> list[Document]:
    """Load all PDF, TXT and Markdown files in the folder, tagged with their file name."""
    documents = PyPDFDirectoryLoader(folder, recursive=True).load()
    documents += DirectoryLoader(
        folder, glob=["*.txt", "*.md"], recursive=True, loader_cls=TextLoader
    ).load()
    for page in documents:
        page.metadata["source"] = Path(page.metadata["source"]).name
    return documents


if __name__ == "__main__":
    pages = load_folder(sys.argv[1])
    shutil.rmtree(KNOWLEDGE_BASE_DIR, ignore_errors=True)
    Chroma.from_documents(
        RecursiveCharacterTextSplitter(
            chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP
        ).split_documents(pages),
        HuggingFaceEmbeddings(model_name=EMBEDDING_MODEL),
        persist_directory=KNOWLEDGE_BASE_DIR,
        collection_metadata={"hnsw:space": "cosine"},
    )
    print(f"Ingested {sorted({p.metadata['source'] for p in pages})}")

uv run python -m mcp_server.ingest path/to/documents

load_folder() loads the PDFs page by page, so every chunk keeps its page number. I only keep the file name instead of the whole path. That way, the source in the answer stays short, and the client doesn't see any paths from my computer.

Every run deletes the old knowledge base with shutil.rmtree and builds it again from scratch. So files you removed from the folder also disappear from the search. This is much simpler than comparing which files changed.

The embeddings come from BAAI/bge-base-en-v1.5, which runs locally with Hugging Face, so ingesting the documents doesn't need an API key either.

Step 2: The MCP server

The whole server has less than 40 lines. Without the imports, it looks like this.

mcp = FastMCP("knowledge-base", host=MCP_HOST, port=MCP_PORT)


@cache
def get_knowledge_base() -> Chroma:
    """Open the knowledge base built by mcp_server.ingest on the first tool call."""
    return Chroma(
        persist_directory=KNOWLEDGE_BASE_DIR,
        embedding_function=HuggingFaceEmbeddings(model_name=EMBEDDING_MODEL),
        collection_metadata={"hnsw:space": "cosine"},
    )


@mcp.tool()
def search_documents(query: str) -> str:
    """Search the company knowledge base and return the most relevant passages with file name and page."""
    return search_vectorstore(get_knowledge_base(), query)


@mcp.tool()
def list_documents() -> list[str]:
    """List the files that are in the knowledge base."""
    sources = {m["source"] for m in get_knowledge_base().get()["metadatas"]}
    return sorted(sources)


if __name__ == "__main__":
    mcp.run(transport="streamable-http")

FastMCP from the official MCP Python SDK does most of the work. @mcp.tool() turns a normal Python function into an MCP tool. The type hints become the input schema, and the docstring becomes the description the model sees.

The docstring is more important than it looks. Claude only sees the name and the description of a tool and decides based on them when to call it. "Search the company knowledge base …" tells it clearly that this tool is meant for questions about documents.

@cache on get_knowledge_base() makes sure that the embedding model and the database are only loaded on the first tool call and reused after that. So the server starts quickly, and later searches don't load the model again.

This is the search function that the chatbot and the MCP server share.

def search_vectorstore(vectorstore: VectorStore, query: str) -> str:
    """Return the most relevant chunks with file and page, shared by the agent and the MCP server."""
    results = vectorstore.similarity_search_with_relevance_scores(query, k=5)
    relevant = [doc for doc, score in results if score >= SIMILARITY_THRESHOLD]

    if not relevant:
        return NO_DOCUMENTS_FOUND

    output = []
    for doc in relevant:
        source = doc.metadata.get("source", "Unknown")
        page = doc.metadata.get("page", None)
        ref = f"[{source}" + (f", page {page + 1}]" if page is not None else "]")
        output.append(f"{ref}\n{doc.page_content}")

    return "Found in documents:\n\n" + "\n\n".join(output)

It takes the 5 most similar chunks, drops every chunk with a similarity score below 0.2 and adds the file name and page to each passage. The score shows how close the meaning of the chunk is to the question, and 1.0 would mean the same meaning. PDF loaders count pages from 0, so page + 1 turns them into the page numbers a reader sees. If nothing is left, it returns NO_DOCUMENTS_FOUND.

Step 3: Connect Claude Code

Start the server and add it to Claude Code with one command.

uv run python -m mcp_server.server   # serves http://127.0.0.1:8001/mcp
claude mcp add --transport http knowledge-base http://127.0.0.1:8001/mcp

For the example, I ingested a sample report about a solar park, the same PDF I use for the evals of my chatbot, and asked Claude Code which bank provided the loan.

At first, Claude didn't call the tool and offered to search the web instead. It only sees the description "Search the company knowledge base …", so it doesn't know that this report is in there. When I added "Search my documents." to the question, it worked.

❯ Search my documents. Which bank provided the loan for the Nordheide Solar Park and at what interest rate?

  Called knowledge-base

⏺ Regionalbank Lüneburg provided the loan, at a fixed interest rate of 3.2% over an 18-year term (covering 60% of the €14.5M total investment).

Claude called search_documents and got five passages with file name and page. Only the one from page 2 contains the loan, and that's the one Claude used. So the tool description decides when Claude searches your documents, and a description that says which documents are in the knowledge base would probably help.

Testing the server without Claude

I test the server through a real MCP client session instead of calling the Python functions directly. That way, the tests also check that the tools are registered and that everything works over the protocol.

async def call(tool: str, arguments: dict) -> str:
    """Call a tool through a real MCP client session and return its text output."""
    async with create_connected_server_and_client_session(
        mcp, raise_exceptions=True
    ) as client:
        result = await client.call_tool(tool, arguments)
    assert not result.isError
    return "\n".join(c.text for c in result.content if isinstance(c, TextContent))


async def test_search_returns_passage_with_file_and_page(server_with):
    """A matching passage should come back with its file name and page number."""
    server_with.add_texts([LOAN_TEXT], metadatas=[{"source": "report.pdf", "page": 2}])

    output = await call("search_documents", {"query": LOAN_TEXT})

    assert "[report.pdf, page 3]" in output
    assert "Sparkasse Nordheide" in output

create_connected_server_and_client_session from the MCP SDK connects a client to the server in memory, so I don't have to start an HTTP server for the tests. The fixture server_with replaces the real knowledge base with an in-memory Chroma that uses fake embeddings (DeterministicFakeEmbedding). So no model has to be downloaded, and the five tests run in a few seconds.

The tests check that the client sees exactly the two tools, that a search returns the passage with file name and page, that an empty knowledge base returns NO_DOCUMENTS_FOUND, that every file is listed only once and that the ingest only reads PDF, TXT and Markdown files.

What's missing for production

This is a demo project, so I deliberately left out a few things a real company would need.

• Authentication. MCP supports OAuth, but that's a whole topic of its own. That's why the server only listens on localhost (127.0.0.1), so nobody else in the network can search the documents.
• Deployment. For a team, the server would run as a container on a server instead of on my laptop.
• Scheduled ingest from the real source. Company documents usually live in Confluence or SharePoint, not in a local folder.
• Restart after ingest. Because of @cache, the server has to be restarted after every ingest to see the new documents.

Conclusion

Turning an existing RAG pipeline into an MCP server was much easier than I expected. I didn't change the search at all. I only wrapped it in two tools with FastMCP, and Claude Code could use it right away. The most important part is the description of the tools, because the model decides based on it when to search your documents. In my test, Claude only searched after I told it to. And because the server calls no LLM, it needs no API keys and works with any MCP client and any model.

You can find the full code in my repo langgraph-rag-agent.

AI helped me with the English, the wording and some parts of the text. I reviewed and edited everything myself.

Comments

[AgentSearch]

"The server calls no LLM" is the decision I'd copy first. It keeps the server cheap and lets whoever is calling pick the model. One thing that helped me with a similar search tool: return the file name, page and a short "last indexed" date with every passage, and say in the tool description that results come from a fixed folder. Then the agent can tell the user when an answer might be older than the latest version of a document, instead of quoting it as current. Did you find Claude Code calls list_documents on its own before searching, or only when asked?