Felicia Grace for BytesRack

Posted on Aug 14 • Originally published at bytesrack.com

How to Build a Self-Hosted RAG Pipeline Using Milvus + Ollama on Bare Metal

#machinelearning #ai #devops #python

Every added document, support ticket, or internal wiki page sent to a cloud LLM API is another line item on next month's bill — and another copy of your company's data sitting on someone else's infrastructure.

For teams working with proprietary documentation, customer records, or regulated data, that trade-off is becoming harder to justify.

A self-hosted RAG (Retrieval-Augmented Generation) pipeline solves both problems at once. RAG is a technique that lets a language model answer questions using your own documents as source material, rather than relying only on what it learned during training. It retrieves relevant chunks of your content first, then generates an answer grounded in that context.

This tutorial walks through building a complete, private RAG stack using Milvus as the vector database and Ollama to run the language model locally, connected with LangChain, all running on a single bare metal server you control.

Zero API keys, no per-token billing, and no documents leaving your hardware. 🔒

🏗️ The Architecture of Our Private AI

Before writing any code, it helps to see how the pieces connect. The pipeline has two phases:

Indexing (Done once, or when documents change): Document → split into chunks → converted to embeddings (Nomic) → stored in Milvus.
Querying (Done per user question): User question → Milvus similarity search retrieves relevant chunks → chunks + question sent to Ollama (Llama 3) → generated answer.

Every component in that chain runs as a local process. Nothing in this flow requires an outbound API call.

Prerequisites & Server Requirements

This pipeline is not overly resource-intensive for small document sets, but running two model-serving processes (embeddings and generation) alongside Milvus benefits from headroom. A reasonable bare metal baseline:

CPU: 8 cores or more
RAM: 32GB+
Storage: NVMe SSD (for fast vector index reads/writes)
OS: Ubuntu 22.04 LTS or later

Note: This guide assumes Milvus is already running on your server, listening on localhost:19530.

Step 1: Installing Ollama on Your Dedicated Server

Ollama handles model serving for both the LLM and the embedding model, exposing a simple local API on port 11434. Install it with the official script:

curl -fsSL [https://ollama.com/install.sh](https://ollama.com/install.sh) | sh

Verify it's running:

systemctl status ollama

Why bare metal matters here: Model inference is CPU/GPU and memory-bandwidth intensive. On a shared VPS, "noisy neighbor" workloads can throttle throughput unpredictably. On a dedicated bare metal server, every core is available exclusively to your model, translating to faster token generation.

Step 2: Pulling the LLM and Embedding Models

With Ollama installed, pull the two models this pipeline needs.

The generation model (Llama 3):

   ollama pull llama3

shell

The embedding model (Nomic):

   ollama pull nomic-embed-text

Note: nomic-embed-text is purpose-built for embedding tasks and is significantly smaller/faster than using a general-purpose LLM for vectorization.

Step 3: Setting Up the Python Environment

Create a project directory and install the required libraries:

mkdir rag-pipeline && cd rag-pipeline
python3 -m venv venv
source venv/bin/activate
pip install pymilvus langchain langchain-community langchain-milvus bs4

Step 4: Writing the RAG Pipeline Script

Create a single file named rag_app.py and add the following Python code to handle loading and chunking your data:

from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load a source document
loader = WebBaseLoader("[https://example.com/your-internal-doc](https://example.com/your-internal-doc)")
documents = loader.load()

# Split into overlapping chunks so context isn't lost
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150
)
chunks = text_splitter.split_documents(documents)

print(f"Document split into {len(chunks)} chunks")

2. Generating Embeddings and Storing in Milvus

Convert chunks into vectors using the local nomic model, then write to Milvus.

from langchain_community.embeddings import OllamaEmbeddings
from langchain_milvus import Milvus

# Local embedding model — no external API call
embeddings = OllamaEmbeddings(model="nomic-embed-text")

vector_store = Milvus.from_documents(
    documents=chunks,
    embedding=embeddings,
    connection_args={"host": "localhost", "port": "19530"},
    collection_name="rag_documents"
)

print("Chunks embedded and stored in Milvus")

3. Querying and Generating the Answer

Chain the retriever and generation model together.

from langchain_community.llms import Ollama
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_retrieval_chain

# Local generation model
llm = Ollama(model="llama3")

# Retriever pulls the most relevant chunks
retriever = vector_store.as_retriever(search_kwargs={"k": 4})

# Prompt template
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer the question using only the following context. "
               "If the answer isn't in the context, say you don't know.\n\n"
               "Context: {context}"),
    ("human", "{input}")
])

combine_docs_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, combine_docs_chain)

question = "What does this document say about deployment requirements?"
result = rag_chain.invoke({"input": question})

print("\nAnswer:", result["answer"])

Step 5: Testing Your Self-Hosted RAG System

Run the script from your terminal:

python3 rag_app.py

If everything is wired correctly, you'll see console output confirming the chunk count, followed by a generated answer grounded strictly in your source document.

🚀 Why Run RAG on Bare Metal Instead of the Cloud?

Zero API costs: No per-1K-token billing for embeddings or generation.
Absolute data privacy: Documents, embeddings, and generated answers never leave the server.
Zero network latency: Milvus and Ollama communicate over localhost, eliminating round-trip internet latency.

You've now built a scalable, private foundation you can extend with more document sources or larger models, entirely under your control.

Deploy your private AI on high-performance bare metal servers today with BytesRack.

DEV Community