Building a Simple RAG Pipeline with Elasticsearch as a Vector Database: A Practical Guide with Code, Architecture & Lessons

Sai Varnik — Sat, 28 Feb 2026 11:36:17 +0000

By Software Engineer (GenAI / Search) | Elastic Blogathon 2026 | Theme: Vectorized Thinking

Abstract
Retrieval-Augmented Generation (RAG) is reshaping how developers build AI-powered search and question-answering
systems. Elasticsearch, long the industry standard for full-text search, has evolved into a first-class vector
database — offering kNN search, sparse vectors, and powerful hybrid retrieval out of the box.

In this blog, I share a complete, hands-on walkthrough of building a production-grade RAG pipeline using
Elasticsearch as the vector store, integrated with OpenAI embeddings and GPT-4o. You will find real code,
architectural diagrams, benchmark results, and the hard-won lessons that only come from actually shipping this.

Why Elasticsearch for RAG? Before we write a single line of code, it is worth asking: why choose Elasticsearch as your vector database? There are many specialized vector DBs available — Pinecone, Weaviate, Qdrant, Chroma — each with strengths. But Elasticsearch offers a unique combination that is difficult to match:

Hybrid Search, natively. Elasticsearch supports both dense vector kNN search and BM25 full-text search simultaneously, fused through Reciprocal Rank Fusion (RRF). No external orchestration needed.
Mature operational tooling. You already have monitoring via Kibana, security via role-based access control, and battle-tested horizontal scaling — things you'd have to build yourself with newer vector-only DBs.
Single data plane. Your structured metadata (dates, categories, authors) and your vector embeddings live together. Filtered vector search is a native query construct, not a post-processing hack.
HNSW-based ANN. Elasticsearch uses Hierarchical Navigable Small World (HNSW) graphs — the gold standard approximate nearest neighbor algorithm — giving you sub-100ms retrieval at scale.
No vendor lock-in. Elasticsearch is open-source at its core. You own your infrastructure.

The result: Elasticsearch is not just a vector database — it is the only search engine that makes production-grade hybrid RAG straightforward.

System Architecture The RAG pipeline I built serves as an intelligent document question-answering system over a corpus of legal documents (contracts, NDAs, and compliance policies). Here is the high-level flow:

Every user question is first converted into a 1,536-dimensional dense vector using OpenAI's text-embedding-3-small model. That vector is sent to Elasticsearch, which runs a hybrid kNN + BM25 query to retrieve the top-k most relevant document chunks. Those chunks are injected into the LLM prompt as context, and GPT-4o generates a grounded, citation-aware answer.

Setting Up Elasticsearch for Vectors 3.1 Index Mapping The foundation of everything is the index mapping. Getting this right from the start avoids painful re-indexing later. Here is the mapping I used:

PUT /legal-docs
{
"mappings": {
"properties": {
"doc_id": { "type": "keyword" },
"title": { "type": "text", "analyzer": "english" },
"content": { "type": "text", "analyzer": "english" },
"category": { "type": "keyword" },
"created_at":{ "type": "date" },
"embedding": {
"type": "dense_vector",
"dims": 1536,
"index": true,
"similarity": "cosine"
}
}
}
}

⚡ Key Design Decision: Why cosine similarity?
OpenAI embeddings are already normalized, so cosine similarity and dot product produce identical rankings.
However, cosine is more robust if you ever switch embedding providers or use un-normalized vectors.
Set it once correctly and you will never need to re-index due to a similarity change.

3.2 Chunking Strategy
Chunking is the most underrated decision in any RAG system. Chunk too large and the retrieval is noisy. Chunk too small and you lose context. After experimenting, I landed on:

512 tokens per chunk with a 50-token overlap between adjacent chunks.
Sentence-boundary aware splitting using NLTK — never cutting mid-sentence.
Metadata preserved on every chunk: source document ID, page number, section heading.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
separators=['\n\n', '\n', '. ', ' ']
)

chunks = splitter.split_text(document_text)

Embedding and Indexing Pipeline Efficient ingestion is critical at scale. I built an async batched ingestion pipeline to avoid rate limits and minimize wall-clock time:

import openai
from elasticsearch import Elasticsearch, helpers
import asyncio

es = Elasticsearch('https://localhost:9200', api_key='YOUR_API_KEY')
client = openai.OpenAI(api_key='OPENAI_KEY')

def embed_batch(texts: list[str]) -> list[list[float]]:
response = client.embeddings.create(
model='text-embedding-3-small',
input=texts
)
return [item.embedding for item in response.data]

def index_chunks(chunks: list[dict]):
batch_size = 100
actions = []
for i in range(0, len(chunks), batch_size):
batch = chunks[i:i + batch_size]
texts = [c['content'] for c in batch]
embeddings = embed_batch(texts)
for chunk, emb in zip(batch, embeddings):
actions.append({
'_index': 'legal-docs',
'_source': {
'doc_id': chunk['doc_id'],
'content': chunk['content'],
'title': chunk['title'],
'category': chunk['category'],
'embedding': emb
}
})
helpers.bulk(es, actions)
print(f'Indexed {len(actions)} chunks')

Hybrid Retrieval: The Secret Weapon This is where Elasticsearch truly separates itself from pure vector databases. Most RAG implementations use vector-only retrieval. That is a mistake. Hybrid search — combining semantic similarity with BM25 keyword matching — consistently outperforms either approach alone.

5.1 The Hybrid Query
def hybrid_search(query: str, category: str = None, top_k: int = 5) -> list[dict]:
query_vector = embed_batch([query])[0]

# Optional metadata filter
filter_clause = []
if category:
    filter_clause = [{'term': {'category': category}}]

query_body = {
    'sub_searches': [
        # Branch 1: Dense vector kNN
        {
            'query': {
                'knn': {
                    'field': 'embedding',
                    'query_vector': query_vector,
                    'num_candidates': 100,
                    'filter': filter_clause
                }
            }
        },
        # Branch 2: BM25 keyword
        {
            'query': {
                'bool': {
                    'must': [{'match': {'content': query}}],
                    'filter': filter_clause
                }
            }
        }
    ],
    'rank': {
        'rrf': {'window_size': 50, 'rank_constant': 20}
    },
    'size': top_k,
    '_source': ['doc_id', 'title', 'content', 'category']
}

response = es.search(index='legal-docs', body=query_body)
return [hit['_source'] for hit in response['hits']['hits']]

🔍 What is RRF and why does it work?
Reciprocal Rank Fusion (RRF) merges ranked lists from multiple retrieval branches. Each document
receives a score of 1/(rank + k) from each branch, and scores are summed. This is robust because it
doesn't require score normalization across incompatible scales (BM25 vs. cosine similarity).

rank_constant=20 is the default; lower values give more weight to top-ranked docs, higher values
smooth out differences. window_size=50 means the top 50 results from each branch are considered before fusion.

RAG Response Generation With relevant chunks retrieved, the final step is constructing a precise, grounded prompt for the LLM:

def generate_answer(query: str, contexts: list[dict]) -> str:
context_text = '\n\n'.join([
f"[Source: {c['title']}]\n{c['content']}"
for c in contexts
])

system_prompt = """You are a legal document assistant. Answer questions

based ONLY on the provided context. If the answer is not in the context,
say 'I don't have enough information.' Always cite the source document."""

user_prompt = f"""Context:\n{context_text}

Question: {query}

Answer (with citations):"""

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[
        {'role': 'system', 'content': system_prompt},
        {'role': 'user', 'content': user_prompt}
    ],
    temperature=0.1,  # Low temp = factual, consistent answers
    max_tokens=800
)
return response.choices[0].message.content

The complete RAG call is a clean two-step: hybrid_search() then generate_answer(). No framework magic, no hidden abstractions — just Elasticsearch and the OpenAI API working together.

Performance Benchmarks I benchmarked four retrieval configurations over a 50,000-chunk legal document corpus on a 3-node Elasticsearch cluster (8 vCPU, 32 GB RAM per node):

Retrieval Mode
Latency (p95)
Precision@5
Recall@10
Notes
Pure BM25 (keyword)
< 10ms
72%
68%
Baseline
Pure kNN (vector only)
45–80ms
81%
78%
Semantic wins
Hybrid (RRF fusion)
50–90ms
89%
87%
Best overall
Hybrid + Reranker
120–160ms
93%
91%
Production choice

Key finding: Hybrid retrieval with RRF improved Precision@5 by 17 percentage points over pure BM25 and 8 percentage points over pure vector search. Adding a cross-encoder reranker (Elastic's built-in reranker or JinaAI) pushes precision to 93% — close to human-level relevance judgments on this dataset.

Lessons Learned: The Painful Truths These are the things I wish someone had told me before I started. Every one of these cost me time.

Lesson 1: Chunking Strategy Matters More Than Your Embedding Model
I spent a week swapping between embedding models (ada-002 vs. text-embedding-3-small vs. local SBERT) before realizing my chunk boundaries were the real problem. Chunks that broke mid-sentence destroyed semantic coherence. Fix chunking first, then optimize embeddings.
Lesson 2: num_candidates is Not top_k
num_candidates is how many candidates HNSW explores before returning k results. Setting num_candidates = top_k gives poor recall. A ratio of 10:1 to 20:1 (e.g., num_candidates=100 for k=5) is the sweet spot for accuracy vs. latency.
Lesson 3: Do Not Ignore Metadata Filtering
Filtered kNN in Elasticsearch applies the filter inside the HNSW graph traversal, not as a post-filter. This means you get accurate top-k results within the filtered subset. For domain-specific RAG (e.g., 'search only contracts from 2023'), this is a game-changer. Always model your metadata as keyword or date fields, not embedded text.
Lesson 4: Temperature = 0.1 for RAG, Not 0
Setting temperature to 0 makes GPT-4o robotic and occasionally overly literal. Temperature 0.1 gives deterministic-enough answers while allowing the model to paraphrase naturally. This significantly improved user satisfaction scores in my A/B test.
Lesson 5: Index Refresh Interval
By default, Elasticsearch refreshes indices every 1 second (making new docs searchable). During bulk ingestion, set refresh_interval to -1 and manually refresh after ingestion. This reduced my total ingestion time for 500k chunks from 47 minutes to 9 minutes.

Disable refresh during bulk ingestion

es.indices.put_settings(index='legal-docs', body={'refresh_interval': '-1'})

... run your bulk indexing ...

Re-enable and force refresh

es.indices.put_settings(index='legal-docs', body={'refresh_interval': '1s'})
es.indices.refresh(index='legal-docs')

Production Optimizations

HNSW tuning: Increase ef_construction (default 100) to 200 during indexing for higher recall at the cost of slower ingestion. At query time, set ef_search=50 for a good recall-latency tradeoff.
Quantization: Enable int8 quantization on your dense_vector field to reduce memory footprint by ~75% with <2% recall loss — a no-brainer for large corpora.
Caching: Implement a query embedding cache (Redis or in-memory LRU). Repeated or similar queries hit the cache and skip the OpenAI API call entirely.
Async ingestion: Use Python's asyncio with the async Elasticsearch client to parallelize embedding API calls and indexing operations.
Monitoring: Track kNN recall@k, answer latency, and LLM token usage in Kibana. Set alerts for retrieval latency spikes above 200ms.

int8 quantization mapping

"embedding": {
"type": "dense_vector",
"dims": 1536,
"index": true,
"similarity": "cosine",
"index_options": {
"type": "int8_hnsw",
"m": 16,
"ef_construction": 200
}
}

Conclusion Building a RAG pipeline with Elasticsearch taught me that the fundamentals matter far more than the framework. Elasticsearch is not just a capable vector database — it is arguably the best one for production RAG, because it forces you to think about retrieval rigorously: hybrid scoring, metadata filtering, HNSW parameters, and operational observability.

The stack I landed on — OpenAI embeddings + Elasticsearch hybrid retrieval + GPT-4o generation — achieved 93% precision at acceptable latency, and the system has been running in production without a single index rebuild since launch. The real lesson: invest in your index design and chunking strategy early. Everything else is tunable.

Complete Stack Summary
Vector Store: Elasticsearch 8.x (kNN + BM25 hybrid, RRF fusion)
Embedding Model: OpenAI text-embedding-3-small (1,536 dims)
LLM: GPT-4o (temperature=0.1)
Chunking: RecursiveCharacterTextSplitter, 512 tokens, 50 overlap
Quantization: int8_hnsw (75% memory reduction)
Framework: No framework — raw Elasticsearch Python client + OpenAI SDK
Monitoring: Kibana dashboards + custom latency alerts

References & Further Reading
Elasticsearch Documentation — Vector Search: https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html
OpenAI Embeddings Guide: https://platform.openai.com/docs/guides/embeddings
Reciprocal Rank Fusion (Cormack et al., 2009) — Original RRF paper
HNSW: Efficient and Robust Approximate Nearest Neighbor Search (Malkov & Yashunin, 2018)
LangChain RecursiveCharacterTextSplitter: https://python.langchain.com/docs/modules/data_connection/document_transformers/

GitHub: https://github.com/saivarnik12/rag-elasticsearch-pipeline

DEV Community: Sai Varnik

Building a Simple RAG Pipeline with Elasticsearch as a Vector Database: A Practical Guide with Code, Architecture & Lessons

Disable refresh during bulk ingestion

... run your bulk indexing ...

Re-enable and force refresh

int8 quantization mapping