Building a Chat Assistant Using Elasticsearch as a Vector Database

Social post Disclaimer: This post is submitted as part of
the Elastic Blogathon

1. Introduction

Large Language Models (LLMs) like GPT-4 and Claude are incredibly good at sounding smart. They can write fluent, well-structured answers and hold impressive conversations. But ask them about something recent, internal, or highly specific, and things can go wrong. You’ll often get an answer that sounds confident—but isn’t actually correct.
This happens because LLMs don’t truly know your data. Their knowledge is fixed at training time, and they don’t have built-in access to live systems, enterprise documents, or private knowledge bases.
Retrieval-Augmented Generation (RAG) addresses this gap by combining an LLM with an external source of truth. Instead of relying only on what the model remembers, the system first retrieves relevant documents at query time and then asks the LLM to generate an answer grounded in that information.
In this blog, we’ll walk through how to build a simple RAG-powered chat assistant using Elasticsearch as the vector database. We’ll look at how document embeddings are indexed, how semantic retrieval works, and how this approach helps generative AI deliver more accurate, real-world answers.

1. Core Concepts

What Is Retrieval-Augmented Generation (RAG)?
Retrieval-Augmented Generation (RAG) is a simple but powerful idea: instead of asking an LLM to answer from memory alone, we let it look things up first.
The process happens in two clear steps. First comes retrieval, where the system searches a structured or unstructured knowledge base to find information that’s actually relevant to the user’s question. Then comes generation, where that retrieved context is added to the prompt and passed to the LLM, allowing it to produce an answer grounded in real data.
This setup helps the model “know what it doesn’t know.” Rather than guessing or fabricating details, the LLM relies on external sources of truth, resulting in responses that are more accurate, trustworthy, and context-aware.

What Are Embeddings and Vector Databases?
Text embeddings are a way of turning text into numbers that represent meaning. Instead of focusing on exact words, embeddings capture the intent behind a sentence and store it as a dense vector.
For example, the phrases “How do I reset my password?” and “Forgot my login credentials” use different words, but they mean almost the same thing. When converted into embeddings, their vectors end up very close to each other in vector space.
A vector database stores these embeddings and makes it possible to search by similarity. When a user submits a query, the system compares the query’s vector with stored vectors and retrieves the closest matches using distance measures such as cosine similarity. This is what enables semantic search—finding relevant content even when the wording doesn’t exactly match.

Why Vector Search Is Critical for Chat Assistants
Traditional keyword-based search works by matching exact words between a query and a document. While this is effective in many cases, it often falls short when the wording doesn’t line up—even if the intent is the same.
Vector search takes a different approach. Instead of looking for lexical overlap, it retrieves content that is conceptually related. This is especially important for conversational queries, where users rarely phrase questions using the same terms found in source documents.
By combining these approaches, a chat assistant can:
Retrieve more meaningful context for open-ended questions

Understand user intent expressed in natural language, not just keywords

Produce responses that are both relevant and grounded in facts

1. Why Elasticsearch as a Vector Database

Elasticsearch started its journey as a full-text search engine, but over time it has evolved into a powerful multimodal retrieval platform. Today, it supports both traditional keyword search and modern vector-based retrieval natively—making it a strong foundation for RAG and GenAI applications.
Native Vector Search Support
Elasticsearch provides the dense_vector field type for storing text embeddings directly in the index. On top of that, it supports approximate nearest-neighbor (ANN) search using kNN with HNSW graphs. This allows similarity searches to remain fast and efficient, even when working with millions of vectors at scale.
Hybrid Search: Keyword + Semantic
In practical RAG pipelines, relying on vector similarity alone is rarely enough. Exact terms, identifiers, and domain-specific language still matter. Elasticsearch enables hybrid search by combining traditional relevance scoring (such as BM25) with vector similarity in a single query. This balance improves precision for important keywords while maintaining strong semantic recall.
Scalability and Production Readiness
Elasticsearch is built as a distributed system, making it easy to scale horizontally, maintain high availability, and recover using snapshot-based backups. For enterprise-grade GenAI systems that continuously ingest and retrieve large volumes of data, this level of reliability is critical.
Why This Matters
In real-world AI applications, factors like retrieval latency, indexing performance, and data freshness directly impact the user experience. Elasticsearch strikes a balance across all three, offering a battle-tested platform that is well-suited for production-scale RAG systems.

1. Architecture Overview

The RAG architecture with Elasticsearch can be described in six logical stages:
Document Ingestion – Gather documents (PDFs, web pages, FAQs, etc.)
Chunking – Split long documents into smaller, semantically coherent chunks.
Embedding Generation – Convert each chunk into a numerical vector using a model like sentence-transformers/all-MiniLM-L6-v2 or OpenAI embeddings.
Vector Indexing – Store embeddings in Elasticsearch as dense_vector fields.
Query-Time Retrieval – Embed the user query, perform vector similarity search to retrieve top-k relevant chunks.
Response Generation – Combine retrieved chunks into a context and prompt the LLM to generate a grounded reply.
Text-Based Architecture Flow
User Query
│
▼
Embed Query → Vector Search in Elasticsearch → Retrieve Top-k Documents
│
▼
LLM Prompt = { User Query + Retrieved Context }
│
▼
Generate Final Response

1. Step-by-Step Implementation

Let’s walk through a simplified implementation using Python-style pseudo-code.
Note: The code below focuses on RAG logic; specific SDK syntax may vary.

Step 1: Create a Vector Index
We create an Elasticsearch index with a dense_vector field to hold embeddings and a text field for metadata.
json
PUT /knowledge_base
{
"mappings": {
"properties": {
"content": {
"type": "text"
},
"embedding": {
"type": "dense_vector",
"dims": 384,
"index": true,
"similarity": "cosine"
}
}
}
}

Step 2: Generate and Store Document Embeddings
For each document chunk:
Python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
docs = chunk_documents(load_knowledge_base())
for doc in docs:
vector = model.encode(doc["text"])
es.index(index="knowledge_base", document={
"content": doc["text"],
"embedding": vector.tolist()
})
This pipeline:
Loads documents
Splits them into manageable chunks (e.g., 200–500 tokens)
Generates embeddings
Indexes both text and vectors in Elasticsearch

Step 3: Perform Vector Similarity Search
When the user submits a query:
python
query_vector = model.encode(user_input)
response = es.search(
index="knowledge_base",
knn={
"field": "embedding",
"query_vector": query_vector.tolist(),
"k": 5,
"num_candidates": 100
}
)
retrieved_docs = [hit["_source"]["content"] for hit in response["hits"]["hits"]]

Step 4: Build the RAG Prompt and Generate a Response
Combine retrieved documents into the context section of the prompt:
python
context = "\n\n".join(retrieved_docs)
prompt = f"""
You are a knowledgeable chat assistant. Use the following context to answer:
Context:
{context}
Question:
{user_input}
"""
answer = llm.generate(prompt)
print(answer)
The LLM now uses retrieved evidence from Elasticsearch, grounding responses in real data.

1. Results and Observations

Improved Relevance
Semantic retrieval ensures that even nuanced or paraphrased queries retrieve contextually relevant passages — not just matching keywords.
Reduced Hallucinations
Grounding the LLM with factual, indexed data significantly reduces hallucination frequency. The assistant becomes more consistent and trustable, especially for enterprise and domain-specific use cases.
Enhanced User Experience
Users experience faster response times, richer context, and verifiable results — all powered by Elasticsearch’s ability to serve low-latency vector search queries at scale.

1. Possible Enhancements

Hybrid Search
Combine keyword filters with vector-based ranking:
json
{
"query": {
"bool": {
"must": {
"knn": { "embedding": { "vector": query_vector, "k": 10 } }
},
"should": {
"match": { "content": user_input }
}
}
}
}
This fusion maintains both precision and semantic richness.
Reranking
Use a cross-encoder or LLM-based reranker to reorder retrieved results for highest contextual fit.
Conversation Memory
Persist user session history and feed previous turns into the retrieval query to maintain continuity across multi-turn chats.
Scaling Considerations
Sharding: Balance shards for large embeddings datasets.
Compression: Consider quantization to save memory.
Freshness: Periodic re-embedding for dynamic content.
Caching: Store frequent queries or embeddings for faster inference.

1. Conclusion

Retrieval-Augmented Generation turns LLMs from static text generators into systems that can actively reason over real knowledge. By combining retrieval with generation, RAG bridges the gap between what a model can say and what it should say.
Using Elasticsearch as the vector database provides a retrieval layer that is both robust and production-ready, while naturally blending semantic understanding with traditional keyword search. This balance is especially important for real-world applications, where accuracy, relevance, and reliability matter as much as intelligence.
For developers and architects building AI-powered search or conversational systems, Elasticsearch offers a practical foundation—bringing together years of search maturity with modern vector capabilities. As GenAI continues to evolve, systems that retrieve context as thoughtfully as they generate responses will define the next generation of intelligent assistants, and Elasticsearch is well positioned to support that shift.

Author: Divya Sree Madduri
Elastic Blogathon 2026 – Vectorized Thinking