Ved Prajapati
Principal Generative AI Architect | Founder, Vedaris
Abstract
Retrieval-Augmented Generation (RAG) enables generative AI systems to ground model responses in external knowledge rather than relying exclusively on information encoded within a foundation model. While basic RAG implementations can retrieve semantically similar text and provide it to a model, production systems introduce considerably more complex architectural challenges involving document ingestion, chunking, embeddings, retrieval quality, vector search, hybrid retrieval, scalability, security, and application integration.
This paper presents the architecture of an enterprise Retrieval-Augmented Generation system designed to process multi-format knowledge sources and provide semantically relevant information to generative AI applications. The system uses Amazon Bedrock and Titan Embeddings to generate vector representations, combines semantic vector retrieval with keyword-based search through a hybrid retrieval strategy, exposes application functionality through FastAPI, and deploys application workloads on Amazon EKS with a React frontend.
The architecture emphasizes retrieval quality rather than treating RAG as a simple vector similarity problem. Document preparation, intelligent chunking, embedding generation, retrieval, ranking, context construction, and generation are treated as separate architectural stages that can be independently optimized and evaluated.
The resulting design demonstrates how cloud-native infrastructure and modern information-retrieval techniques can be combined to create scalable, production-oriented RAG applications on AWS.
Keywords: Generative AI, Retrieval-Augmented Generation, RAG, Semantic Search, Hybrid Search, Embeddings, Vector Search, Amazon Bedrock, Titan Embeddings, Amazon EKS
- Introduction
Foundation models possess powerful language understanding and generation capabilities, but their knowledge is constrained by their training data and model context.
Enterprise applications frequently require information that is:
organization-specific,
recently created,
frequently updated,
proprietary,
domain-specific,
unavailable during model training.
Retrieval-Augmented Generation addresses this limitation by retrieving relevant external information at inference time and supplying that information as context to a generative model.
A simplified RAG pipeline can be represented as:
Documents
↓
Chunking
↓
Embedding Generation
↓
Vector Index
↓
User Query
↓
Semantic Retrieval
↓
Context
↓
Foundation Model
However, production RAG systems require considerably more engineering than this simplified flow suggests.
Retrieval quality depends on how documents are segmented, how embeddings represent information, how queries are interpreted, how candidate passages are retrieved and ranked, and how the final context is constructed.
This project therefore approaches RAG as an information-retrieval architecture, rather than simply connecting a vector database to a foundation model.
- Research Question
The primary question explored by the system is:
How can semantic retrieval, intelligent document chunking, embeddings, and hybrid search be combined to improve the quality and scalability of enterprise Retrieval-Augmented Generation systems on AWS?
Secondary questions include:
How does document chunking affect retrieval quality?
When does semantic retrieval outperform keyword matching?
When does keyword matching recover information missed by semantic retrieval?
Can combining both approaches improve retrieval quality?
How should retrieved context be prepared for downstream generation?
How can the resulting architecture scale as document and query volume increases?
- System Objectives
The architecture was designed around several objectives.
Retrieval Quality
Relevant information should be retrieved even when user terminology differs from the source document.
Exact-Match Retrieval
The system should remain capable of retrieving exact identifiers, technical terminology, names, codes, and other lexical information.
Multi-Format Ingestion
The ingestion pipeline should accommodate different forms of enterprise information rather than assuming a single document structure.
Scalability
Retrieval and application workloads should support increasing document and request volumes.
Modularity
Ingestion, embedding, retrieval, generation, APIs, and frontend components should remain logically separated.
Production Deployment
The system should operate using cloud-native infrastructure rather than remaining a local RAG prototype.
- Architecture Overview
The system consists of several logical stages:
Enterprise Documents
↓
Document Processing
↓
Intelligent Chunking
↓
Amazon Titan Embeddings
↓
Vector Representation / Indexing
↓
User Query
↓
Query Embedding
↓
Semantic + Keyword Retrieval
↓
Hybrid Search
↓
Relevant Context
↓
Amazon Bedrock / Foundation Model
↓
Generated Response
↓
FastAPI
↓
React Application
The application infrastructure is deployed using Amazon EKS, providing a scalable container orchestration environment for backend and application workloads.
- Multi-Format Document Processing
Enterprise knowledge rarely exists in one standardized format.
Information may be distributed across:
PDF documents,
technical documentation,
source code,
structured data,
internal knowledge bases,
operational documentation.
Before information can participate in retrieval, it must be transformed into a representation suitable for indexing.
The ingestion pipeline therefore separates document extraction from retrieval preparation.
Each source is processed into textual or structured content while retaining relevant metadata.
Metadata can include information such as:
document_id
document_type
source
section
page
timestamp
chunk_id
Maintaining this relationship between retrieved content and its source is important for traceability and downstream citation.
- Intelligent Chunking
Embedding entire documents as single vectors can produce representations that are too broad for precise retrieval.
Conversely, extremely small chunks can remove the context required to understand a passage.
Chunking therefore represents an important RAG design decision.
The system divides documents into smaller retrievable units while attempting to preserve meaningful context.
Possible approaches include:
Fixed-Size Chunking
Documents are divided according to predefined token or character boundaries.
Hierarchical Chunking
Smaller child chunks provide precise retrieval while maintaining relationships with larger parent contexts.
Semantic Chunking
Document boundaries are determined according to semantic changes rather than arbitrary character counts.
Different document types may benefit from different strategies.
Source code, structured documentation and long-form prose do not necessarily have the same optimal segmentation strategy.
- Embedding Generation
After chunking, textual content is converted into vector representations using Amazon Titan Embeddings through Amazon Bedrock.
An embedding represents semantic characteristics of text as a numerical vector.
Conceptually:
"How do I rotate an API key?"
↓
Embedding Model
↓
[0.018, -0.291, 0.443, ...]
Document chunks undergo the same transformation.
Similarity between a query embedding and stored document embeddings can then be used to identify semantically related information.
This allows retrieval to identify conceptual similarity even when a user's wording differs from the source material.
- Semantic Retrieval
Semantic retrieval searches according to meaning rather than exact lexical overlap.
For example, a query such as:
"How can credentials be changed safely?"
may be semantically related to documentation discussing:
"secret rotation procedures"
even though the phrases contain few identical words.
The user's query is converted into an embedding and compared against indexed document vectors.
The most similar candidates are returned according to a similarity measure.
This provides a major advantage over purely keyword-based retrieval for natural-language queries.
However, semantic retrieval also has limitations.
- Keyword Retrieval
Certain enterprise queries depend heavily on exact terms.
Examples include:
API identifiers,
product names,
error codes,
version numbers,
function names,
technical acronyms.
Semantic similarity may not always prioritize these exact lexical matches.
Keyword retrieval therefore remains useful alongside vector search.
Rather than treating semantic retrieval as a complete replacement for traditional information retrieval, the system incorporates both approaches.
- Hybrid Search
The retrieval architecture combines:
Semantic Search
and
Keyword Search
into a hybrid retrieval strategy.
Conceptually:
User Query
↙ ↘
Vector Search Keyword Search
↘ ↙
Candidate Results
↓
Combination / Ranking
↓
Final Context
Semantic search provides conceptual matching.
Keyword search provides lexical precision.
Combining them allows the system to benefit from both retrieval paradigms.
This is particularly valuable in technical enterprise environments where queries frequently contain a mixture of natural language and exact technical terminology.
- Retrieval Ranking
Initial retrieval can return multiple candidate chunks.
Not every candidate should necessarily receive equal priority.
The retrieval pipeline can therefore introduce ranking or reranking before context reaches the foundation model.
A retrieval pipeline may take the form:
Initial Retrieval
↓
Candidate Documents
↓
Relevance Ranking
↓
Top-K Context
↓
Generation
This separates candidate discovery from final relevance selection.
The distinction becomes increasingly important as knowledge bases grow.
- Context Construction
Retrieving relevant information does not automatically guarantee a good generated answer.
Retrieved passages must be assembled into useful model context.
The context-construction stage must consider:
relevance,
duplication,
ordering,
available context window,
source metadata,
conflicting information.
Too little context can omit important information.
Too much context can introduce irrelevant information and consume unnecessary model tokens.
The system therefore treats context construction as an independent optimization problem rather than simply forwarding every retrieved result to the model.
- Generation with Amazon Bedrock
After retrieval, relevant context is supplied to a foundation model through Amazon Bedrock.
A simplified generation request contains:
System Instructions
Retrieved Enterprise Context
User Query
The model is instructed to produce an answer grounded in the supplied information.
This architecture separates knowledge retrieval from language generation.
The foundation model therefore does not need to permanently encode the enterprise knowledge being queried.
Knowledge can instead be updated independently through the retrieval system.
- FastAPI Application Layer
The system exposes application functionality through a FastAPI backend.
The API layer provides a controlled boundary between frontend clients and the underlying RAG architecture.
Potential API responsibilities include:
receiving user queries,
validating requests,
invoking retrieval,
invoking generation,
formatting responses,
handling errors,
returning source metadata.
A conceptual request may take the form:
{
"query": "How does the deployment approval process work?"
}
The backend then executes the RAG pipeline and returns the generated response with relevant metadata.
- React Frontend
A React frontend provides the user-facing application layer.
Separating the frontend from the RAG backend prevents retrieval and model logic from becoming tightly coupled to the interface.
The frontend can provide:
natural-language querying,
generated answers,
retrieved sources,
document references,
loading states,
error handling.
This architecture allows alternative clients to consume the same backend APIs in the future.
- Deployment on Amazon EKS
Application workloads are deployed using Amazon Elastic Kubernetes Service (EKS).
Containerization provides consistent runtime environments, while Kubernetes provides mechanisms for:
service deployment,
scaling,
workload isolation,
health management,
rolling updates,
resource allocation.
The use of EKS also separates the application architecture from individual compute instances.
As request volume increases, application components can scale independently according to workload requirements.
- Scalability
RAG systems contain several independently scalable components.
These include:
document ingestion,
embedding generation,
vector retrieval,
backend APIs,
model inference,
frontend delivery.
A production architecture should avoid assuming that these components scale at identical rates.
For example, embedding generation may be intensive during ingestion but minimal during normal querying.
Retrieval infrastructure may experience consistent query load.
Generation may represent the most expensive inference stage.
Separating these components allows resources to be scaled according to their individual workload characteristics.
- Security Considerations
Enterprise RAG introduces security concerns beyond conventional application security.
Retrieved information may contain proprietary or sensitive organizational data.
The system should therefore consider:
identity and authentication,
authorization,
document-level access,
encryption,
secrets management,
API protection,
network isolation,
logging and auditing.
A particularly important requirement is ensuring that retrieval does not expose documents a user is not authorized to access.
RAG security therefore must extend beyond protecting the model endpoint.
Retrieval itself must respect authorization boundaries.
- Evaluation Methodology
Retrieval quality should be measured rather than assessed exclusively through visual inspection of generated responses.
A representative evaluation dataset can contain:
user queries,
expected relevant documents,
expected relevant chunks,
expected answer information.
Different retrieval configurations can then be compared.
For example:
Configuration A
Semantic retrieval only.
Configuration B
Keyword retrieval only.
Configuration C
Hybrid retrieval.
Configuration D
Hybrid retrieval with reranking.
This enables objective comparison between retrieval strategies.
- Evaluation Metrics
Useful metrics include:
Precision
The proportion of retrieved chunks that are actually relevant.
Recall
The proportion of relevant information successfully retrieved.
Precision@K
The relevance of the top K retrieved results.
Recall@K
Whether expected relevant information appears within the top K results.
Retrieval Latency
Time required to retrieve relevant context.
End-to-End Latency
Total time from user request to generated response.
Groundedness
The degree to which generated answers are supported by retrieved context.
Answer Relevance
The degree to which the generated answer addresses the user's query.
Together, these metrics provide a more meaningful view of RAG quality than simply evaluating whether an answer appears fluent.
- Expected Trade-Offs
Improving retrieval quality can introduce additional computational cost and latency.
Hybrid retrieval requires multiple retrieval strategies.
Reranking introduces another processing stage.
Larger candidate sets may improve recall but increase downstream processing.
Larger chunks preserve context but may reduce retrieval precision.
Smaller chunks improve precision but risk removing surrounding meaning.
Production RAG architecture therefore involves balancing:
Quality
Latency
Cost
Context
Scalability
rather than optimizing one dimension in isolation.
- Observability
Production RAG systems require visibility across the entire retrieval and generation pipeline.
Useful telemetry includes:
ingestion failures,
embedding latency,
retrieval latency,
retrieved document count,
retrieval scores,
model latency,
token consumption,
API errors,
end-to-end response time.
Retrieval traces can be especially valuable when debugging poor model responses.
If an answer is incorrect, engineers should be able to determine whether the failure originated from:
document processing,
chunking,
embedding,
retrieval,
ranking,
context construction,
generation.
Without this separation, retrieval failures can easily be mistaken for model failures.
- Limitations
RAG does not guarantee factual correctness.
If the correct information is absent from the source corpus, retrieval cannot provide it.
If chunking separates important information incorrectly, retrieval quality may decline.
Embeddings may fail to represent certain domain-specific relationships.
Hybrid search increases architectural complexity.
Retrieved documents may themselves contain inaccurate or contradictory information.
Foundation models may still generate unsupported statements despite receiving relevant context.
These limitations demonstrate why RAG systems require evaluation at both the retrieval layer and generation layer.
- Future Work
Future improvements to the architecture could include:
Hierarchical Retrieval
Retrieve precise child chunks while providing broader parent context to the model.
Semantic Chunking
Dynamically identify document boundaries according to meaning.
Query Decomposition
Break complex user questions into smaller retrieval tasks.
Reranking
Apply dedicated relevance models to improve candidate ordering.
Metadata Filtering
Restrict retrieval according to document type, organization, user permissions, date, or other metadata.
Agentic Retrieval
Allow an AI agent to iteratively determine when additional retrieval is required.
Retrieval Evaluation Pipeline
Continuously measure retrieval quality against representative enterprise queries.
Model Evaluation
Evaluate generated responses for relevance, groundedness and correctness.
- Conclusion
This paper presented the architecture of an enterprise Retrieval-Augmented Generation system with semantic and hybrid search on AWS.
The system processes multi-format enterprise information through an intelligent document-processing and chunking pipeline, generates vector representations using Amazon Titan Embeddings, and combines semantic similarity with keyword retrieval to improve information discovery.
Relevant context is provided to foundation models through Amazon Bedrock, application functionality is exposed through FastAPI, user interaction is provided through a React frontend, and application workloads are deployed using Amazon EKS.
The architecture demonstrates that production RAG should not be treated simply as:
LLM + vector database
Instead, RAG is an end-to-end information-retrieval architecture involving:
ingestion → chunking → embeddings → indexing → retrieval → ranking → context construction → generation → evaluation
The quality of the final generative AI application depends on each of these stages.
By combining semantic retrieval, keyword matching, cloud-native infrastructure, and systematic evaluation, enterprise RAG systems can provide more relevant and grounded access to organizational knowledge while retaining the scalability and operational controls required for production environments.
Technology Stack
Generative AI: Amazon Bedrock
Embeddings: Amazon Titan Embeddings
Retrieval: Semantic Search, Vector Search, Hybrid Search
Compute & Orchestration: Amazon EKS
Backend: FastAPI
Frontend: React
Primary Domain: Generative AI, RAG, Semantic Retrieval, Enterprise AI
Author
Ved Prajapati
Principal Generative AI Architect | Founder, Vedaris
AWS Certified Generative AI Developer – Professional
Stanford CS234 · MIT 6.S191 · Harvard CS50x
Top comments (1)
The requirement that separates enterprise RAG from the tutorial version is that permissions have to survive the whole pipeline. Documents inherit access rules from wherever they came from, and once they are chunked and embedded that context is gone unless it is carried as metadata and enforced as a pre-filter at query time. Filtering after retrieval is the tempting shortcut and it is wrong twice over: you leak through citation counts and snippets, and you silently degrade recall because the top-k was spent on chunks the user was never allowed to see. The other enterprise-specific problem is reindexing. Embedding models get upgraded, and on a corpus of any size you cannot do that atomically - so the design needs to tolerate two index versions coexisting, with queries pinned to one, rather than discovering mid-migration that scores from different spaces are being compared.