Why RAG Matters More Than the Model
Large language models are impressive, but they have two problems that block most enterprise use cases. They do not know your private data, and they do not know what changed last week. Retrieval-Augmented Generation (RAG) solves both by fetching relevant passages from your own documents at query time and placing them in the prompt. The model then answers from evidence instead of memory.
The idea is simple. The execution is where teams struggle. A demo that answers three questions from a PDF takes an afternoon. A system that answers thousands of questions over ten thousand documents, with citations, access control, and measurable quality, takes a real engineering effort. The quality of a RAG system is almost always decided before the model is called: in ingestion, chunking, embedding, and retrieval.
This article walks through a complete RAG pipeline in Java. We will build it in six stages: document ingestion, chunking, embeddings, vector storage, retrieval evaluation, and re-ranking. Every stage has a small, testable interface so you can swap implementations later, for example moving from an in-memory store to pgvector or from a generic embedding model to one tuned for your domain.
The Architecture at a Glance
A RAG system has two pipelines that share a vector index.
The indexing pipeline runs offline or on events. It reads documents, splits them into chunks, computes an embedding for each chunk, and stores the chunk text, its metadata, and its vector.
The query pipeline runs per request. It embeds the user question, retrieves the top candidate chunks by vector similarity, optionally re-ranks them with a more precise model, assembles a prompt with the best passages and their sources, and calls the LLM.
Documents → Ingest → Chunk → Embed → Vector store
↓
User question → Embed → Retrieve (top-k) → Re-rank → Prompt → LLM → Answer + citations
Most quality problems are visible in one of these stages. Keeping the stages separate lets you measure each one.
Stage 1: Document Ingestion
Ingestion turns raw files into normalized text with metadata. Keep it boring and deterministic: the same file must always produce the same text, or your index will drift.
public record SourceDocument(
String id,
String sourceUri,
String tenantId,
String text,
Map<String, String> metadata) {}
public interface DocumentLoader {
boolean supports(Path file);
SourceDocument load(Path file, String tenantId) throws IOException;
}
A plain-text or Markdown loader is straightforward. For PDFs, use Apache PDFBox for text extraction and add a fallback to OCR when a page has no text layer. Record page numbers in metadata whenever the format provides them, because citations that point to "page 14" build far more trust than citations to a whole file.
public final class TextFileLoader implements DocumentLoader {
@Override
public boolean supports(Path file) {
String name = file.getFileName().toString().toLowerCase(Locale.ROOT);
return name.endsWith(".txt") || name.endsWith(".md");
}
@Override
public SourceDocument load(Path file, String tenantId) throws IOException {
String text = Files.readString(file, StandardCharsets.UTF_8);
String id = sha256(file.toAbsolutePath().toString());
return new SourceDocument(id, file.toUri().toString(), tenantId,
normalize(text),
Map.of("filename", file.getFileName().toString()));
}
private static String normalize(String raw) {
return raw.replace("\r\n", "\n")
.replaceAll("[ \\t]+\\n", "\n")
.replaceAll("\\n{3,}", "\n\n")
.trim();
}
private static String sha256(String value) {
try {
byte[] digest = MessageDigest.getInstance("SHA-256")
.digest(value.getBytes(StandardCharsets.UTF_8));
return HexFormat.of().formatHex(digest);
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException(e);
}
}
}
Two decisions in this code matter later. The document ID is derived from the source path, so re-ingesting the same file overwrites its chunks instead of duplicating them. And the tenant ID is attached at load time, so every chunk carries the information needed to enforce access control at retrieval time.
Stage 2: Chunking Strategies
Embedding models have input limits, and retrieval works better when each vector represents one coherent idea. Chunking is the step that decides both.
There are three common strategies:
- Fixed-size with overlap. Split every N characters with an overlap of M characters. Simple and predictable, but it cuts sentences and tables in half.
- Recursive splitting. Try to split on paragraphs first, then sentences, then words, until chunks fit the target size. This respects natural boundaries and is a good default.
- Structure-aware splitting. Use headings, sections, or document elements (tables, list items, code blocks) as boundaries. Best quality for well-structured documents, but it requires a parser that understands the structure.
The recursive splitter below is a solid default. It targets a maximum size, keeps a small overlap so context is not lost at boundaries, and attaches the chunk's position to its metadata.
public record Chunk(
String id,
String documentId,
int ordinal,
String text,
String tenantId,
Map<String, String> metadata) {}
public final class RecursiveChunker {
private static final List<String> SEPARATORS = List.of("\n\n", "\n", ". ", " ");
private final int maxChars;
private final int overlapChars;
public RecursiveChunker(int maxChars, int overlapChars) {
if (overlapChars >= maxChars) {
throw new IllegalArgumentException("overlap must be smaller than max size");
}
this.maxChars = maxChars;
this.overlapChars = overlapChars;
}
public List<Chunk> chunk(SourceDocument doc) {
List<String> pieces = split(doc.text(), 0);
List<Chunk> chunks = new ArrayList<>();
StringBuilder current = new StringBuilder();
for (String piece : pieces) {
if (current.length() + piece.length() > maxChars && current.length() > 0) {
chunks.add(toChunk(doc, chunks.size(), current.toString()));
String tail = tail(current.toString(), overlapChars);
current.setLength(0);
current.append(tail);
}
current.append(piece);
}
if (current.length() > 0) {
chunks.add(toChunk(doc, chunks.size(), current.toString()));
}
return chunks;
}
private List<String> split(String text, int separatorIndex) {
if (text.length() <= maxChars || separatorIndex >= SEPARATORS.size()) {
return List.of(text);
}
String separator = SEPARATORS.get(separatorIndex);
List<String> result = new ArrayList<>();
for (String part : text.split(Pattern.quote(separator), -1)) {
if (part.isEmpty()) continue;
if (part.length() > maxChars) {
result.addAll(split(part, separatorIndex + 1));
} else {
result.add(part + separator);
}
}
return result;
}
private static String tail(String text, int chars) {
if (text.length() <= chars) return text;
int start = text.length() - chars;
int space = text.indexOf(' ', start);
return space == -1 ? text.substring(start) : text.substring(space + 1);
}
private Chunk toChunk(SourceDocument doc, int ordinal, String text) {
Map<String, String> meta = new HashMap<>(doc.metadata());
meta.put("ordinal", String.valueOf(ordinal));
meta.put("source", doc.sourceUri());
return new Chunk(doc.id() + "#" + ordinal, doc.id(), ordinal,
text.strip(), doc.tenantId(), meta);
}
}
Chunk size is the most important tuning parameter, and there is no universal answer. Start with 500 to 800 characters and an overlap of about 10 to 15 percent. Then measure. Chunks that are too small lose context; chunks that are too large dilute the vector so it matches many questions weakly. Section 5 shows how to compare settings with numbers instead of intuition.
Stage 3: Embeddings
An embedding model maps text to a vector where semantically similar texts are close. Your choice of model determines what "similar" means, so treat it as a core dependency.
Define an interface so the rest of the system does not depend on a specific provider:
public interface EmbeddingModel {
String modelId();
int dimensions();
List<float[]> embed(List<String> texts);
}
Two practical points. First, batch requests. Embedding one chunk per HTTP call is slow and expensive. Send 64 to 256 texts per call, depending on the provider's limits. Second, store the model ID with every vector. If you upgrade the model, you must re-embed the whole corpus, and a stored model ID lets you detect and migrate mixed indexes safely.
For a concrete provider, Spring AI offers an EmbeddingModel abstraction that works with OpenAI, Azure OpenAI, Ollama, and others. For local experiments, a small open model served through Ollama or ONNX Runtime keeps costs at zero.
Stage 4: Vector Store
The vector store holds chunks and vectors and answers one question quickly: what are the k nearest vectors to this query, among the chunks this user is allowed to see?
public record ScoredChunk(Chunk chunk, double score) {}
public interface VectorStore {
void upsert(List<Chunk> chunks, List<float[]> vectors, String modelId);
List<ScoredChunk> search(float[] query, int k, String tenantId, Map<String, String> filters);
}
The tenantId parameter is not optional. Filtering by tenant must happen inside the store, not in application code after retrieval, because a post-filter on a top-k list can return fewer results than requested or, worse, leak data if someone forgets it.
Here is a minimal in-memory implementation using cosine similarity. It is fine for tests and small corpora, and it shows exactly what a production store must do:
public final class InMemoryVectorStore implements VectorStore {
private record Entry(Chunk chunk, float[] vector, String modelId) {}
private final List<Entry> entries = new CopyOnWriteArrayList<>();
@Override
public void upsert(List<Chunk> chunks, List<float[]> vectors, String modelId) {
for (int i = 0; i < chunks.size(); i++) {
String id = chunks.get(i).id();
entries.removeIf(e -> e.chunk().id().equals(id));
entries.add(new Entry(chunks.get(i), normalize(vectors.get(i)), modelId));
}
}
@Override
public List<ScoredChunk> search(float[] query, int k, String tenantId,
Map<String, String> filters) {
float[] q = normalize(query);
return entries.stream()
.filter(e -> e.chunk().tenantId().equals(tenantId))
.filter(e -> matches(e.chunk(), filters))
.map(e -> new ScoredChunk(e.chunk(), dot(q, e.vector())))
.sorted(Comparator.comparingDouble(ScoredChunk::score).reversed())
.limit(k)
.toList();
}
private static boolean matches(Chunk chunk, Map<String, String> filters) {
return filters.entrySet().stream()
.allMatch(f -> f.getValue().equals(chunk.metadata().get(f.getKey())));
}
private static float[] normalize(float[] v) {
double norm = 0;
for (float x : v) norm += x * x;
norm = Math.sqrt(norm);
float[] out = new float[v.length];
for (int i = 0; i < v.length; i++) out[i] = (float) (v[i] / norm);
return out;
}
private static double dot(float[] a, float[] b) {
double sum = 0;
for (int i = 0; i < a.length; i++) sum += a[i] * b[i];
return sum;
}
}
This brute-force search is O(n) per query. For production, use an approximate nearest neighbor index. HNSW gives low latency with high recall, and most systems offer it:
-
pgvector in PostgreSQL, if you already run Postgres and have up to a few million chunks. Use an HNSW index and a
WHERE tenant_id = ?clause. -
OpenSearch or Elasticsearch with
knn_vectorfields, when you need hybrid search with BM25 and metadata filters at scale. - Dedicated vector databases such as Qdrant, Milvus, or Pinecone, when vector volume dominates.
The interface above stays the same. Only the implementation changes.
Stage 5: Retrieval Evaluation
This is the stage most teams skip, and it is the one that tells you whether the system works. Without evaluation you are guessing.
Build a small evaluation set first: 50 to 200 real questions, each paired with the chunk IDs that contain the answer. You can write them by hand or generate candidates with an LLM and then review them. Then measure two metrics:
- Recall@k: the fraction of questions where at least one relevant chunk appears in the top k results.
- Mean Reciprocal Rank (MRR): how high the first relevant result ranks, averaged across questions. It rewards putting the right chunk first.
public record EvalCase(String question, Set<String> relevantChunkIds) {}
public record RetrievalMetrics(double recallAtK, double mrr, int k, int cases) {}
public final class RetrievalEvaluator {
private final EmbeddingModel embeddings;
private final VectorStore store;
public RetrievalEvaluator(EmbeddingModel embeddings, VectorStore store) {
this.embeddings = embeddings;
this.store = store;
}
public RetrievalMetrics evaluate(List<EvalCase> cases, String tenantId, int k) {
int hits = 0;
double reciprocalRankSum = 0;
for (EvalCase c : cases) {
float[] query = embeddings.embed(List.of(c.question())).get(0);
List<ScoredChunk> results = store.search(query, k, tenantId, Map.of());
int firstRelevantRank = -1;
for (int i = 0; i < results.size(); i++) {
if (c.relevantChunkIds().contains(results.get(i).chunk().id())) {
firstRelevantRank = i + 1;
break;
}
}
if (firstRelevantRank > 0) {
hits++;
reciprocalRankSum += 1.0 / firstRelevantRank;
}
}
return new RetrievalMetrics(
(double) hits / cases.size(),
reciprocalRankSum / cases.size(),
k, cases.size());
}
}
Use this evaluator to compare configurations. Run it once with chunks of 500 characters, once with 800, and once with 1,200. Change the embedding model and run it again. Keep the change that improves recall without hurting the others. Store each run's metrics with the configuration that produced them, so you can explain later why the system behaves as it does.
Retrieval metrics do not measure answer quality. Once retrieval is solid, add end-to-end evaluation with answer faithfulness (is every claim supported by the retrieved context?) and answer relevance. Frameworks such as Ragas or LLM-as-judge setups automate this, but they need the same discipline: a labeled set and a fixed baseline.
Stage 6: Re-Ranking
Vector search is fast but approximate. It compares the query and each chunk separately, so it cannot judge how well a specific passage answers a specific question. A re-ranker can. Cross-encoder models read the query and the candidate together and produce a relevance score. They are slower, so you apply them to a short list.
The standard pattern is to retrieve a generous candidate set, for example the top 50 from vector search, then re-rank and keep the top 5 for the prompt.
public interface Reranker {
List<ScoredChunk> rerank(String query, List<ScoredChunk> candidates, int topN);
}
public final class CrossEncoderReranker implements Reranker {
private final HttpClient http = HttpClient.newHttpClient();
private final URI endpoint;
private final ObjectMapper json = new ObjectMapper();
public CrossEncoderReranker(URI endpoint) {
this.endpoint = endpoint;
}
@Override
public List<ScoredChunk> rerank(String query, List<ScoredChunk> candidates, int topN) {
if (candidates.isEmpty()) return candidates;
try {
List<String> docs = candidates.stream().map(c -> c.chunk().text()).toList();
String body = json.writeValueAsString(Map.of("query", query, "documents", docs));
HttpRequest request = HttpRequest.newBuilder(endpoint)
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(body))
.build();
HttpResponse<String> response =
http.send(request, HttpResponse.BodyHandlers.ofString());
JsonNode results = json.readTree(response.body()).get("results");
List<ScoredChunk> reranked = new ArrayList<>();
for (JsonNode r : results) {
int index = r.get("index").asInt();
reranked.add(new ScoredChunk(candidates.get(index).chunk(),
r.get("relevance_score").asDouble()));
}
reranked.sort(Comparator.comparingDouble(ScoredChunk::score).reversed());
return reranked.subList(0, Math.min(topN, reranked.size()));
} catch (IOException | InterruptedException e) {
if (e instanceof InterruptedException) Thread.currentThread().interrupt();
// Degrade gracefully: fall back to vector order if the re-ranker is unavailable.
return candidates.subList(0, Math.min(topN, candidates.size()));
}
}
}
The fallback matters. A re-ranker is an extra network dependency, and the system should still answer if it fails, just with slightly lower precision. Measure the re-ranker with the same evaluator from stage 5. Compare recall@5 and MRR with and without it. If the gain is small, the extra latency and cost may not be worth it.
Putting It Together: The Query Path
With the stages in place, the query path is short:
public final class RagService {
private final EmbeddingModel embeddings;
private final VectorStore store;
private final Reranker reranker;
private final ChatClient llm;
public RagService(EmbeddingModel embeddings, VectorStore store,
Reranker reranker, ChatClient llm) {
this.embeddings = embeddings;
this.store = store;
this.reranker = reranker;
this.llm = llm;
}
public Answer answer(String tenantId, String question) {
float[] query = embeddings.embed(List.of(question)).get(0);
List<ScoredChunk> candidates = store.search(query, 50, tenantId, Map.of());
List<ScoredChunk> top = reranker.rerank(question, candidates, 5);
String context = top.stream()
.map(c -> "[" + c.chunk().id() + "]\n" + c.chunk().text())
.collect(Collectors.joining("\n\n"));
String prompt = """
Answer the question using only the context below.
Cite the chunk IDs in square brackets for every claim.
If the context does not contain the answer, say so.
Context:
%s
Question: %s
""".formatted(context, question);
String response = llm.prompt(prompt).call().content();
return new Answer(response, top.stream().map(c -> c.chunk().id()).toList());
}
}
public record Answer(String text, List<String> sourceChunkIds) {}
Three details make this work in practice. The prompt tells the model to cite and to admit when the answer is missing, which is the main defense against hallucination. The source IDs are returned separately so the application can show links to the original documents. And the tenant ID flows from the authenticated request into the search call, never from the user's text.
Best Practices
Keep metadata rich. Store the tenant, source, page, section, document date, and version on every chunk. You will use these for filtering, citations, and debugging.
Version everything. Record the chunking parameters, embedding model, and re-ranker used for each index. When quality changes, you need to know what changed.
Re-index incrementally. Use the stable document IDs from stage 1. When a file changes, delete its old chunks and insert the new ones instead of rebuilding the whole index.
Measure before optimizing. Tune chunking and retrieval with your evaluation set, not with a handful of questions you happen to try.
Watch the context budget. More chunks are not always better. Five precise passages usually beat fifteen loosely related ones, and they cost less.
Handle access control in the store. Tenant and permission filters belong in the vector query. Post-filtering in application code is a data leak waiting to happen.
Common Pitfalls
- Chunks that lack context. A chunk that says "the limit is 500" is useless without the product it refers to. Prepend the document title or section heading to each chunk before embedding.
- Mixed embedding models. Querying an index built with a different model returns nonsense without any error. Store and check the model ID.
- Evaluating only with easy questions. Include paraphrased, multi-part, and ambiguous questions in your set, and include some questions the corpus cannot answer.
- Ignoring tables and code. Plain-text extraction often destroys tables. Parse them into structured text or Markdown before chunking.
Key Takeaways
- RAG is two pipelines sharing one index: indexing (ingest, chunk, embed, store) and querying (embed, retrieve, re-rank, generate).
- Chunking and embedding decide most of the quality, so tune them with measurements.
- Build the vector store interface with tenant filters inside the query, not after it.
- Evaluate retrieval with recall@k and MRR on a labeled set before you judge the LLM's answers.
- Use a cross-encoder re-ranker over a generous candidate set, and keep a fallback if it fails.
- Record the configuration behind every index so results are reproducible.
The full pipeline above is a foundation, not a finished product. The production version swaps the in-memory store for pgvector or OpenSearch, adds document parsers for your formats, and runs the evaluator in CI so a change that hurts retrieval quality fails the build. Start with evaluation. Everything else is easier to improve once you can measure it.
Top comments (0)