author: "RAJश्री"
A practical guide to diagnosing and improving Retrieval-Augmented Generation systems
Most RAG systems don't fail because the LLM can't answer the question. They fail because the LLM was given the wrong information to begin with.
RAG has become one of the most common architectures for building applications around Large Language Models.
The basic idea is simple:
User Question
↓
Search Knowledge Base
↓
Retrieve Relevant Information
↓
Give Context to LLM
↓
Generate Answer
It sounds straightforward.
And in a demo, it often works beautifully.
Upload a few PDFs.
Create embeddings.
Put them into a vector database.
Ask a question.
Get an answer.
Then reality arrives.
You add hundreds or thousands of documents.
Some documents are long.
Some contain tables.
Some are outdated.
Some contain similar information.
Some belong to different departments.
Users ask questions in completely different ways than the documents are written.
Suddenly the system starts producing answers like:
"I couldn't find relevant information."
Or worse:
A confident answer based on completely irrelevant context.
At this point, many teams blame the LLM.
They change the model.
They increase the context window.
They modify the system prompt.
They try a more expensive model.
But the real problem is often much earlier in the pipeline.
User Query
↓
Query Processing
↓
Retrieval
↓
Filtering
↓
Ranking
↓
Context Construction
↓
LLM
↓
Answer
If the wrong information enters the context, the LLM is already starting from a disadvantage.
That leads to a fundamental principle:
Garbage in, grounded garbage out.
RAG quality is therefore not just an LLM problem.
It is a retrieval engineering problem.
1. First, Understand What RAG Actually Does
Retrieval-Augmented Generation combines two separate capabilities:
Retrieval
+
Generation
The retrieval system finds information.
The language model uses that information to generate an answer.
A simplified architecture looks like this:
Knowledge Base
│
↓
Document Loader
│
↓
Chunking
│
↓
Embeddings
│
↓
Vector Store
│
│
User Query ───────────┘
↓
Query Embedding
↓
Similarity Search
↓
Relevant Chunks
↓
Context
↓
LLM
↓
Answer
The LLM does not magically search your entire knowledge base.
It receives the context selected by your retrieval system.
That means the final answer depends heavily on what happened before the LLM was called.
A useful mental model is:
Final Answer Quality
≈
Retrieval Quality
×
Context Quality
×
Generation Quality
This isn't a mathematical law.
It's an engineering intuition.
If retrieval quality is close to zero, a powerful model cannot completely compensate for it.
2. The Most Common RAG Mistake: "Just Put Everything in a Vector Database"
One of the most common beginner architectures looks like this:
Documents
↓
Split Every N Characters
↓
Generate Embeddings
↓
Vector Database
↓
Top-K Search
↓
LLM
Technically, this is RAG.
But production-quality RAG requires much more thought.
Consider a 100-page employee policy document.
Suppose it contains:
Leave Policy
Travel Policy
Medical Reimbursement
Work From Home
Performance Reviews
Promotion Policy
Resignation
Notice Period
Termination
If you blindly split the document every 500 characters, you may end up with chunks like:
Chunk 1:
"...employees may apply for leave..."
Chunk 2:
"...approval from the reporting manager..."
Chunk 3:
"...subject to organizational requirements..."
The chunks are technically valid pieces of text.
But they may no longer represent meaningful units of knowledge.
The retrieval system doesn't understand the document's original structure unless you preserve that structure.
This is where the first major RAG engineering problem begins.
3. Chunking Is Not a Preprocessing Detail
Chunking is often treated as a boring preprocessing step.
It shouldn't be.
Chunking determines the units that your retrieval system can discover.
Think of it this way:
Your retriever cannot retrieve what your indexing strategy failed to represent properly.
Suppose a document contains:
Section:
Enterprise Refund Policy
Rule:
Enterprise customers can request a refund within 30 days.
Exception:
Annual contracts require account-manager approval.
Restriction:
Refunds cannot be issued after service termination.
A naive chunking strategy might separate the rule from the exception.
Then a user asks:
"Can an enterprise customer get a refund after termination?"
The retriever may return:
Enterprise customers can request a refund within 30 days.
That sentence looks relevant.
But the important restriction may exist in another chunk.
The problem isn't necessarily the embedding model.
The problem is context fragmentation.
4. Good Chunking Preserves Meaning
There is no universally perfect chunk size.
A useful chunking strategy depends on the type of content.
For example:
Documentation
Chunk around:
- headings
- sections
- subsections
- procedures #### Legal documents
Preserve:
- clauses
- sections
- definitions
- exceptions #### Technical documentation
Preserve:
- concepts
- code examples
- configuration instructions
- troubleshooting sections #### FAQs
A question and its answer should usually remain together.
Tables
Treating every row as meaningless text can destroy relationships between columns.
The goal isn't:
"Create chunks of exactly 500 tokens."
The goal is:
Create retrievable units that preserve enough semantic context to answer real questions.
5. The Chunk Size Trade-Off
Chunk size creates a fundamental trade-off.
Chunks that are too small
You get:
High precision
+
Low context
The retrieved information may be very specific but incomplete.
For example:
"Requires manager approval."
What requires approval?
A refund?
Leave?
Travel?
The chunk doesn't tell you.
Chunks that are too large
You get:
More context
+
More noise
The relevant information may be buried inside thousands of unrelated tokens.
The LLM now has to process unnecessary information.
Therefore:
The best chunk isn't the smallest chunk or the largest chunk. It's the smallest meaningful unit that retains the context required for the task.
6. Chunking Strategies Worth Knowing
There are several approaches.
Fixed-size chunking
Document
↓
Every N tokens
Simple.
Fast.
Easy to implement.
Useful for prototypes.
But often insufficient for complex documents.
Recursive chunking
The system attempts to split content using progressively smaller separators.
Conceptually:
Document
↓
Paragraph
↓
Sentence
↓
Word
This often preserves structure better than completely fixed-size splitting.
Structure-aware chunking
Use the document's natural structure:
Document
├── Chapter
│ ├── Section
│ │ ├── Paragraph
│ │ └── Paragraph
│ └── Section
└── Chapter
This is often more appropriate for documentation, manuals, policies, and technical content.
Parent-child retrieval
A useful strategy is to retrieve a smaller child chunk while providing its larger parent context.
Parent Section
│
┌────┼────┐
↓ ↓ ↓
C1 C2 C3
Search may identify C2.
But the system can return:
Parent Section + C2
This gives retrieval precision without completely sacrificing context.
7. Embeddings Are Not a Search Engine
This is another important misconception.
Many developers think:
"Once I have embeddings, semantic search will understand everything."
Not exactly.
An embedding represents semantic information numerically.
Conceptually:
"How can I reset my password?"
↓
Embedding Model
↓
[0.021, -0.182, 0.441, ...]
A document chunk also becomes a vector.
"Password reset instructions..."
↓
Embedding Model
↓
[0.019, -0.176, 0.438, ...]
The retrieval system compares these vectors using a similarity metric.
Common approaches include:
- cosine similarity
- dot product
- Euclidean distance
The important point is:
Semantic similarity is not the same thing as relevance.
Two pieces of text can be semantically similar but still answer different questions.
8. Similar Does Not Always Mean Relevant
Imagine a knowledge base contains:
Password Reset
Password Security
Password Expiration
Password Recovery
Password Policy
User asks:
"How do I reset my password?"
A semantic search system might retrieve:
Password Security
Password Policy
Password Expiration
Password Reset
Several results are semantically related.
But only one may contain the exact procedure.
This is why modern retrieval systems often need more than one retrieval mechanism.
9. Vector Search vs Keyword Search
Keyword search is often underestimated.
Suppose a user asks:
"What is the SLA for ticket P1-8472?"
A semantic search system might focus on the general concept of support SLAs.
But the identifier:
P1-8472
is extremely important.
Keyword or lexical search can handle exact identifiers much better.
This leads to an important insight:
Semantic search understands meaning. Keyword search understands exact terms.
They complement each other.
10. Hybrid Search
A practical retrieval architecture often combines:
User Query
│
┌──────────┴──────────┐
↓ ↓
Semantic Search Keyword Search
│ │
└──────────┬──────────┘
↓
Result Fusion
↓
Reranking
↓
Final Context
Semantic retrieval can find conceptual matches.
Keyword retrieval can find exact:
- product names
- ticket IDs
- error codes
- employee IDs
- policy numbers
- technical terms
Together they can outperform either approach alone for many enterprise-style workloads.
11. Metadata Can Be More Important Than Another Embedding
Consider a knowledge base containing:
Department:
Finance
HR
Engineering
Legal
Region:
India
US
Europe
Document Type:
Policy
Guide
FAQ
Contract
Version:
2024
2025
2026
Now the user asks:
"What is the current travel reimbursement policy for employees in India?"
The system shouldn't search the entire database equally.
Metadata can narrow the search:
department = Finance
region = India
document_type = Policy
version = current
Then semantic retrieval runs over a much smaller and more relevant candidate set.
This can dramatically improve retrieval quality.
12. Metadata Is Part of Your Retrieval Architecture
Useful metadata might include:
{
"department": "finance",
"region": "india",
"document_type": "policy",
"version": "2026",
"access_level": "employee",
"source": "official_policy",
"updated_at": "2026-07-12"
}
Now retrieval can become:
User Query
↓
Metadata Filtering
↓
Semantic / Keyword Search
↓
Reranking
↓
Context
Instead of:
User Query
↓
Search Everything
This is one of the simplest ways to improve a RAG system.
13. Query Rewriting: The User Doesn't Always Ask the Right Question
Users rarely write queries in the same language as your documents.
Knowledge base:
"Employee reimbursement eligibility for domestic business travel"
User:
"Can I claim hotel expenses when I travel for work?"
These are conceptually related.
But real-world queries can be much more ambiguous.
For example:
"What happens if I cancel it?"
What does "it" mean?
The system may need conversation history or query rewriting.
A query transformation layer can turn:
"What happens if I cancel it?"
into something like:
"What is the cancellation policy for the user's current subscription?"
Now retrieval has a much clearer target.
14. Query Expansion
Sometimes one query isn't enough.
Suppose the user asks:
"How do I recover my account?"
The system might generate related search formulations:
account recovery
password recovery
account access restoration
forgot password procedure
login recovery
These can be searched separately and combined.
This can improve recall.
But query expansion also introduces noise.
Therefore:
More queries do not automatically mean better retrieval.
They need to be evaluated.
15. Top-K Is Not a Magic Number
You will often see:
top_k = 5
or:
top_k = 10
But why 5?
Why not 3?
Why not 20?
There is no universal answer.
If K is too small:
Relevant Information
↓
Not Retrieved
If K is too large:
Relevant Information
+
Noise
+
Conflicting Information
+
Redundant Chunks
The LLM receives too much context.
So the correct question isn't:
"What is the best K?"
It is:
"How many retrieved items provide sufficient evidence without introducing unnecessary noise?"
That number should come from evaluation.
16. Reranking: The Missing Layer in Many RAG Systems
Initial retrieval is often optimized for speed.
The first stage might retrieve:
Top 20 or Top 50 candidates
Then a reranker evaluates those candidates more carefully.
Conceptually:
User Query
↓
Fast Retrieval
↓
Top 50 Candidates
↓
Reranker
↓
Top 5 Relevant Results
↓
LLM
This is a classic two-stage retrieval architecture.
Stage 1: Recall
Find enough potentially relevant documents.
Stage 2: Precision
Determine which candidates are actually most useful.
This distinction is extremely important.
The first retriever should find candidates. The reranker should help decide which candidates deserve attention.
17. Retrieval Recall and Retrieval Precision
Two useful concepts are:
Recall
How much of the relevant information did we successfully retrieve?
If 5 relevant chunks exist and we retrieve only 1:
Recall = poor
Precision
How much of what we retrieved is actually relevant?
If we retrieve 20 chunks and only 2 are useful:
Precision = poor
A good retrieval system needs a useful balance.
High Recall
+
High Precision
=
Useful Context
This is why simply increasing top_k isn't a reliable solution.
18. The Context Window Is Not a Dumpster
A common reaction to poor retrieval is:
"Let's just send more context to the LLM."
This sounds reasonable.
But context has costs.
More context can mean:
- higher latency
- higher token usage
- higher cost
- more irrelevant information
- conflicting information
- harder reasoning
The goal isn't to maximize context.
The goal is to maximize useful context.
A better architecture is:
Retrieve
↓
Filter
↓
Rerank
↓
Compress
↓
Construct Context
↓
LLM
19. Context Compression
Suppose retrieval returns:
20 chunks
But only a few sentences from those chunks actually answer the question.
Context compression can reduce the payload before it reaches the model.
Conceptually:
20 Retrieved Chunks
↓
Relevant Information
↓
Compressed Context
↓
LLM
This can help reduce:
- token usage
- latency
- noise
But compression itself must be evaluated carefully.
If the compressor removes an important exception, the final answer can become wrong.
20. Conflicting Documents Are a Real Problem
Imagine your knowledge base contains:
Refund Policy — 2024
Refund Policy — 2025
Refund Policy — 2026
The user asks:
"What is our current refund policy?"
If retrieval returns all three equally, the model may combine them.
Now you have a retrieval problem, not necessarily a model problem.
The system should understand:
2026 > 2025 > 2024
or, better, use explicit metadata such as:
status = current
This is why document freshness and versioning are critical.
21. Your Documents Need Governance Too
RAG quality depends heavily on knowledge quality.
If your knowledge base contains:
- outdated documents
- duplicate documents
- contradictory policies
- broken OCR
- missing sections
- incorrect metadata
then improving your LLM may not solve the problem.
A strong RAG pipeline therefore begins before embeddings.
Raw Documents
↓
Validation
↓
Cleaning
↓
Deduplication
↓
Structure Extraction
↓
Metadata
↓
Chunking
↓
Embedding
↓
Indexing
This is why:
RAG is as much a data engineering problem as it is an AI problem.
22. PDF Doesn't Mean Knowledge
PDFs are especially dangerous for naive RAG pipelines.
A PDF may visually contain:
Heading
Paragraph
Table
Image
Footnote
Header
Footer
Page Number
But text extraction might produce:
Page number
Footer
Column 2
Column 1
Header
Random text
The visual structure humans understand may disappear.
If the ingestion pipeline destroys structure, the retrieval pipeline inherits the damage.
Therefore:
Before asking an LLM to understand your documents, make sure your ingestion system understands the documents first.
23. Tables Are a Special Retrieval Problem
Consider:
Product | Region | Price | Discount
------------------------------------
A | India | $100 | 10%
B | India | $200 | 15%
C | US | $150 | 5%
Flattening this into plain text may preserve some information.
But questions involving relationships across columns can become difficult.
For example:
"What is the discounted price of Product B in India?"
Depending on the parser and chunking strategy, the model may receive incomplete or incorrectly ordered information.
For structured data, it may be better to use:
SQL
+
Structured Retrieval
rather than forcing everything through vector search.
This leads to another important principle:
Not every piece of enterprise knowledge belongs in a vector database.
24. RAG vs SQL vs APIs
Consider three questions.
Question 1
"What does the refund policy say?"
RAG is appropriate.
Question 2
"How many refunds were processed last month?"
SQL is probably more appropriate.
Question 3
"Cancel this customer's subscription."
An API or business service should execute the action.
A mature AI system might therefore use:
User Question
↓
Intent Router
↓
┌───────────────┼───────────────┐
↓ ↓ ↓
RAG SQL API
Knowledge Data Action
│ │ │
└───────────────┼───────────────┘
↓
LLM
The key is not:
"How do I put everything into RAG?"
It is:
"What is the correct source of truth for this question?"
25. The Source of Truth Matters
For every piece of information, ask:
Where does the authoritative version actually live?
For example:
Company Policy
↓
Document Repository
Current Customer Balance
↓
Database
Current Order Status
↓
Order API
Employee Permission
↓
Identity / Access System
If the AI retrieves a stale PDF to answer a question that should have been answered from a live database, the architecture is wrong.
Not the model.
Not the embedding.
The architecture.
26. Security: Retrieval Must Respect Permissions
This is one of the most important production concerns.
Suppose the knowledge base contains:
Public Documents
Internal Documents
Finance Documents
Executive Documents
A user from Engineering asks a question.
If your retrieval system searches everything and simply asks the LLM not to reveal sensitive information, you have a serious security problem.
The model should not receive unauthorized information in the first place.
A safer flow is:
User Identity
↓
Authorization
↓
Allowed Data Scope
↓
Retrieval
↓
Context
↓
LLM
Authorization should be enforced at the data access layer.
Prompt instructions are not a replacement for access control.
27. The RAG Pipeline Should Be Treated Like Software
One of the biggest mindset shifts for AI engineers is this:
A RAG system is not a prompt. It is a software system.
That means it needs:
- version control
- tests
- observability
- monitoring
- logging
- evaluation
- failure handling
- security
- performance optimization
You should be able to answer:
Why did this document get retrieved?
Why wasn't another document retrieved?
Which query was actually searched?
Which filters were applied?
What ranking score did the result receive?
What context reached the model?
Which model generated the answer?
Without this information, debugging becomes guesswork.
28. Stop Evaluating RAG With Five Questions
Another common mistake is testing a RAG system manually with a handful of questions.
For example:
Question 1 → Looks good
Question 2 → Looks good
Question 3 → Looks good
Question 4 → Looks good
Question 5 → Looks good
Then:
"Our RAG works."
It doesn't prove much.
Real users will ask:
- short questions
- long questions
- ambiguous questions
- misspelled questions
- multi-part questions
- follow-up questions
- questions with identifiers
- questions requiring multiple documents
You need a proper evaluation dataset.
29. Build a Retrieval Evaluation Dataset
A useful evaluation dataset can contain:
{
"question": "What is the enterprise refund period?",
"expected_sources": [
"refund-policy-2026"
],
"expected_answer": "Enterprise customers can request..."
}
Now you can evaluate the retrieval system independently from the LLM.
This distinction is critical.
30. Evaluate Retrieval Before Generation
Suppose the final answer is wrong.
There are two possibilities:
Wrong Answer
↓
┌───────────────┐
│ │
Retrieval Generation
Problem Problem
If the correct document was never retrieved, changing the prompt may accomplish very little.
Therefore debug in this order:
1. Was the right source retrieved?
2. Was the relevant chunk retrieved?
3. Was it ranked highly enough?
4. Did the final context contain the necessary evidence?
5. Did the LLM interpret the evidence correctly?
6. Did the answer follow the required format?
This dramatically reduces blind debugging.
31. Observability: See What the AI Actually Saw
Suppose the user asks:
"What is the cancellation policy?"
Your logs should ideally let you inspect:
Query
↓
Normalized Query
↓
Metadata Filters
↓
Retrieved Candidates
↓
Scores
↓
Reranked Results
↓
Final Context
↓
Model
↓
Response
This gives you a trace.
Without it, you only see:
Question → Bad Answer
and have no idea where the failure occurred.
32. RAG Failures Can Be Classified
A useful debugging framework is to categorize failures.
Retrieval failure
The correct information wasn't found.
Query
↓
Wrong Documents
Ranking failure
The correct information was found but ranked too low.
Retrieved
↓
Buried Under Irrelevant Results
Context failure
The relevant information was retrieved but wasn't included in the final context.
Generation failure
The correct evidence was provided, but the model misunderstood or ignored it.
Data failure
The source itself is outdated, incomplete, or contradictory.
Permission failure
The system retrieved information the user should not have been able to access.
This classification makes debugging much more systematic.
33. A Better Production Retrieval Pipeline
A mature retrieval flow might look like:
User Query
│
↓
Query Understanding
│
↓
Query Rewriting
│
↓
Permission Filter
│
↓
Metadata Filtering
│
┌──────────┴──────────┐
↓ ↓
Semantic Search Keyword Search
│ │
└──────────┬──────────┘
↓
Result Fusion
↓
Reranking
↓
Deduplication
↓
Context Compression
↓
Context Builder
↓
LLM
↓
Validation
↓
Answer
Not every application needs every layer.
That is important.
The architecture should be driven by the problem.
34. Don't Build the Most Complicated RAG System First
There is another trap.
After reading about advanced RAG architectures, developers sometimes immediately build:
Query Rewriting
+
Hybrid Search
+
Reranking
+
Knowledge Graph
+
Agent
+
Memory
+
Multiple Models
+
Complex Routing
before understanding whether the basic problem exists.
That's unnecessary complexity.
A better progression is:
Level 1
Basic retrieval
↓
Measure
↓
Identify failure
↓
Improve chunking
↓
Measure again
↓
Add metadata
↓
Measure again
↓
Add hybrid retrieval
↓
Measure again
↓
Add reranking
↓
Measure again
Add complexity because the evaluation says you need it—not because the architecture diagram looks impressive.
35. A Practical RAG Improvement Checklist
When a RAG system performs poorly, walk through this checklist.
Data
- Are the documents authoritative?
- Are they current?
- Are duplicates removed?
- Are conflicting versions identified?
- Is the text extraction reliable? #### Chunking
- Does each chunk preserve meaning?
- Are headings preserved?
- Are exceptions kept with their rules?
- Are tables handled correctly?
- Is chunk size appropriate for the content? #### Metadata
- Can results be filtered by source?
- Version?
- Region?
- Department?
- Document type?
- Access level? #### Retrieval
- Is semantic search sufficient?
- Are exact keywords important?
- Should hybrid search be used?
- Is query rewriting necessary? #### Ranking
- Are relevant documents appearing near the top?
- Would reranking improve precision? #### Context
- Is too much information being passed?
- Is important information being truncated?
- Are duplicate chunks removed? #### Security
- Is retrieval permission-aware?
- Can users access only authorized data? #### Evaluation
- Do you have representative test questions?
- Are retrieval metrics measured?
- Are generation metrics measured?
- Are failures categorized? #### Production
- Is latency measured?
- Is token usage monitored?
- Are retrieval traces available?
- Are failures observable?
36. The Most Important Lesson
If there is one idea worth remembering from this entire article, it is this:
Don't immediately blame the model when your RAG system produces a bad answer.
Trace the pipeline.
Bad Answer
↓
What context did the model receive?
↓
Was that context relevant?
↓
Why was it retrieved?
↓
Was the query transformed correctly?
↓
Were permissions and metadata applied?
↓
Was the correct document indexed?
↓
Was the document chunked correctly?
↓
Was the source itself correct?
You may discover that the LLM was never the primary problem.
37. RAG Is a Retrieval System Before It Is a Generation System
The name itself tells us:
Retrieval-Augmented Generation.
Generation gets most of the attention because users see the final answer.
But retrieval determines what evidence the model gets.
That means a useful architectural principle is:
Better Retrieval
↓
Better Context
↓
Better Grounding
↓
More Reliable Generation
Not always.
But often enough to make retrieval one of the first things worth investigating.
38. Where RAG Is Going
The future of RAG isn't simply:
Vector Database + LLM
It is becoming a broader retrieval and reasoning architecture:
User
│
↓
Query Understanding
│
↓
Intent Routing
│
┌──────────┼──────────┐
↓ ↓ ↓
RAG SQL APIs
│ │ │
↓ ↓ ↓
Documents Live Data Actions
│ │ │
└──────────┼──────────┘
↓
Reasoning
↓
Verification
↓
Answer
RAG becomes one component inside a larger AI system.
And that's the direction modern AI engineering is moving toward.
39. Final Takeaway
Your RAG system probably doesn't need a more expensive model first.
It may need a better retrieval pipeline.
Before changing the LLM, ask:
Are my documents clean?
↓
Is my chunking meaningful?
↓
Is my metadata useful?
↓
Is my retrieval strategy appropriate?
↓
Do I need hybrid search?
↓
Would reranking help?
↓
Is the query being understood correctly?
↓
Is the context actually relevant?
↓
Are permissions enforced?
↓
Can I measure retrieval quality?
Because a RAG application is only as good as the information it puts in front of the model.
The model can reason over the context you provide.
It cannot retrieve the document you failed to index.
It cannot use the information you filtered out.
It cannot magically reconstruct an exception that your chunking separated.
And it cannot turn outdated knowledge into authoritative knowledge.
So the next time your RAG application gives a terrible answer, don't immediately say:
"The LLM is bad."
Instead, ask:
"What exactly did we retrieve?"
That question often leads you much closer to the real problem.
The Practical RAG Mental Model
Remember this pipeline:
┌─────────────────────┐
│ User Query │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Query Understanding │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Metadata / Security │
└──────────┬──────────┘
↓
┌────────────┴────────────┐
↓ ↓
Semantic Search Keyword Search
│ │
└────────────┬────────────┘
↓
Result Fusion
↓
Reranking
↓
Deduplication
↓
Context Construction
↓
LLM
↓
Verification
↓
Response
And behind all of this:
Good Data
+
Good Chunking
+
Good Retrieval
+
Good Ranking
+
Good Context
+
Good Evaluation
=
Reliable RAG
RAG isn't broken.
The retrieval pipeline probably needs engineering.
Key Takeaways
- RAG quality starts before the LLM.
- Chunking is an architectural decision, not just preprocessing.
- Semantic similarity does not always mean relevance.
- Keyword search still matters for exact terms, IDs, codes, and names.
- Hybrid retrieval can combine semantic and lexical strengths.
- Metadata filtering can dramatically reduce irrelevant retrieval.
- Query rewriting can help when user language differs from knowledge-base language.
- top_k should be determined through evaluation, not guesswork.
- Reranking can improve precision after broad candidate retrieval.
- More context is not automatically better context. Outdated and conflicting documents can cause retrieval failures.
- Tables and structured data may require SQL or specialized extraction rather than plain vector search.
- Authorization must be enforced before sensitive information reaches the model.
- RAG should be evaluated as a software system, not just through a few manual prompts.
- Retrieval failures and generation failures should be debugged separately.
- Observability is essential for understanding what the model actually received. -The right question is not "Which LLM should I use?" but often "Did I retrieve the right evidence?"
About the Author
RAJश्री
Software Developer → AI Researcher | Founder, Shree Labs
Rajshree is a Software Engineer focused on building modern software systems while exploring Artificial Intelligence, Machine Learning, LLMs, and AI Engineering.
Through Shree Labs, a growing technology and knowledge platform, he explores and publishes work around technology, projects, research, technical articles, and practical learning—with a focus on understanding how modern technology actually works beneath the surface.
His approach to AI is rooted in software engineering fundamentals:
Don't just make AI work in a demo. Understand the system behind it.
🌐 Portfolio: https://rjshree.com
💼 LinkedIn: https://linkedin.com/in/rjshree
Shree Labs
Shree Labs is a technology and knowledge platform featuring technical articles, projects, tutorials, research work, and poetry—bringing together technology, engineering, learning, and ideas under one platform.
If you enjoyed this article, follow along for more practical writing on Software Engineering, AI, LLMs, Machine Learning, and the evolution of modern technology.
Thanks for reading.
Top comments (0)