From PDF to Evidence: Designing a Production-Ready RAG Pipeline
RAG is often described as a simple architecture:
Documents → Embeddings → Vector Database → LLM
That diagram is useful for understanding the concept, but it is far from enough for a production document-processing system.
When the source documents are PDFs containing tables, scanned pages, headings, footnotes, forms, and complex layouts, the real challenge is not simply retrieving similar text.
The real challenge is:
Can the system generate an answer that can be traced back to the correct evidence in the original document?
This article describes the architecture and engineering considerations behind a production-oriented RAG pipeline for document processing.
1. The Real Problem Starts Before RAG
Consider a system where users upload contracts, invoices, reports, or other business documents.
A typical workflow looks like this:
PDF
↓
Document Intelligence / OCR
↓
Layout Analysis
↓
Page-aware Document Structure
↓
Chunking + Metadata
↓
Search Index
↓
Hybrid Retrieval
↓
LLM
↓
Structured Finding
↓
Evidence + Citation
The important observation is that retrieval quality depends heavily on document processing quality.
If the original PDF is poorly converted into text, even the best embedding model cannot recover information that was lost during extraction.
2. PDF Extraction Is More Than OCR
A PDF is not necessarily a collection of plain text.
It can contain:
- Text
- Tables
- Images
- Scanned pages
- Headers and footers
- Multiple columns
- Forms
- Signatures
- Page numbers
- Different visual sections
Simply extracting all text and splitting it every 500 tokens can destroy important relationships.
For example:
Invoice Amount: ¥12,500,000
Tax: ¥1,250,000
Total: ¥13,750,000
If these values are separated incorrectly during chunking, the LLM may retrieve only part of the information.
Therefore, the ingestion pipeline should preserve document structure whenever possible.
3. Keep Page Information
One of the most important pieces of metadata in a document RAG system is the source page.
Instead of storing only:
{
"text": "The contract amount is..."
}
I prefer a structure closer to:
{
"text": "The contract amount is...",
"document_id": "contract-001",
"page_number": 14,
"section": "Contract Amount",
"chunk_id": "contract-001-page14-chunk03"
}
This gives the retrieval system additional context and, more importantly, allows the generated result to point back to the original source.
For reviewed business documents, this distinction is extremely important.
A generated statement without evidence is difficult to trust.
A generated statement with:
Finding:
The contract amount exceeds the configured threshold.
Evidence:
Contract.pdf — Page 14
is much easier for a human reviewer to validate.
4. Chunking Should Follow Meaning
A common approach is to split documents by a fixed number of tokens.
For example:
Every 500 tokens → one chunk
This is simple, but document structure can be more important than chunk size.
A better strategy can combine:
- Headings
- Paragraph boundaries
- Table boundaries
- Sections
- Page boundaries
- Semantic relationships
For example:
Page 14
├── Contract Overview
│ ├── Contractor
│ ├── Contract Number
│ └── Contract Amount
│
└── Payment Terms
├── Payment Schedule
└── Conditions
The resulting chunks carry both the content and its context.
5. Hybrid Search Is Often More Practical
Pure vector search is powerful for semantic similarity.
However, business documents often contain exact identifiers:
ABC-2026-00125
¥13,750,000
Project ID: PJ-10293
These values may be better handled by keyword or exact matching.
This is where hybrid search becomes useful.
Conceptually:
User Query
│
├───────────────┐
↓ ↓
Keyword Search Vector Search
│ │
└───────┬───────┘
↓
Result Ranking
↓
Top Evidence
Keyword search can capture exact terms.
Vector search can capture semantic meaning.
Combining them gives the system more flexibility across different document types and query patterns.
6. Retrieval Is Not the End
A common mistake is to evaluate a RAG system only by asking:
“Does the LLM produce a good answer?”
That is too late in the pipeline.
We should separately evaluate retrieval.
For example:
Retrieval metrics
Precision
How many retrieved documents are actually relevant?
Recall
How many of the relevant documents did we successfully retrieve?
MRR (Mean Reciprocal Rank)
How high does the first relevant result appear?
These metrics help identify whether a problem comes from:
Retrieval
↓
Context construction
↓
Prompt
↓
LLM generation
Without separating these stages, it becomes difficult to determine why the system is failing.
7. Structured Output Makes Validation Easier
For business applications, free-form LLM responses can be difficult to validate.
Instead of asking the model to return:
I found that the amount appears to exceed...
we can define a structured schema:
{
"finding": "Contract amount exceeds threshold",
"severity": "warning",
"value": 13750000,
"threshold": 10000000,
"evidence": [
{
"document": "contract.pdf",
"page": 14
}
]
}
Now the application can validate:
- Required fields
- Data types
- Allowed values
- Evidence presence
- Citation format
before displaying the result to the user.
This creates a much stronger boundary between the probabilistic LLM layer and the deterministic application layer.
8. Evidence Grounding Should Be a First-Class Feature
One of the most important design decisions is to treat evidence as part of the generated result—not as an optional UI feature.
For example:
Finding
│
├── Generated statement
│
├── Supporting evidence
│
├── Source document
│
└── Source page
The application can then allow a reviewer to move directly from the finding to the relevant page.
This creates a human verification loop:
AI Finding
↓
Evidence
↓
Human Review
↓
Approve / Reject / Correct
For systems used in document review, this workflow can be more valuable than simply trying to maximize the amount of text generated by the model.
9. Separate Finding Generation From Evidence Retrieval
Another useful architectural principle is to distinguish between:
What the model thinks is happening
and
What evidence supports that conclusion
For example:
Finding Generation
↓
"Amount exceeds threshold"
↓
Evidence Retrieval
↓
Page 14
↓
Source text / table
↓
Validation
This allows the application to verify that an AI-generated finding actually has supporting evidence.
It also makes debugging easier.
If a finding is incorrect, we can ask:
- Was the document extracted incorrectly?
- Was the wrong chunk retrieved?
- Was the correct evidence retrieved but ignored?
- Did the LLM interpret the evidence incorrectly?
- Did schema or validation fail?
That is much more actionable than simply saying:
“The AI hallucinated.”
10. A Production Architecture
Putting these ideas together:
┌─────────────────┐
│ PDF Upload │
└────────┬────────┘
↓
┌───────────────────┐
│ OCR + Layout │
│ Understanding │
└─────────┬─────────┘
↓
┌───────────────────┐
│ Page-aware │
│ Document Model │
└─────────┬─────────┘
↓
┌───────────────────┐
│ Chunking + │
│ Metadata │
└─────────┬─────────┘
↓
┌───────────────────┐
│ Search Index │
│ Keyword + Vector │
└─────────┬─────────┘
↓
User Query
↓
┌───────────────────┐
│ Hybrid Retrieval │
└─────────┬─────────┘
↓
┌───────────────────┐
│ LLM Generation │
│ Structured Output │
└─────────┬─────────┘
↓
┌───────────────────┐
│ Evidence + │
│ Citation Validation│
└─────────┬─────────┘
↓
Human Review
The LLM is only one component of the system.
The surrounding engineering determines whether the system is reliable enough for production.
11. What I Have Learned
Building AI systems around real-world documents has changed how I think about RAG.
The difficult part is rarely:
“How do I call an LLM?”
The difficult part is designing the entire pipeline around it.
A production system needs to consider:
- Document extraction
- Layout preservation
- Chunking
- Metadata
- Retrieval
- Ranking
- Prompt design
- Structured output
- Validation
- Evidence grounding
- Evaluation
- Monitoring
- Human review
Most importantly, the system should make it easy to verify what the AI is saying.
For many enterprise AI applications, trust does not come from the model alone.
It comes from the combination of:
Good retrieval + structured generation + verifiable evidence + human review.
That is the foundation I would use when designing a production RAG system for document-heavy applications.
Final Thought
RAG should not be viewed simply as:
“Give documents to an LLM.”
A better mental model is:
“Build a reliable evidence pipeline around an LLM.”
Once you think about RAG this way, many engineering decisions—from page-aware ingestion to hybrid search and citation validation—become much clearer.
Top comments (0)