Introduction
Organizations across almost every industry eventually run into the same wall: information grows faster than any team can reasonably review it. A single business, research project, investigation, healthcare workflow, insurance operation, financial process, compliance program, or enterprise archive can accumulate thousands — sometimes tens of thousands — of pages of documents.
Those documents rarely arrive in one clean format. A typical collection might include PDFs, scanned PDFs, Word documents, spreadsheets, emails, reports, forms, tables, images, diagrams, handwritten notes, and historical records, mixing structured and unstructured data freely.
The hard part was never storage. The hard part is turning thousands of heterogeneous pages into something that can be searched, understood, connected, summarized, and verified. A simple document search system isn't enough for that job, and a large language model on its own isn't either. What actually works is a document intelligence and retrieval pipeline — an architecture that converts raw files into a structured knowledge layer, so that AI can reason over 10,000+ pages without ever needing the entire archive stuffed into a single prompt.
At a high level, the full pipeline looks like this:
Documents
↓
Document Processing
↓
OCR / Extraction
↓
Classification & Separation
↓
Chunking
↓
Embeddings
↓
Vector Search
↓
Evidence Retrieval
↓
AI Reasoning
↓
Summaries / Answers / Reports
This approach lets an application work with 10,000+ pages without repeatedly sending the entire document collection to a large language model.
1. The Real Problem Isn't Document Storage
Uploading 10,000 pages into cloud storage is trivial — S3 handles that without complaint. The difficulty starts the moment someone asks a real question: "Find everything related to the previous condition." "What changed between 2020 and 2024?" "Show all records that mention this subject." "Summarize the entire history."
At that point the system has to understand the content, not just hold onto it. It helps to think of the problem in layers:
10,000+ PAGES
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
Storage Processing Understanding
│ │ │
▼ ▼ ▼
S3 OCR/Parsing AI Models
│
▼
Retrieval
│
▼
Answer
The most important insight in the whole architecture is this: document storage, document understanding, retrieval, and reasoning are four different problems, and they shouldn't all be handled by one service.
2. Why 10,000+ Pages Break the Naive Approach
The obvious first idea is:
10,000 pages
↓
Large Language Model
↓
Summary
That architecture creates several problems:
- Context limitations. Even models with very large context windows shouldn't be handed an entire archive for every operation.
- Cost. Reprocessing the full corpus repeatedly gets expensive fast.
- Latency. Massive prompts take longer to process.
- Retrieval quality. When thousands of unrelated pages are dumped into one context, the model has to do the hard work of separating signal from noise itself.
- Lack of provenance. A generic summary often can't tell you exactly which document or page backs up a given statement.
- Repeated computation. Unchanged documents shouldn't have to be reanalyzed every time a new question comes in.
The better pattern flips the order of operations:
Process once
↓
Store structured knowledge
↓
Search efficiently
↓
Send only relevant evidence to the model
3. The Solution: A Knowledge Pipeline, Not a Prompt
Rather than treating uploaded files as one giant prompt, the goal is to convert them into a reusable knowledge system:
Raw Documents
│
▼
Document Processing
│
▼
Normalized Content
│
├───────────────┐
▼ ▼
Structured Data Chunks
│ │
▼ ▼
PostgreSQL Embeddings
│
▼
Vector Index
│
▼
Retrieval
│
▼
AI Reasoning
This produces a two-speed architecture. The first stage — ingestion — is expensive but happens mostly once. The second stage — search and question-answering — is fast and runs every time a user interacts with the system.
4. Stage One: Ingesting the Documents
Ingestion starts when a large collection is uploaded — a project folder full of PDFs, scans, spreadsheets, and images:
Case / Project / Dataset
│
├── document-001.pdf
├── document-002.pdf
├── report-003.docx
├── scan-004.pdf
├── image-005.png
├── spreadsheet-006.xlsx
└── ...
All original files go into durable object storage, organized by project:
s3://document-ai/
projects/
PROJECT-001/
original/
processed/
pages/
ocr/
chunks/
summaries/
reports/
The original data should always remain untouched. AI processing creates derived representations rather than replacing the source.
5. Determining What Each Document Actually Contains
A 10,000-page archive is almost never one uniform dataset:
10,000 pages
│
├── Reports
├── Forms
├── Correspondence
├── Financial records
├── Technical records
├── Historical records
├── Images
└── Scanned documents
The application therefore needs document classification. Once classified, each document carries metadata like:
{
"document_id": "DOC-00021",
"type": "technical_report",
"date": "2025-04-12",
"page_start": 143,
"page_end": 161
}
Once documents are classified, downstream processing becomes much more intelligent.
6. Splitting Large Files into Logical Documents
A single large PDF often contains dozens of separate logical documents stitched together:
Pages 1–20
→ Report A
Pages 21–37
→ Report B
Pages 38–55
→ Form Set
Pages 56–92
→ Technical Documentation
Instead of asking "what is this 10,000-page file?" the system can ask "what is this 15-page report?" — a much easier question that makes classification, summarization, metadata extraction, and retrieval significantly more precise.
Managed document services such as Amazon Bedrock Data Automation can help with document processing and splitting workflows; AWS currently documents splitting for source documents up to 3,000 pages. For collections beyond that, the workload can be divided into multiple source files or processing units.
7. Handling Scanned Documents
A large share of enterprise archives aren't really text documents at all — they're images sitting inside PDFs:
PDF
↓
Scanned Page
↓
Image
↓
No searchable text
A normal PDF text parser may return "" even though a human can clearly read the page. This is why OCR is a required processing stage:
Scanned page
↓
OCR
↓
Machine-readable text
Amazon Textract is one AWS option for extracting text from documents, while open-source OCR engines such as PaddleOCR can provide another implementation path. The important architectural principle is: never discard the original page image after OCR. Store the original page alongside the OCR text, because OCR can contain errors.
8. OCR Is Not the Same as AI Understanding
These steps are often incorrectly treated as one operation. They are actually different:
Image
↓
OCR
↓
Text
↓
Embedding
↓
Vector search
↓
LLM reasoning
- OCR answers: what text is visible?
- Embedding answers: what is the semantic representation of this content?
- Vector search answers: which stored pieces of content are semantically similar to this query?
- LLM answers: what does the retrieved evidence mean in context?
This separation makes the architecture much easier to scale.
9. Documents Are Multimodal, Not Just Text
A single page may contain text, a table, a chart, an image, and a diagram all at once:
SOURCE PAGE
│
┌──────────┴──────────┐
▼ ▼
Text Visual
│ │
▼ ▼
Text Embedding Multimodal Embedding
Amazon Nova Multimodal Embeddings is designed for multimodal retrieval and supports text, images, documents, video, and audio in a common embedding framework — especially valuable for collections containing important visual information.
10–11. Chunking: Turning Documents into Searchable Units
After extraction, large documents must be divided into smaller logical pieces. Instead of creating one embedding for a 20-page report:
20 pages
↓
1 vector
the report is split into logical chunks:
20 pages
↓
Section 1
Section 2
Section 3
Section 4
...
↓
Multiple vectors
There's no universal perfect chunk size — the better strategy is structure-aware chunking, respecting headings, paragraphs, sections, tables, page boundaries, and semantic meaning rather than cutting text blindly every N characters.
A practical starting point for many document-search systems is around 700–1,000 words per chunk with moderate overlap, but the real answer comes from benchmarking sizes like 500, 800, 1,200, and 1,500 words against real queries. The goal is maximum retrieval accuracy with minimum unnecessary context.
12–13. Embeddings: Turning Content into Searchable Meaning
A keyword search for "thermal issue" may fail to match "the machine experienced repeated overheating problems," even though the concepts are clearly related. A semantic embedding captures that relationship:
Text
↓
Embedding Model
↓
[0.14, -0.82, 0.31, ...]
The numbers themselves aren't meaningful to a human — their purpose is to place semantically related content near each other in vector space.
For text-heavy workloads, Amazon Titan Text Embeddings V2 is a practical option, supporting multiple output dimensions including 1024. For mixed text and visual content, Nova Multimodal Embeddings becomes attractive:
Text-heavy
↓
Titan Text Embeddings V2
↓
S3 Vectors
Multimodal
↓
Nova Multimodal Embeddings
↓
S3 Vectors
The final choice should be made through retrieval benchmarking rather than model specifications alone.
14–15. Vector Search and Metadata
Once millions of words become vectors, the application needs an efficient way to find similar content:
Document chunks
↓
Embeddings
↓
S3 Vectors
↓
Vector index
When a user searches "previous equipment failures," the system generates a query embedding and searches the index:
Question
↓
Embedding Model
↓
Query Vector
↓
S3 Vectors
↓
Top-K similar chunks
A vector alone isn't enough — every vector should carry metadata back to its source:
{
"project_id": "PROJECT-001",
"document_id": "DOC-00421",
"document_type": "report",
"year": 2024,
"page_start": 143,
"page_end": 145,
"chunk_id": "CHUNK-00872",
"source_s3_key": "projects/PROJECT-001/chunks/CHUNK-00872.json"
}
This hybrid approach — metadata filtering plus semantic similarity — is much stronger than pure vector search on its own.
16–17. Retrieve Evidence Before Generating
One of the key lessons from this architecture: do not send raw embedding vectors to the language model and expect it to summarize them. The correct pipeline is:
Source content
↓
Embedding
↓
Vector index
↓
Query
↓
Nearest vectors
↓
Original source text
↓
AI model
↓
Answer
For example, a user asking "Find records discussing previous system failures" triggers this flow:
User Question
↓
Query Embedding
↓
Vector Search
↓
Top 5–20 Results
↓
Retrieve Source Chunks
↓
Build Context
↓
Nova
↓
Answer
The final model receives something like:
Question:
Find records discussing previous system failures.
Evidence 1:
Report A, page 18
"Repeated overheating was observed..."
Evidence 2:
Maintenance Report, page 4
"Previous thermal failures..."
Evidence 3:
Inspection Report, page 12
"The system had experienced..."
The model can then reason over these specific pieces of evidence, and the answer stays traceable back to a document and page.
18–19. Hierarchical Summarization
A 10,000-page collection shouldn't have only one summary. Instead, create multiple levels:
10,000+ Pages
↓
Documents
↓
Document Summaries
↓
Category Summaries
↓
Year / Time Summaries
↓
Cross-Document Analysis
↓
Final Summary
Organized another way:
PROJECT
│
├── Technical
│ ├── 2022
│ ├── 2023
│ └── 2024
│
├── Financial
│ ├── 2022
│ ├── 2023
│ └── 2024
│
├── Operational
│ ├── 2022
│ └── 2024
│
└── Correspondence
Instead of feeding 500 documents into one huge prompt:
500 documents
↓
one huge prompt
the hierarchy compresses the corpus step by step:
500 documents
↓
500 document summaries
↓
20 category summaries
↓
5 yearly summaries
↓
1 final synthesis
This is effectively a compression hierarchy — the amount of information reaching the final reasoning model shrinks dramatically at each level while the important signal survives.
20–21. Batch Processing for Large Archives
A 10,000-page system shouldn't process every document through a synchronous API request. Instead:
Upload
↓
Create Processing Job
↓
Queue
↓
Workers
↓
Parallel Processing
↓
Store Results
Independent tasks run asynchronously:
Document 1 → Summary
Document 2 → Summary
Document 3 → Summary
...
Document 500 → Summary
Amazon Bedrock Batch Inference is designed for exactly this kind of asynchronous batch model processing using data stored in S3, and it's especially useful for work that doesn't depend on live user interaction.
22. Not Everything Should Be Batch
Batch processing is excellent for classification, summaries, extraction, and embedding generation. But a live user question needs a real-time path:
Understand query
↓
Search vectors
↓
Retrieve evidence
↓
Reason
↓
Respond
A good architecture therefore combines:
Batch processing
+
Real-time retrieval
+
Real-time reasoning
23. The Role of the Backend
A backend framework such as AdonisJS can function as the orchestration layer. Its role isn't to perform every AI operation itself:
AdonisJS
│
├── Authentication
├── Project Management
├── Upload APIs
├── Job Creation
├── Processing Status
├── AWS Integration
├── Search API
├── Report API
└── Business Logic
The heavy work runs through background workers.
24. Queue-Based Processing Architecture
A robust workflow chains jobs together:
Upload
│
▼
Create Project
│
▼
Document Processing Job
│
▼
Classification Job
│
▼
OCR Job
│
▼
Chunking Job
│
▼
Embedding Job
│
▼
Vector Index Job
│
▼
Summary Job
│
▼
Analysis Job
│
▼
Final Report Job
Each job maintains its own status, progress, attempts, error, start time, and completion time.
This means a failed operation doesn't require processing the entire archive again.
25. PostgreSQL, S3, and S3 Vectors Have Different Jobs
A common architecture mistake is trying to put everything into the vector database. Instead:
PostgreSQL stores projects, documents, document types, dates, structured fields, summaries, timeline, processing status, users, and jobs.
S3 stores original documents, page images, OCR files, chunks, and generated reports.
S3 Vectors stores embeddings, vector metadata, and the retrieval index.
This gives each storage layer a clear responsibility.
26. The Full Knowledge Lifecycle
RAW INFORMATION
│
▼
Document Processing
│
▼
Normalized Content
│
┌─────────┴─────────┐
▼ ▼
Structured Data Chunks
│ │
▼ ▼
PostgreSQL Embeddings
│
▼
S3 Vectors
│
▼
Retrieval
│
▼
Source Evidence
│
▼
AI Reasoning
│
┌────────────────┼───────────────┐
▼ ▼ ▼
Answers Summaries Reports
27. Cost Optimization
Large-document AI systems should be designed around cost per useful answer, not simply cost per page. The expensive mistake:
10,000 pages
↓
Large model
↓
Every question
The scalable strategy:
Process once
↓
Store knowledge
↓
Retrieve only relevant information
↓
Use AI reasoning on small context
28. Use Different Models for Different Jobs
Classification
↓
Small / inexpensive model
High-volume extraction
↓
Small / inexpensive model
Document summary
↓
Cost-efficient model
Semantic search
↓
Embedding model
Complex synthesis
↓
More capable reasoning model
In the AWS ecosystem, Amazon Nova models can be divided according to task complexity rather than treating one model as the universal solution.
29. Managed Automation vs. a Custom Pipeline
Managed document intelligence:
S3
↓
BDA
↓
Structured document output
↓
Embeddings
↓
Vector search
Custom pipeline:
S3
↓
Parser
↓
OCR
↓
Classification
↓
Chunking
↓
Embeddings
↓
Vector search
Neither is universally superior — the choice depends on engineering budget, processing volume, document diversity, required accuracy, operational complexity, and AWS cost. A hybrid architecture is often the most practical choice.
30. The Hybrid Architecture
10,000+ PAGES
│
▼
S3
│
▼
Document Processing
BDA / OCR
│
▼
Classification
│
▼
Document Separation
│
┌───────────┴───────────┐
▼ ▼
Structured Data Text
│ │
▼ ▼
PostgreSQL Chunking
│
┌────────────┴────────────┐
▼ ▼
Text Embeddings Visual Embeddings
│ │
└────────────┬────────────┘
▼
S3 Vectors
│
▼
Semantic Search
│
▼
Evidence Retrieval
│
▼
Amazon Nova
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Answers Analysis Reports
31. Reliability Is More Important Than a Beautiful Demo
A production system needs to answer: where did this statement come from? Every important generated claim should be traceable to:
Project
↓
Document
↓
Page
↓
Chunk
↓
Original source
For example:
Finding:
The issue was documented before 2022.
Evidence:
Document: Maintenance Report
Page: 17
Date: 2021-08-04
This is much more trustworthy than a paragraph generated without provenance.
32. The Source-of-Truth Principle
AI should never become the permanent source of truth. The source of truth remains the original document:
Original
↓
OCR
↓
Extracted text
↓
Structured facts
↓
Embeddings
↓
Summaries
↓
AI analysis
When the AI says something important, the application should be able to travel backward through that chain.
33. Contradiction Detection, Timelines, and Missing Information
Once documents are structured and searchable, the same architecture supports higher-level analysis. For example:
Document A:
"System was operational."
Document B:
"System was offline."
Document C:
"Maintenance required due to failure."
The system can identify a potential contradiction — but this should be treated as an AI-generated analytical finding, not automatically as established truth. The application should show both source statements so the user can verify the issue.
Dates extracted across the archive can be assembled into a chronological layer:
2021-02-10 → First reported issue
2021-06-15 → Maintenance performed
2022-01-09 → New failure
2022-03-21 → Investigation
2023-04-17 → Final report
And gaps in the record can be flagged the same way — if the archive contains 2021, 2022, and 2024 reports but nothing for 2023, the AI can flag that 2023 records may be missing, again as a finding that requires verification rather than an absolute fact.
34. From Search Engine to Knowledge System
Early on, the application may look like:
Upload
+
Search
After building the complete pipeline, it becomes:
DOCUMENT INTELLIGENCE PLATFORM
│
├── Search
├── Question Answering
├── Summarization
├── Timeline
├── Classification
├── Structured Extraction
├── Contradiction Detection
├── Evidence Retrieval
├── Reports
└── Cross-Document Analysis
The vector database is only one component. The real product is the knowledge layer built around the document corpus.
35. A Practical MVP
Phase 1: S3 upload, PDF/document extraction, OCR, document classification, chunking, embeddings, S3 Vectors, semantic search.
Phase 2: Document summaries, category summaries, structured extraction, AI Q&A.
Phase 3: Timeline, contradiction detection, missing information, final reports.
This reduces implementation risk by sequencing the hardest infrastructure first and the more advanced analytical features later.
36. Recommended AWS Architecture
USER
│
▼
Next.js
│
▼
AdonisJS
│
┌──────────────┼──────────────┐
▼ ▼ ▼
S3 PostgreSQL SQS
│ │
│ ▼
│ Workers
│ │
▼ ▼
Document Processing OCR / Classification
│
▼
Chunking
│
▼
Embeddings
│
▼
S3 Vectors
│
▼
Retrieval
│
▼
Nova
│
┌──────────────────┼─────────────────┐
▼ ▼ ▼
Answers Analysis Reports
This architecture separates storage, processing, orchestration, retrieval, and reasoning.
37. Ten Rules for Building This Kind of System
- Never treat 10,000 pages as one prompt. Break information into manageable, meaningful units.
- Preserve the original documents. AI-generated derivatives should never replace source evidence.
- OCR and embeddings solve different problems. OCR extracts text; embeddings create searchable semantic representations.
- Vectors are for retrieval, not generation. Retrieve the source content before asking a reasoning model to answer.
- Use metadata with vectors. Category, date, document ID, page and source information greatly improve retrieval.
- Use batch processing for independent large-volume work. Don't make a single HTTP request responsible for processing an entire archive.
- Use hierarchical summarization. Document → category → period → final synthesis.
- Keep evidence provenance. Every important AI finding should be traceable to its source.
- Use different models for different jobs. Cheap models for repetitive work; stronger models for difficult reasoning.
- Benchmark the complete pipeline. Accuracy, retrieval quality, processing time, cost, and reliability matter more than any individual model specification.
38. The Final Architecture in One Picture
┌─────────────────────────────┐
│ 10,000+ PAGES │
│ PDFs / DOCX / Scans / Images│
└──────────────┬──────────────┘
│
▼
┌───────────┐
│ S3 │
└─────┬─────┘
│
▼
┌─────────────────────┐
│ Document Processing │
│ BDA / OCR │
└──────────┬──────────┘
│
▼
┌──────────────────┐
│ Classification │
│ & Separation │
└────────┬─────────┘
│
┌─────────────┴──────────────┐
▼ ▼
┌─────────────────┐ ┌────────────────┐
│ Structured Data │ │ Logical Chunks │
└────────┬────────┘ └───────┬────────┘
│ │
▼ ▼
PostgreSQL Embedding Model
│
┌─────────────┴─────────────┐
▼ ▼
Text Embedding Multimodal
Embedding
│ │
└─────────────┬─────────────┘
▼
┌────────────┐
│ S3 Vectors │
└──────┬─────┘
│
▼
Query / Search
│
▼
Relevant Evidence
│
▼
Amazon Nova
│
┌───────────────────────┼──────────────────────┐
▼ ▼ ▼
Q&A Answer Analysis Report
Conclusion
The 10,000+ page problem is not fundamentally a problem of storing large files. Storage is relatively easy.
The real challenge is transforming massive amounts of heterogeneous information into a system that can answer questions accurately, retrieve supporting evidence, understand relationships across documents, and generate useful summaries.
The solution is not:
10,000 pages → one giant AI prompt
It is:
10,000+ pages
↓
Document processing
↓
OCR / extraction
↓
Classification
↓
Logical separation
↓
Chunking
↓
Embeddings
↓
Vector indexing
↓
Semantic retrieval
↓
Relevant source evidence
↓
AI reasoning
↓
Answers + summaries + reports
This architecture turns an enormous document collection into a reusable knowledge layer. Once that knowledge layer exists, the same underlying infrastructure can support many applications across industries: enterprise archives, compliance, research, insurance, healthcare, financial analysis, engineering, investigations, operations, and other document-heavy workflows.
The key idea is simple: do the expensive document understanding once, store the resulting knowledge intelligently, retrieve only what matters, and use AI reasoning on the relevant evidence rather than the entire archive. That's what makes AI analysis of 10,000+ pages practical, scalable, and economically sustainable.
Top comments (0)