TLDR; When building an AI knowledge retrieval pipeline that extracts text from documents, I discovered that PDF is the worst format for AI and Markdown is the best. Here's the multi-step pipeline I had to build just to handle PDFs, why it was necessary, and the generic pattern you can steal to handle document ingestion in your own RAG systems.
The Problem: PDFs Are Pixel-Perfect Hell for AI Parsers
I was building a RAG (Retrieval-Augmented Generation) pipeline for NEXT4I the kind of system that reads your documents first, then answers questions from them. Standard stuff: document ingestion → chunking → embedding → vector search → LLM answer generation.
I chose a beautiful Thai tourism PDF as my test document. Professional design, complex Thai typography, images, tables, charts the works. Real-world document, real-world pain.
Here's what the naive approach looked like:
PDF File → PDF Parser → Extracted Text → Chunk → Embed → Search
And here's what actually worked:
PDF File
├─→ PDF Parser → Raw Text (broken Thai, missing punctuation)
├─→ Page Renderer → Full-Color Images
│ └─→ B&W Converter → High-Contrast Images
├─→ AI Vision Model (color images) → Image Descriptions
├─→ AI Vision Model (B&W images) → Text Extraction
└─→ Cross-Validation Layer
├─→ Multi-Model Synthesis
├─→ Spell-Check Model (critical for Thai)
└─→ Human Review
└─→ Final Structured Text → Chunk → Embed → Search
Why the complexity? Because PDF is fundamentally a presentation format, not a data format. When you extract text from a PDF, you're not reading structured data you're reverse-engineering a rendered page layout. For languages with complex typography like Thai (where vowels can appear above, below, left, or right of consonants, and tone marks float above), this is especially brutal.
The Generic Pattern: Multi-Path Document Ingestion with Cross-Validation
If you're building any system that ingests arbitrary documents, you'll inevitably hit the PDF wall. Here's the reusable pattern I settled on:
Architecture
┌──────────────┐
│ Document │
│ Ingest │
└──────┬───────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Direct │ │ Image │ │ Image │
│ Text │ │ (Color) │ │ (B&W) │
│ Extract │ │ Render │ │ Render │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Text │ │ Vision │ │ Vision │
│ Output │ │ Model │ │ Model │
│ │ │ (Desc) │ │ (OCR) │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
└────────────┼────────────┘
│
▼
┌─────────────────┐
│ Cross-Validate │
│ & Synthesize │
│ (Multi-Model) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Spell-Check │
│ & Normalize │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Human Review │
│ (Optional) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Structured │
│ Output → Embed │
└─────────────────┘
The key insight: no single extraction path is reliable enough on its own. You need multiple independent paths producing results, then a synthesis layer that cross-validates. Think of it like sensor fusion each path is a noisy sensor, and the truth emerges from the overlap.
Why Spell-Check is Non-Negotiable for Non-English Languages
For English, you might get away without a dedicated spell-check pass. For Thai where a single misplaced tone mark changes the entire word you absolutely cannot. OCR and vision models hallucinate characters constantly on decorated fonts or text-over-image backgrounds. A dedicated language model fine-tuned for spell correction is the difference between "usable" and "garbage."
The Real Takeaway: Markdown is AI-Native. Everything Else Is Legacy.
After building this entire pipeline, I had a moment of clarity. If that same document had been authored in Markdown:
## Top Destinations
| Province | Highlight | Best Season |
|----------|-----------|-------------|
| Krabi | Islands | Nov–Apr |
| Chiang Mai | Mountains | Nov–Feb |
See the [full itinerary](#itinerary) for details.
mermaid
graph TD
A[Arrive Bangkok] --> B[Fly to Krabi]
B --> C[Island Hopping]
C --> D[Return]
markdown
...the entire pipeline collapses to: read the file → chunk → embed → search. That's it.
No OCR. No vision models. No B&W conversion. No multi-path cross-validation. No spell-check model. No human review for format-induced errors.
Markdown is structured, plain-text, and both human-readable and machine-parseable by default. It's the only format where:
-
Headings are unambiguously
#/##not inferred from font size -
Tables are
| column | row |syntax not pixel grids - Diagrams are Mermaid text not flattened raster images
- Code is fenced not monospaced-font heuristics
"And here is how the human user experiences it:"
Top Destinations
| Province | Highlight | Best Season |
|---|---|---|
| Krabi | Islands | Nov–Apr |
| Chiang Mai | Mountains | Nov–Feb |
See the full itinerary for details.
graph TD
A[Arrive Bangkok] --> B[Fly to Krabi]
B --> C[Island Hopping]
C --> D[Return]"In reality, we can't always control the documents we ingest, and we can't just ignore them because they might contain critical data. But if we were to start from scratch, Markdown is definitely the go-to choice."
How This Shapes Our Architecture at NEXT4I
At NEXT4I, we treat Markdown as a first-class format throughout our stack. When building AI knowledge retrieval systems for everyday users and organizations, we encourage Markdown as the source of truth and handle PDFs as a necessary-but-painful compatibility layer.
The design principle is simple: AI Integration by Design. Make AI a first-class citizen of your content architecture, not something you bolt on later and hope it works. The format you choose today determines the ceiling of your AI capabilities tomorrow.
Thanks for reading all the way to the end, I'll keep working on more articles like this.
Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i/devnotes/en
Top comments (2)
PDF is a print format that describes where glyphs go, not what the document means, which is why every parser is reconstructing structure that was thrown away at export. Markdown wins because headings and lists are semantics rather than font sizes, and that is exactly what chunking needs in order to cut at a boundary instead of mid-argument. Thai is a hard test case for a specific reason worth naming: no spaces between words, so any text extractor that infers tokens from whitespace has nothing to work with, and errors there propagate straight into the embeddings. The rendering fallback is the right instinct - when the layout carries the meaning, treating the page as an image and reading it that way beats trying to repair the extracted string. Two things I would keep from a pipeline like this: preserve the page number on every chunk so a citation can point somewhere a human can verify, and store the intermediate markdown, because the day the parser improves you want to reingest without re-rendering everything.
Keeping the page number in every chunk is a great idea. It helps us target and correct errors directly when we get feedback. Thanks for the comment!
Have you done RAG with Thai or other similar languages before? Could you share your perspective or experience regarding accuracy? In your opinion, is around 75-80% accuracy considered good enough?