DEV Community: NEXT4I DEV

How I Learned to Stop Worrying and Love Markdown The PDF-to-AI Pipeline War Story

NEXT4I DEV — Thu, 27 Aug 2026 04:50:00 +0000

TLDR; When building an AI knowledge retrieval pipeline that extracts text from documents, I discovered that PDF is the worst format for AI and Markdown is the best. Here's the multi-step pipeline I had to build just to handle PDFs, why it was necessary, and the generic pattern you can steal to handle document ingestion in your own RAG systems.

The Problem: PDFs Are Pixel-Perfect Hell for AI Parsers

I was building a RAG (Retrieval-Augmented Generation) pipeline for NEXT4I the kind of system that reads your documents first, then answers questions from them. Standard stuff: document ingestion → chunking → embedding → vector search → LLM answer generation.

I chose a beautiful Thai tourism PDF as my test document. Professional design, complex Thai typography, images, tables, charts the works. Real-world document, real-world pain.

Here's what the naive approach looked like:

PDF File → PDF Parser → Extracted Text → Chunk → Embed → Search

And here's what actually worked:

PDF File
  ├─→ PDF Parser → Raw Text (broken Thai, missing punctuation)
  ├─→ Page Renderer → Full-Color Images
  │     └─→ B&W Converter → High-Contrast Images
  ├─→ AI Vision Model (color images) → Image Descriptions
  ├─→ AI Vision Model (B&W images) → Text Extraction
  └─→ Cross-Validation Layer
        ├─→ Multi-Model Synthesis
        ├─→ Spell-Check Model (critical for Thai)
        └─→ Human Review
              └─→ Final Structured Text → Chunk → Embed → Search

Why the complexity? Because PDF is fundamentally a presentation format, not a data format. When you extract text from a PDF, you're not reading structured data you're reverse-engineering a rendered page layout. For languages with complex typography like Thai (where vowels can appear above, below, left, or right of consonants, and tone marks float above), this is especially brutal.

The Generic Pattern: Multi-Path Document Ingestion with Cross-Validation

If you're building any system that ingests arbitrary documents, you'll inevitably hit the PDF wall. Here's the reusable pattern I settled on:

Architecture

                    ┌──────────────┐
                    │   Document   │
                    │   Ingest     │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  Direct  │ │  Image   │ │  Image   │
        │  Text    │ │  (Color) │ │  (B&W)   │
        │ Extract  │ │  Render  │ │  Render  │
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  Text    │ │  Vision  │ │  Vision  │
        │  Output  │ │  Model   │ │  Model   │
        │          │ │  (Desc)  │ │  (OCR)   │
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             └────────────┼────────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Cross-Validate │
                 │  & Synthesize   │
                 │  (Multi-Model)  │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Spell-Check    │
                 │  & Normalize    │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Human Review   │
                 │  (Optional)     │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Structured     │
                 │  Output → Embed │
                 └─────────────────┘

The key insight: no single extraction path is reliable enough on its own. You need multiple independent paths producing results, then a synthesis layer that cross-validates. Think of it like sensor fusion each path is a noisy sensor, and the truth emerges from the overlap.

Why Spell-Check is Non-Negotiable for Non-English Languages

For English, you might get away without a dedicated spell-check pass. For Thai where a single misplaced tone mark changes the entire word you absolutely cannot. OCR and vision models hallucinate characters constantly on decorated fonts or text-over-image backgrounds. A dedicated language model fine-tuned for spell correction is the difference between "usable" and "garbage."

The Real Takeaway: Markdown is AI-Native. Everything Else Is Legacy.

After building this entire pipeline, I had a moment of clarity. If that same document had been authored in Markdown:

## Top Destinations

| Province | Highlight | Best Season |
|----------|-----------|-------------|
| Krabi    | Islands   | Nov–Apr     |
| Chiang Mai | Mountains | Nov–Feb   |

See the [full itinerary](#itinerary) for details.

mermaid
graph TD
A[Arrive Bangkok] --> B[Fly to Krabi]
B --> C[Island Hopping]
C --> D[Return]


...the entire pipeline collapses to: **read the file → chunk → embed → search.** That's it.

No OCR. No vision models. No B&W conversion. No multi-path cross-validation. No spell-check model. No human review for format-induced errors.

Markdown is structured, plain-text, and both human-readable and machine-parseable by default. It's the only format where:
- **Headings** are unambiguously `#` / `##` not inferred from font size
- **Tables** are `| column | row |` syntax not pixel grids
- **Diagrams** are Mermaid text not flattened raster images
- **Code** is fenced not monospaced-font heuristics


> "And here is how the human user experiences it:"
> 
## Top Destinations

| Province | Highlight | Best Season |
|----------|-----------|-------------|
| Krabi    | Islands   | Nov–Apr     |
| Chiang Mai | Mountains | Nov–Feb   |

See the [full itinerary](#itinerary) for details.

mermaid
graph TD
A[Arrive Bangkok] --> B[Fly to Krabi]
B --> C[Island Hopping]
C --> D[Return]




---


*"In reality, we can't always control the documents we ingest, and we can't just ignore them because they might contain critical data. But if we were to start from scratch, Markdown is definitely the go-to choice."*


---


### How This Shapes Our Architecture at NEXT4I

At NEXT4I, we treat Markdown as a first-class format throughout our stack. When building AI knowledge retrieval systems for everyday users and organizations, we encourage Markdown as the source of truth and handle PDFs as a necessary-but-painful compatibility layer.

The design principle is simple: **AI Integration by Design.** Make AI a first-class citizen of your content architecture, not something you bolt on later and hope it works. The format you choose today determines the ceiling of your AI capabilities tomorrow.

---

Explore the NEXT4I journey & dev-notes to read the original article at: [NEXT4I/dev-notes/en](https://go.next4i.com/next4i/devnotes/en)

Alternatively, you can register to join **NEXT4I** the AI-Native Ecosystem I am currently building at: [NEXT4I.com](https://go.next4i.com/next4i-devto-en)

How to Build a "Second Brain" with Obsidian That Your AI Agent Can Read, Without Building a Custom RAG Pipeline

NEXT4I DEV — Mon, 17 Aug 2026 11:40:46 +0000

Ever run into this? You wrote a detailed technical spec three months ago, and today someone asks "how did we design this module again?" You end up spending 20 minutes hitting Cmd+F across Google Docs, Trello, and README.md files scattered across different repos, or sometimes there's no documentation at all, so you have to dig through the code and reverse-engineer it by eye.

TLDR; I built a knowledge-base system for NEXT4I using Obsidian (plain Markdown) together with Git and a VS Code AI agent, without writing a single line of custom integration. The core of the system is choosing tools that already "speak the same language" from day one (plain text, open formats), so AI can read our knowledge base practically for free, no RAG pipeline required.

Pain Point: Scattered Knowledge and Requirements Make Search Hard and Nothing Stays in Sync

Before this system, all of NEXT4I's knowledge was scattered across many places: notebooks, Apple Notes, Google Docs, Google Sheets, Trello, README.md files, or, even worse, sometimes there was no documentation at all, just buried in the code, spread across multiple repos each written in a different language, frontend and backend alike.

This isn't just an inconvenience. It makes search genuinely hard, sometimes it's Cmd+F and pray. Worse: my AI coding agent could read the entire codebase, but it had zero visibility into the reasoning behind that code, because that reasoning was scattered somewhere the AI couldn't reach.

Design Constraints: 3 Non-Negotiables

Before picking a tool, I set 3 rules:

It has to be affordable, or free if possible.
It has to be accessible online anytime, from my phone, and still work offline.
Plain text, zero lock-in. If the tool disappears tomorrow, the files must remain immediately readable and usable.
Local-first, Git-friendly. It has to be a normal folder I can git init and track right away.
AI-readable without building extra infrastructure. My AI agent already lives in VS Code, so the knowledge base has to sit inside that workspace without me building an extra pipeline.

Obsidian passed everything, because at its core, an Obsidian vault is just a folder of .md files.

Architecture: Vault Structure

vault/
├── Ideas/          # Raw concepts, brainstorming
├── Manifesto/      # Vision, mission, core policies
├── Principles/     # Design rules, engineering guidelines
├── Infrastructure/ # Deployment topology, IaC specs
├── Platform/       # Domain model, API contracts
├── Script/         # Utility scripts, automation, runbooks
├── Skill/          # Patterns, checklists, reusable knowledge
└── Appendix/       # Domain glossary, citations

Every file is a plain .md. Links use [[wiki-link]] syntax. Metadata lives in YAML frontmatter, and Graph View renders the relationships as a visible dependency graph.

Version control is just git init inside the vault folder, then committing every change with a rationale. git log -- "Infrastructure/sharding-strategy.md" shows the full decision history for that topic. Push it to a private GitHub repo and you get backup, an audit trail, and branching for major revisions, all for free.

Key Insight: AI Access Without Building Anything

This is the part that changed everything.

Obsidian vault = a folder of .md files.
VS Code = opens any folder as a workspace.
AI coding agent (running as a VS Code extension) = reads every file in that workspace.

So the "integration" here is: open the vault folder in VS Code.

That's it. No API, no embedding pipeline, no vector database, no chunking strategy. Just plain Markdown files that the AI agent reads natively.

Prompt Patterns I Actually Use

Contextual search + reasoning:

Search all documents that mention our sharding strategy.
Summarize every trade-off we've considered
and tell me which approach we ultimately chose and why.

Gap analysis:

Look at everything in the Infrastructure/ folder
and tell me which architectural decisions are still undocumented,
compared against the template in Skill/.

Drafting from conventions, not from a blank page:

Using the patterns in Skill/go-backend/ and the domain model in Platform/core/,
draft a design doc for a new message consumer
following our established conventions.

Impact analysis via link traversal:

If I change the authentication rule in Principles/auth.md,
trace every file in Platform/ and Skill/ that links to it via [[links]]
and tell me what needs updating.

The AI reads across multiple files, follows [[wiki-links]], understands the relationships, and synthesizes an answer, without me writing a single line of integration code.

Philosophy: Seamless Integration by Design

The pattern here isn't "integrate 3 tools." It's choosing components that already speak the same language.

Obsidian chose Markdown, the most universally readable format in computing, over a proprietary database. Git works with any plain text. AI agents already know how to read files in a VS Code workspace.

Bonus: Obsidian Canvas as a Visual Layer

Obsidian's Canvas is an infinite whiteboard, place document cards, text, media, then draw connections between them.

I use Canvas for:

System architecture sketches: each service as a card, data flow arrows, real specs embedded right on the board
Decision trees: "if we pick X, then Y and Z are affected," with linked evidence
Plan strategy & flow: for planning work, sequencing, and various NEXT4I workflows

Canvas files are Markdown too under the hood (JSON-like structure), so they're Git-versioned and AI-readable as well. Drop a .canvas file into your VS Code workspace and ask the AI to analyze it for circular dependencies or single points of failure.

What I Learned

What makes this system work isn't the technology, it's what I didn't build. No middleware, no proprietary pipeline, no vendor lock-in.

The discipline of plain text + Git + open formats is a feature, not a limitation.

If you're a dev or a small team drowning in scattered documents, try this before jumping to a heavyweight knowledge-management platform. Plain Markdown with a good folder structure will take you further than you'd expect.

Thanks for reading all the way to the end, I'll keep working on more articles like this.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i/journey/en

Alternatively, you can register to join NEXT4I the AI-Native Ecosystem I am currently building at: https://go.next4i.com/next4i-devto-en

tags:
#buildinpublic
#secondbrain
#next4i
#obsidian

How I Built an AI-Readable Second Brain with Obsidian, Git, and a VS Code AI Agent

NEXT4I DEV — Fri, 07 Aug 2026 07:55:45 +0000

When you're a solo founder and lead architect, your knowledge base is your most valuable asset. Lose the thread on why a decision was made, and you spend hours — sometimes days — reconstructing context that you already figured out once.

I want to share the exact setup I use at NEXT4I to turn a folder of Markdown files into a fully searchable, version-controlled, AI-readable knowledge system. No proprietary SaaS, no vendor lock-in, no custom integration work.

This is the Key Highlight of this post: a genuinely useful, generic pattern you can apply to your own projects today. The NEXT4I-specific business logic stays abstracted (per our security rules), but the pattern itself is 100% reusable.

1. The Architecture: Three Layers, Zero Magic

┌─────────────────────────────────────────┐
│           AI Agent (VS Code)            │
│   Reads, Searches, Summarizes, Drafts   │
└──────────────────┬──────────────────────┘
                   │ reads plain .md files
┌──────────────────▼──────────────────────┐
│       Git-tracked Obsidian Vault        │
│  ├── Idea/           (brainstorms)      │
│  ├── Infrastructure/ (architecture docs)│
│  ├── Platform/       (product specs)    │
│  ├── Script/         (automation)       │
│  └── Skill/          (reusable limits)  │
└──────────────────┬──────────────────────┘
                   │ committed & pushed
┌──────────────────▼──────────────────────┐
│         GitHub (remote backup)          │
└─────────────────────────────────────────┘

Layer 1 — The Vault (Obsidian): A folder of interconnected .md files. The key insight is that Obsidian uses plain Markdown with [[wiki-links]] for connections — no database, no proprietary format.

Layer 2 — Version Control (Git): Every vault is a git repo. Every change to any document has a commit message, a timestamp, and a diff. You can git log --oneline -- Idea/ to see the evolution of a concept.

Layer 3 — AI Agent (VS Code Extension): Because the vault is just a file tree of .md files, any AI coding agent that can read a codebase can also read your knowledge base. Point the agent at the vault folder, and it has full context.

2. The Setup: Step-by-Step

Step 1: Create the Vault


mkdir next4i-knowledge

cd next4i-knowledge

mkdir Idea Infrastructure Platform Script Skill

git init

Open this folder in Obsidian: Open folder as vault.

Step 2: Link Everything

Inside a note, link to another note with [[Note Name]]. Obsidian auto-suggests as you type. Over time, this builds a graph you can visualize with Cmd/Ctrl + G.

Pro tip: Create a _INDEX.md in each folder that links to the most important notes. This becomes a human-readable table of contents AND a search anchor for the AI.

Step 3: Add Git Discipline


git add -A && git commit -m "infra: initial sharding strategy decision"

git remote add origin git@github.com:your-org/knowledge-vault.git

git push -u origin main

Treat commit messages like code. Use prefixes: idea:, infra:, platform:, script:, skill:. This makes git log --oneline --grep="infra:" instantly useful.

Step 4: Open in VS Code and Activate the AI


code /path/to/vault

With an AI agent extension active (Copilot, Cline, Cody, etc.), try prompts like:

"Summarize the key architectural decisions in the Infrastructure folder."
"Find any contradiction between documents in /Platform/ and /Infrastructure/."
"Draft a new document in /Idea/ based on the sharding notes in /Infrastructure/."

The agent reads the files as context, just like it would for code.

3. The Design Pattern: Folder-Convention-as-API

Here's the key pattern: your folder structure IS your API.

By keeping a consistent vault structure, both humans and AI know where to look:

Folder	Contains	AI Use Case
`Idea/`	Raw, unstructured thinking	Generate summaries, find related concepts
`Infrastructure/`	System topology, deployment, config	Validate consistency, trace dependencies
`Platform/`	Feature specs, user flows	Draft task tickets, check requirement coverage
`Script/`	Automation, one-liners	Explain what a script does, suggest improvements
`Skill/`	Reusable patterns, checklists	Retrieve relevant patterns for new tasks

This is essentially a convention-based RAG (Retrieval-Augmented Generation) setup without any vector database, embedding pipeline, or chunking strategy. The "chunking" is the natural boundary of each .md file. The "retrieval" is the AI agent's file-reading capability.

4. Why This Beats a Wiki

Wiki / Confluence	This Setup (Obsidian + Git)
Vendor lock-in	Plain `.md` files, portable anywhere
Search is siloed within the tool	VS Code AI searches across the whole vault
No version control (or poor built-in)	Full version control (`git blame`, `git diff`, `git log`)
Hard to automate	Scriptable — `grep`, `sed`, and AI prompts all work
AI needs API integration	AI reads files natively, zero setup required

5. What I Learned

The biggest surprise: The AI agent became better at finding connections in my own notes than I was. It doesn't have recency bias. It doesn't forget what I wrote 8 months ago. It reads everything with equal attention.

The biggest lesson: AI-native doesn't mean "add an AI button." It means design your systems — including your thinking systems — so that AI can participate as a first-class citizen without special plumbing.

I'm building NEXT4I as an AI-native ecosystem from the ground up. If you're interested in following a solo founder's engineering journey — or want early access — join here:
Subscribe NEXT4I or want early access — join here