Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

LLM Knowledge Base Architecture Guide [2026]: Wiki vs Notes vs RAG

LLM knowledge base architecture is the set of choices that decides whether your docs are merely readable by humans or reliably usable by agents.

And yeah, it matters now. Teams went from “we have documentation” to “our AI agents can answer and act on it,” and the gap between those two is where the money goes to die. You ship a RAG demo. Everyone claps. Then you discover your “source of truth” is unchunkable PDFs, permission leaks, and search that returns the wrong page with terrifying confidence.

In this llm knowledge base architecture guide, I’m going to compare three real architectures (wiki-style, notes-first, and RAG-first) and give you a migration plan that doesn’t require a rewrite-from-scratch.

[Suggested inline image: a simple 3-column diagram showing Wiki / Notes / RAG-first and where “source of truth” lives]

Wiki vs notes vs RAG-first: the decision matrix

Most “RAG guides” quietly assume you already have a clean corpus. Most “documentation guides” assume humans are the only readers. Neither is true in 2026.

When an engineering org asks me, “What should we standardize on so our agents stop hallucinating internal facts?”, I don’t start with vector databases. I start with where the truth lives, how it changes, and who is allowed to see it.

Decision criterion	Wiki-style KB (Confluence / MediaWiki)	Notes-first KB (Obsidian / Notion-like)	RAG-first KB (index + pipeline as the product)
Source of truth	Pages + hierarchy	Files/blocks + backlinks	Canonical content store + retrieval contracts
Best for	Stable policies, runbooks, onboarding	Fast capture, personal/team workflows	High-stakes agent workflows, multi-source corpora
Storage format	HTML-ish / vendor formats	Markdown-ish / blocks	Markdown/HTML with strong metadata + IDs
Sync	Manual edits, occasional exports	Local-first sync or API pull	Continuous ingestion + incremental indexing
Search	Keyword + sometimes semantic	Depends on tool; often weak enterprise search	Hybrid search + reranking + metadata filters
Embeddings	Optional bolt-on	Optional bolt-on	First-class cost + versioning problem
Access control	Usually mature page ACLs	Often coarse or messy	Must be enforced at retrieval time
Agent-readability	Low by default	Medium with conventions	High by design (citations, anchors, contracts)
Migration cost	Low if you’re already there	Low if you’re already there	Medium upfront, lower long-term chaos
Main risk	Great UI, terrible portability	“Everyone writes differently” entropy	Over-engineering without evaluation gates

If you only take one thing from this post, take this:

If retrieval is unreliable, your “knowledge base” is just a content lake with a chat UI.

Limitations of foundation models (why a KB exists at all)

Foundation models are impressive. They’re also not your company’s memory. A real knowledge base exists because the model has three problems you can’t hand-wave away.

1) Knowledge cutoffs. Even if your model is “new,” its training data is frozen at a point in time. Jenna Pederson (Pinecone) uses the obvious example: ask about last week’s sports results or the latest iPhone feature and the model can answer confidently with outdated or fabricated info.

2) Private and proprietary data. Your incident runbooks, customer-specific SLAs, internal system diagrams, and “how we do deploys” docs are not in the public training set. That’s a feature, not a bug. But it means the model is blind without retrieval.

3) Domain depth. Broad models can still be shallow in your niche. In practice, “domain depth” means “the weird details only your org learned after three outages and two postmortems.” That stuff isn’t a dataset. It’s institutional memory.

I’ve seen this at scale. On the Walmart conversational commerce chatbot I built at Firework (Zealsight), the model choice mattered less than retrieval quality. We were handling millions of queries daily with sub-second responses, and when answers were wrong, it was usually because the wrong product facts were retrieved or the context was stale, not because the LLM “wasn’t smart enough.”

What is Retrieval-Augmented Generation (RAG) in a knowledge base?

Retrieval-Augmented Generation (RAG) is a pattern where an LLM generates an answer after retrieving relevant private or domain content and injecting it into the prompt as grounding context.

This isn’t just “chat with your PDFs.” In a real rag knowledge base architecture, retrieval is the system. The generation step is the final renderer.

Jenna Pederson’s breakdown at Pinecone is still the cleanest mental model: ingestion → retrieval → augmentation → generation. I like it because it forces you to build the parts most teams ignore until they’re on fire. Ingestion quality. Retrieval evaluation.

When people say “we built RAG,” they often mean “we embedded some text and did vector search.” That’s one sub-step of one stage.

How does Retrieval-Augmented Generation work end-to-end?

Here’s the pipeline in practical terms. Not the blog-demo version.

1) Ingestion (turning docs into a corpus)

Ingestion is where most teams lose months.

• You decide what content is in-scope: pages, PDFs, tickets, Slack exports, runbooks, code comments.
• You normalize it into a consistent representation.
• You attach metadata (owner, system, environment, security labels, updated_at, canonical_id).

A real ingestion pipeline is usually at least 3 sources (wiki + Git + ticketing) within the first quarter. If you’re only ingesting one source, you’re still doing a demo.

2) Retrieval (finding candidates)

Retrieval is rarely one query.

Classic pattern: a single hybrid search query returns top-k chunks.

Agentic pattern: the system decomposes the question into sub-queries, hits multiple sources, and returns structured results. Microsoft’s Azure AI Search docs explicitly distinguish “agentic retrieval” from a “classic RAG pattern” in their RAG overview.

3) Augmentation (stuffing context into a prompt)

Augmentation is where you decide:

• How many chunks (k=5, k=20)?
• Token budget for retrieved context (e.g., 2,000–8,000 tokens reserved)
• Whether to include citations, titles, and section anchors

This is also where you apply last-mile filters. If you do nothing else, do this: dedupe near-identical chunks and prefer newest versions.

4) Generation (answering, citing, acting)

Generation is the part everyone demos. It’s also the part you’ll swap models on every 6–12 months.

Your knowledge base architecture should assume generation is replaceable. Your retrieval contracts and content IDs should not be.

If you’re building AI agents, this is also the tool-call layer. The agent might retrieve, summarize, then call an internal API. That’s why “agent-ready” knowledge bases need stricter ACLs and better provenance.

For adjacent reading: if you’re trying to connect the dots between RAG and agents, my post on AI agents and the difference between agentic AI and chatbots will make the rest of this guide click.

[Suggested inline image: pipeline diagram showing ingestion → index → retrieve → rerank → prompt → model]

The challenges of RAG, and what they force you to decide

Microsoft’s Azure AI Search RAG overview lists the challenges in a way I like because it reads like an architecture checklist: query understanding, multi-source access, token constraints, response time expectations, and security and governance.

Here’s what those categories mean when you actually have to ship.

Query understanding → hybrid search, metadata, and reranking

If your retrieval is vector-only, you will miss obvious exact matches (error codes, feature flags, SKU IDs). If it’s keyword-only, you miss synonymous phrasing.

So you end up with:

• Hybrid search (keyword + vector) as the default.
• Metadata filters (system=payments, env=prod, region=ca-central-1).
• Reranking with a cross-encoder or semantic ranker to fix “top-k is close but wrong.”

If you want the search comparison mindset, my write-up on Postgres full text search vs Elasticsearch is the same decision pattern, just without embeddings.

Multi-source access → canonical IDs and dedupe rules

If your content lives in Confluence, Notion, GitHub, and Google Drive, you need a canonical identity layer. Full stop.

Concrete rule: every document gets a stable canonical_id that does not change when you rename it, move it, or re-render it.

If you don’t do this, you’ll create three very predictable disasters:

1. Duplicates in the index.
2. Broken citations (the agent can’t point to a stable source).
3. Full re-embedding every time content moves.

Token constraints → chunk size, sectioning, and “citability”

LLMs accept limited input. That forces chunking discipline.

In practice, I see most teams land on chunks in the 300–800 token range for general docs, and smaller for runbooks where headings matter.

More important than size is boundary logic: chunk at semantic section boundaries, not arbitrary character counts.

If this topic is biting you, you’ll probably also like my post on RAG context window limits. Bigger windows don’t fix sloppy retrieval.

Response time expectations → precompute + cache + streaming

Your users expect answers in seconds. In production, that pushes you toward:

• Precomputed embeddings (obviously)
• Cached retrieval results for common questions
• Fast rerankers or approximate reranking

On the Walmart chatbot, we learned that pipeline latency was dominated by plumbing more than model tricks. Event-streaming the context pipeline (Kafka) mattered for latency more than micro-optimizing generation.

Security and governance → retrieval-time enforcement, not UI ACLs

Microsoft calls out security challenges as a core category for RAG systems. Good. Most teams still treat it like a checkbox.

If you only enforce permissions in the UI, you’ve already lost. Your retriever must filter results before anything hits the model.

That’s the difference between “we respect Confluence permissions” and “we don’t leak the CTO’s comp doc into an on-call chat.”

Content preparation for RAG: formats, IDs, chunking, versioning

This is the part people skip. Then they act shocked that their “knowledge base” became a six-week rewrite project.

Storage formats: Markdown wins, PDFs lose

If your goal is agent consumption, pick a format that’s:

• easy to diff
• easy to parse
• easy to chunk
• easy to render for humans

Markdown with frontmatter is the boring answer and it keeps winning.

• Markdown/MDX: best for incremental updates and stable anchors.
• HTML: fine if you control rendering and can preserve headings/IDs.
• PDF: worst. You can ingest it, but you’ll fight layout noise forever.

If you’re starting from a wiki, you don’t need to migrate the UI on day 1. Normalize wiki exports into Markdown behind the scenes.

Stable identifiers and anchors (don’t break citations)

Agent answers are only trustworthy if they can cite stable sources.

Practical conventions:

• Document ID: kb:payments:runbook:deploy-v3
• Section anchor: #rollback-procedure
• Chunk ID: kb:payments:runbook:deploy-v3#rollback-procedure::chunk-02

If you change headings every week, your anchors churn. If your anchors churn, your citations rot.

Chunking rules that enable incremental re-embedding

OpenAI’s embeddings docs are explicit: requests are billed based on input tokens. The cost of a knowledge base is mostly the cost of re-embedding change.

So your chunking strategy is a cost strategy.

Rules I use:

1. Chunk by heading sections (##, ###) first.
2. Only split within a section if it exceeds a token ceiling.
3. Compute a chunk hash from normalized text.
4. Only re-embed chunks whose hash changed.

With this, a one-line fix in a 2,000-line doc doesn’t force a full re-embed.

Based on the tooling I maintain on this site, the most common cost mistake is treating “tokens per call” as the whole story. Real workloads include retries, re-indexing, and cache hit rates. My LLM cost calculator tooling and the live pricing data I maintain at kunalganglani.com/llm-prices exist for exactly this reason.

Agent-readable documentation conventions (this is the “notes vs wiki” unlock)

“Agent-readable” is not a vibe. It’s a writing spec.

If you’re staying wiki-style or notes-first, these conventions matter more than your vector database choice:

• One page = one job. A runbook is not a design doc. Don’t mix.
• Put the answer first. The first 5–8 lines should contain the policy / command / decision.
• Always include constraints. “Only for prod.” “Only after 2023-01-01.”
• Add ownership. owner: platform-oncall in frontmatter.
• Add freshness. last_verified: 2026-08-01.
• Add a changelog section for operational docs.
• Use stable headings. Stop renaming ## Rollback to ## Undo because it “reads nicer.”

If you want templates that match this style, I wrote AI-readable documentation templates specifically for agent consumption.

[Suggested inline image: screenshot-style illustration of a markdown doc with frontmatter + stable headings]

Choose the right architecture: wiki-style vs notes-first vs RAG-first

This is where I’m going to be opinionated.

When a wiki-style knowledge base is the right source of truth

A wiki is still a great human system. It’s often the fastest way to get:

• approvals
• page-level ACLs
• discoverability via links
• non-engineering contributions

Pick wiki-first if:

• your primary content is policies, handbooks, and runbooks
• you need mature page permissions on day 1
• you have lots of non-technical authors

But if you’re building agent workflows, you need to treat the wiki as a UI, not a storage layer.

Concrete advice: export pages nightly into a canonical content store (Markdown/HTML) with stable IDs, then index that. Your wiki can stay. Your agent-facing corpus becomes portable.

When a notes-first system is acceptable (and how to make it LLM-friendly)

Notes-first is fine when the cost of structure is higher than the cost of occasional wrong answers.

Examples:

• a small team’s engineering journal
• incident scratchpads
• personal knowledge management

The problem is entropy. A notes-first system turns into ten different writing styles and zero ownership.

If you insist on notes-first for team knowledge, you need conventions:

• mandatory frontmatter (owner, scope, system, created, updated)
• “source of truth” flag (source_of_truth: true/false)
• a rule that every operational note must graduate into a runbook within 7 days

If you don’t enforce graduation, you end up retrieving unverified notes in production. That’s how you teach your agent to hallucinate with citations.

When you should go RAG-first instead of building a wiki

RAG-first means the knowledge base is an engineered product: a content store + ingestion pipeline + index + eval harness.

Go RAG-first if any of these are true:

• you have 3+ content sources you can’t consolidate politically
• you need strict doc-level security at retrieval time (multi-tenant, regulated)
• your agents take actions (tickets, deployments, customer emails)
• you can’t tolerate “maybe correct” answers

I’m biased here because I’ve built this style of system in production. On the Walmart chatbot, GraphRAG only paid off for relationship queries (like product compatibility), not general Q&A. But the broader lesson held: once you have millions of queries a day, “just build a wiki” is not an architecture. It’s wishful thinking.

If you’re thinking about going deeper on retrieval graphs, start from the GraphRAG concept and then decide whether the extra complexity actually maps to your query types.

Document-level security for RAG (patterns that actually work)

This is where teams either do the real work or ship a data leak.

Here are three patterns I trust.

Pattern 1: ACL propagation into metadata filters

• Store allowed_principals (or roles/groups) as metadata per doc/chunk.
• At query time, filter retrieval by the caller’s principals.

This is simple and fast. It also explodes in size if you literally store every user ID. Prefer group IDs.

Pattern 2: Per-tenant indexes

• One vector index per tenant/customer/business unit.
• No cross-tenant retrieval possible by construction.

This is the boring enterprise pattern. It’s also operationally heavier. If you have 100 tenants, you now have 100 indexes.

Pattern 3: Redaction + dual-corpus

• Maintain a “safe” corpus that can be used broadly.
• Maintain a “sensitive” corpus with stronger controls.
• Run redaction on ingestion.

If you’re doing redaction, you should have tests for it. I’ve written a practical implementation guide for field-level redaction for RAG pipelines and a more general LLM data leakage playbook.

Also: permission leakage isn’t hypothetical. Treat it like a vulnerability class. If you’re not doing regression tests for prompt injection, you’re not serious about agent security.

External reference worth keeping bookmarked: Microsoft’s overview explicitly frames “Security and governance” as a RAG challenge in Azure AI Search’s RAG documentation.

Knowledge base migration plan to RAG (phased, reversible, measurable)

The fastest way to fail is to announce, “We’re migrating our docs to be agent-ready,” and start a rewrite.

Do this instead.

1. Phase 0: Inventory and scope (1–2 weeks)

   • Identify the top 25–50 documents that answer 80% of questions (on-call runbooks, policies, top SOPs).
   • Define “done” metrics: citation accuracy, freshness, permission leakage rate.

2. Phase 1: Establish canonical IDs + metadata (1–2 weeks)

   • Add stable IDs without moving any content.
   • Define frontmatter schema (canonical_id, owner, system, security_label, updated_at).

3. Phase 2: Chunking and incremental indexing (2–4 weeks)

   • Implement chunk boundaries + hashing.
   • Store chunk IDs and chunk hashes.
   • Re-embed only changed chunks.

4. Phase 3: Retrieval quality (ongoing, but start now)

   • Add hybrid search.
   • Add reranking.
   • Add metadata filters.
   • Create an eval set of 100 real questions with expected citations.

5. Phase 4: Hardening (2–6 weeks, parallelizable)

   • Permission filters wired end-to-end.
   • Leakage tests.
   • Observability: retrieval hit rate, “no answer” rate, P95 latency.

This plan is intentionally reversible. At any point, you can stop and still have improved docs.

If you want the broader “don’t rewrite from scratch” philosophy applied to engineering decisions, my post on why software rewrites are usually a trap is the same lesson in a different costume.

How to get started: the portable reference architecture

If you want a vendor-neutral default stack, here’s what I’d ship for a mid-sized team.

• Content store: Git repo of Markdown (or an object store with versioned blobs)
• Ingestion: scheduled pull from wiki/drive + webhook from Git
• Normalizer: HTML → Markdown, PDF → text (only if you must)
• Index: vector DB + keyword index (or a system that supports hybrid)
• Retrieval: hybrid search + metadata filters + reranker
• Policy layer: ACL resolution service
• Evaluation: curated Q/A set + citation checks + leakage tests
• Observability: traces for retrieval and generation, plus cost tracking

If you’re already investing in agent systems, build the eval harness early. I’ve shared a lightweight process in Agent evaluation roadmap for small teams and more production depth in How to build vendor-neutral LLM observability monitoring.

For external grounding, keep these three references handy:

• Jenna Pederson for the clean ingestion→retrieval→augmentation→generation model.
• Microsoft’s Azure AI Search RAG overview for the “agentic retrieval vs classic RAG” distinction and security framing.
• OpenAI’s Vector embeddings docs for the operational reality: embeddings are vectors, distance drives relatedness, and you pay by token.

Here’s my prediction: within 12 months, “we have docs” stops being an acceptable answer in engineering leadership. You’ll be asked, “Can our agents use them safely?”

Design your knowledge base like a product. IDs. Contracts. Permissions. Eval gates.

Do that, and you’ll be ahead of the curve instead of getting crushed by it.

---

Originally published on kunalganglani.com