i spent 3 days on a rag pipeline and the embedding model didn't matter at all

it's 2am. PagerDuty fires. you're on-call.

you open the wiki. search for "CrashLoopBackOff." get 47 results. none of them are the runbook you need. you try "pod stuck pending." different results, equally useless. you tab over to Slack and ask "anyone remember which runbook covers Kafka consumer lag?" and wait.

I've done this more times than I can count. I've watched engineers grep through Confluence at 3am, tab through 20 browser windows, and page someone who retired last year because they wrote the runbook in 2019. first-response time is slow, and resolution quality depends entirely on who's on call. the senior engineer has the runbooks memorized. the new hire is reading them for the first time — during a SEV1.

RAG should fix this. retrieve the right runbook section, generate a grounded answer, cite the source. three seconds instead of fifteen minutes.

so I built one. and I discovered something I didn't expect: the biggest lever in the pipeline wasn't the embedding model, the LLM, or the vector database. it was how I split the documents into chunks.

---

what I built

RAGoncall — a local-first RAG assistant that indexes 55 markdown runbooks and answers on-call queries with sourced citations. four delivery channels off the same backend: CLI, Discord bot, Streamlit dashboard, and a FIFO alert pipeline.

the corpus is 55 markdown runbooks from four public sources: Kubernetes troubleshooting guides, Grafana OnCall docs, Google SRE book chapters, and community runbooks. the embedding model is all-MiniLM-L6-v2 — local, free, 384-dim. the LLM is pluggable — local Llama-3.2-3B, OpenAI, or Anthropic — swappable via a config flag with zero code changes.

but before I wrote a single line of retrieval code, I did something most RAG tutorials skip: I built an eval set.

---

the eval set nobody talks about

fourteen question-answer pairs. real questions an on-call engineer would ask, each with a known correct source file and line range. stored as JSON:

{
  "id": "k8s-001",
  "question": "Why is my pod stuck in Pending state and how do I debug it?",
  "answer": "A pod stuck in Pending means the scheduler cannot place it...",
  "source_file": "corpus/kubernetes/debug-pods.md",
  "source_lines": "11-19"
}

fourteen pairs is small. I know it's small. but it's enough to measure the difference between chunking strategies — which is the whole point. without this eval set, "it looks right" is the only quality signal you have. that's how teams ship RAG that hallucinates in production.

the eval harness computes MRR (Mean Reciprocal Rank) — how high the first relevant result appears. rank 1 gives 1.0, rank 2 gives 0.5, not in results gives 0. when you're on-call at 2am, you read the first result, not the fifth. MRR is the metric that matters.

---

the chunking experiment

same embedding model. same vector DB. same LLM. same corpus. same eval set. the only variable: how I split the documents.

header-split chunking hit MRR 0.9524. recursive chunking hit 0.9048. fixed-size chunking hit 0.8893. all three achieved perfect Recall@5 — the correct document was always in the top 5. the difference was MRR: how high the right chunk ranked.

six points of MRR from chunking alone. the embedding model, LLM, and vector DB moved it by zero.

I didn't expect this. I assumed the embedding model would be the biggest lever. that's what every RAG tutorial implies — "choose the right embedding model." so I tested it. the embedding model was never the bottleneck. the chunking was.

---

why header-split won

runbooks are structured by headers. ## Diagnosis. ## Resolution. ## Escalation. when you split on those boundaries, each chunk is a complete section — the entire resolution procedure, not a fragment that starts mid-sentence. split on headers first, then merge small sections up to 1.5× chunk_size so you don't get tiny one-line chunks. any section still too big gets a fallback recursive split.

fixed-size chunking doesn't respect any of this. a 600-character resolution procedure gets split into a 500-char chunk and a 100-char fragment. neither contains the complete answer. the embedding captures half the procedure. retrieval returns half the procedure. the LLM generates an answer from half the procedure.

recursive chunking does better — it respects paragraph boundaries — but it can still split mid-section when a section exceeds 500 chars. it loses ~5% MRR vs header-split because relevant content sometimes straddles chunk boundaries.

the lesson isn't "header-split is always best." the lesson is: your chunking strategy has to match your document structure. for runbooks, split on headers. for code, split on functions. for legal docs, split on clauses. the embedding model matters less than the chunking — that surprised me.

---

what broke

this is the section I wish more RAG posts had. the patterns are easy. the edge cases are the real education.

YAML frontmatter — a 38-point MRR drop. four out of fourteen eval questions targeted Grafana OnCall runbooks. these files start with YAML frontmatter — ---\ntitle: "Grafana OnCall"\ntags: ["oncall"]\n---. the header-split chunker didn't know about frontmatter. it treated it as regular content and merged it with the first markdown section. the semantic signal got diluted. retrieval missed the correct source file for all four Grafana questions. MRR for those 4 files: 0.571. MRR for the other 10: 0.9524. a 38-point swing from a production edge case that no toy corpus would expose.

the fix is a frontmatter-aware splitter that strips ---\n...\n--- before chunking. I didn't build it. it's a known limitation. this is the kind of thing that only surfaces when you test against real documents — not the curated test corpora that most RAG tutorials use.

no incremental re-indexing. index --rebuild drops and recreates the entire collection. for a production system where runbooks update daily, you need file-watching and incremental upsert. fine for 55 files. not fine for 5,000.

cold model load. first request with local Llama-3.2-3B takes about 30 seconds for the model to load into memory. subsequent queries are fast — 5 to 10 seconds. a cloud model (gpt-4o-mini, claude-3-haiku) has no cold start and 2-5s latency. the cost difference is negligible: about $0.0003 per query either way.

the eval set is 14 pairs. enough to measure chunking differences. not enough to claim production-grade retrieval quality. a real system needs 50+ pairs covering edge cases, multi-hop queries, and ambiguous questions. I'm honest about this.

---

the part that actually worked

on June 27, I tested the full pipeline live from my phone. I opened the Discord mobile app, sent /query "CrashLoopBackOff", and got back a sourced answer with kubectl describe pod and kubectl logs --previous steps. the bot posted a rich embed with a clickable source link. I tapped 👍. it logged to data/feedback.jsonl.

five minutes of wiki-diving → three seconds to a sourced answer. from a phone.

the 👍/👎 feedback is the cheapest improvement mechanism in the entire system. zero compute. zero labeling. structured data. the dashboard's "knowledge gaps" tab shows runbooks that get queried but receive 👎 — those are the runbooks that need rewriting.

---

what I'd tell you if you're building RAG

build the eval set first. before the pipeline, before the vector DB, before the LLM. 10-15 question-answer pairs with known correct source files. if you can't measure retrieval quality, you can't improve it, and you definitely can't ship it to someone making production decisions at 2am.

then test 3 chunking strategies on that eval set. don't assume fixed-size is fine because the tutorial used it. don't assume your embedding model is the bottleneck. measure MRR for each strategy and pick the one that fits your document structure.

one more thing: swapping the LLM doesn't change MRR. the LLM generates the answer from the chunks the retriever returns. if the retriever returns the wrong chunks, the best LLM in the world can't save you. retrieval quality is upstream of generation quality. fix your chunking before you fix your LLM.

---

discussion

I want to hear from teams running RAG in production, not tutorial experiments:

what chunking strategy are you using? if it's fixed-size, have you benchmarked it against header-split or recursive on your own eval set? I'm curious whether the 6-point MRR swing holds across different document types.

how big is your eval set? my 14 pairs are enough to measure chunking differences but not enough to claim production quality. what's the minimum you'd trust before shipping RAG to a user who makes decisions at 2am?

has anyone hit the YAML frontmatter problem? it seems niche, but any document with metadata headers (Jekyll, Hugo, Obsidian, Grafana) will have it. did you build a frontmatter-aware splitter, or just strip it before indexing?

thumbs up/down feedback — is anyone using this for re-ranking? I'm collecting the data but haven't built a re-ranker yet. the 👍/👎 log could train a cross-encoder re-ranker that moves well-received answers up. curious if anyone has done this and whether it moved MRR.

if you're building RAG for production, drop a comment with your eval set size, chunking strategy, and MRR. especially if your MRR is higher than 0.9524 — I want to know what you did differently.