This is the data appendix for Posts 1–4.
The narrative and takeaways live in the main posts. This page is pure measurement.
Eval set: 22 questions (P1) → expanded to 31 questions (P2 onward)
Corpus evolution:
- P1/P2: ~3,700 docs (raw threads + code chunks)
- P3/P4: 16,315 docs (distilled threads + bursts + code)
Quick comparison (same 31-question set)
| Metric | Vector P2 | Hybrid P2 | Vector P3 | Hybrid P3 | Hybrid + Rerank (P4) |
|---|---|---|---|---|---|
| recall@1 | 0.68 | 0.61 | 0.52 | 0.39 | 0.87 |
| recall@3 | 0.84 | 0.65 | 0.71 | 0.65 | 0.94 |
| recall@10 | 0.90 | 0.90 | 0.81 | 0.94 | 0.94 |
| MRR | 0.77 | 0.67 | 0.63 | 0.57 | 0.90 |
Takeaway:
Hybrid alone never beat pure vector on this corpus.
Hybrid + LLM rerank is the first clear win.
P1 — Naive vector baseline
Corpus: 3,000 raw issue threads + 687 code chunks
Embeddings: BGE-M3 (1024d), max_seq_length=1024, HNSW cosine
Numbers (22 questions)
| Metric | Score |
|---|---|
| recall@10 | 1.00 (22/22) |
| recall@3 | 0.95 |
| recall@1 | 0.77 (17/22) |
Main k=1 misses
- Exact error pastes (
TypeError: Object of type int64...,AttributeError: 'Depends'...) — ranked 4–5 instead of 1 -
jsonable_encodercode chunk outranked by issues about the function - API key header implementation (code vs similar issues)
- Paraphrase questions (dependency injection outside routes, custom 404)
Pattern: Dense search is strong on recall@10 but weak when the query has a sharp lexical signal.
Ops notes
- Ingest wall time ~40 min (GitHub API is the bottleneck)
- BGE-M3 OOM on Apple Silicon fixed by capping
max_seq_length=1024 - Python 3.13 + uv editable install issue fixed by pinning 3.12
P2 — Hybrid (vector + FTS + RRF)
Corpus: Same size as P1, with better comment pagination and symbol-based code IDs
Eval set: Expanded to 31 questions (added exact error pastes + rare identifiers)
Numbers
| Metric | Vector | FTS | Hybrid |
|---|---|---|---|
| recall@1 | 0.68 | 0.42 | 0.61 |
| recall@3 | 0.84 | 0.48 | 0.65 |
| recall@10 | 0.90 | 0.65 | 0.90 |
| MRR | 0.77 | 0.47 | 0.67 |
Headline: Hybrid is not a strict win over vector-only.
Where hybrid helped
- Exact error pastes (e.g.
TypeError: int64 is not JSON serializable) → moved from rank 5 → 1 - Near-duplicate titles (HTTPBearer 403 vs 401) → correctly disambiguated
Where hybrid hurt
- Code-symbol questions (
jsonable_encoder, background tasks, APIKeyHeader) — FTS noise pushed the correct code chunk out of the top ranks - Strong semantic matches were diluted by diffuse lexical overlap
Pattern: Hybrid helps when the lexical signal is sharp. It hurts when the signal is diffuse or the answer is a code chunk.
P3 — Distillation + Bursting
Corpus rebuilt:
- 3,002 issue parents (2,542 distilled + 460 raw fallback)
- 12,626 burst rows (high-signal comments as separate vector-only documents)
- 687 code chunks → 16,315 total documents
Numbers vs P2
| Metric | Vector P2 | Vector P3 | Hybrid P2 | Hybrid P3 |
|---|---|---|---|---|
| recall@1 | 0.68 | 0.52 | 0.61 | 0.39 |
| recall@3 | 0.84 | 0.71 | 0.65 | 0.65 |
| recall@10 | 0.90 | 0.81 | 0.90 | 0.94 |
| MRR | 0.77 | 0.63 | 0.67 | 0.57 |
Headline: Distillation + bursting raised the recall ceiling but made top-of-ranking worse.
Why distillation hurt vector
The embedding is now computed on the clean LLM summary, not the raw thread.
Exact error strings and identifiers that used to live in the vector are gone → error-paste queries drop out of the top-10.
Why bursting helped
High-signal comments get their own embedding instead of being buried in long threads.
This is what pushed hybrid recall@10 from 0.90 → 0.94.
Remaining hard misses (even at k=10)
- “Which function converts arbitrary objects into JSON-compatible data structures?” (
jsonable_encoder) - “Where does FastAPI implement running background tasks after a response is returned?”
Both are natural-language → code-location lookups. Retrieval tuning alone cannot solve them.
P4 — Hybrid + LLM Rerank
Corpus: Same as P3 (16,315 docs)
Mode: hybrid + LLM reranker over the fused top-20
Final numbers
| Metric | Vector | Hybrid | Hybrid + Rerank |
|---|---|---|---|
| recall@1 | 0.52 | 0.39 | 0.87 (27/31) |
| recall@3 | 0.71 | 0.65 | 0.94 (29/31) |
| recall@10 | 0.81 | 0.94 | 0.94 (29/31) |
| MRR | 0.63 | 0.57 | 0.90 |
Headline: This is the first unqualified win in the series.
- Hybrid supplies the high-recall pool (0.94 @10)
- Reranker supplies the ordering (0.90 MRR)
Still missing (by design)
The same two code-location questions.
Reranking can only reorder what is already in the pool. These two never enter the top-20 → they require a different retriever (symbol / grep style), which is the subject of later posts.
Cost note
One extra LLM call per query (≈20 candidates). Retrieval legs remain LLM-free.
Summary of the arc so far
| Stage | What improved | What got worse | Net effect |
|---|---|---|---|
| P1 Naive vector | Strong baseline recall@10 | Weak precision on error pastes | Good starting point |
| P2 Hybrid | Better on sharp lexical queries | Worse on code + paraphrase | Slight regression overall |
| P3 Distill + Burst | Higher recall ceiling (0.94 @10) | Lower precision | Raised the ceiling for P4 |
| P4 + Rerank | Large jump in precision (MRR 0.90) | — | Clear winner |
Current default: hybrid --rerank
Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.
Top comments (0)