DEV Community: fr_trenton02

Rebuilding the Cerebras Knowledge Base: Results Appendix (P1–P4)

fr_trenton02 — Sun, 16 Aug 2026 12:23:40 +0000

This is the data appendix for Posts 1–4.

The narrative and takeaways live in the main posts. This page is pure measurement.

Eval set: 22 questions (P1) → expanded to 31 questions (P2 onward)

Corpus evolution:

P1/P2: ~3,700 docs (raw threads + code chunks)
P3/P4: 16,315 docs (distilled threads + bursts + code)

Quick comparison (same 31-question set)

Metric	Vector P2	Hybrid P2	Vector P3	Hybrid P3	Hybrid + Rerank (P4)
recall@1	0.68	0.61	0.52	0.39	0.87
recall@3	0.84	0.65	0.71	0.65	0.94
recall@10	0.90	0.90	0.81	0.94	0.94
MRR	0.77	0.67	0.63	0.57	0.90

Takeaway:

Hybrid alone never beat pure vector on this corpus.

Hybrid + LLM rerank is the first clear win.

P1 — Naive vector baseline

Corpus: 3,000 raw issue threads + 687 code chunks

Embeddings: BGE-M3 (1024d), max_seq_length=1024, HNSW cosine

Numbers (22 questions)

Metric	Score
recall@10	1.00 (22/22)
recall@3	0.95
recall@1	0.77 (17/22)

Main k=1 misses

Exact error pastes (TypeError: Object of type int64..., AttributeError: 'Depends'...) — ranked 4–5 instead of 1
jsonable_encoder code chunk outranked by issues about the function
API key header implementation (code vs similar issues)
Paraphrase questions (dependency injection outside routes, custom 404)

Pattern: Dense search is strong on recall@10 but weak when the query has a sharp lexical signal.

Ops notes

Ingest wall time ~40 min (GitHub API is the bottleneck)
BGE-M3 OOM on Apple Silicon fixed by capping max_seq_length=1024
Python 3.13 + uv editable install issue fixed by pinning 3.12

P2 — Hybrid (vector + FTS + RRF)

Corpus: Same size as P1, with better comment pagination and symbol-based code IDs

Eval set: Expanded to 31 questions (added exact error pastes + rare identifiers)

Numbers

Metric	Vector	FTS	Hybrid
recall@1	0.68	0.42	0.61
recall@3	0.84	0.48	0.65
recall@10	0.90	0.65	0.90
MRR	0.77	0.47	0.67

Headline: Hybrid is not a strict win over vector-only.

Where hybrid helped

Exact error pastes (e.g. TypeError: int64 is not JSON serializable) → moved from rank 5 → 1
Near-duplicate titles (HTTPBearer 403 vs 401) → correctly disambiguated

Where hybrid hurt

Code-symbol questions (jsonable_encoder, background tasks, APIKeyHeader) — FTS noise pushed the correct code chunk out of the top ranks
Strong semantic matches were diluted by diffuse lexical overlap

Pattern: Hybrid helps when the lexical signal is sharp. It hurts when the signal is diffuse or the answer is a code chunk.

P3 — Distillation + Bursting

Corpus rebuilt:

3,002 issue parents (2,542 distilled + 460 raw fallback)
12,626 burst rows (high-signal comments as separate vector-only documents)
687 code chunks → 16,315 total documents

Numbers vs P2

Metric	Vector P2	Vector P3	Hybrid P2	Hybrid P3
recall@1	0.68	0.52	0.61	0.39
recall@3	0.84	0.71	0.65	0.65
recall@10	0.90	0.81	0.90	0.94
MRR	0.77	0.63	0.67	0.57

Headline: Distillation + bursting raised the recall ceiling but made top-of-ranking worse.

Why distillation hurt vector

The embedding is now computed on the clean LLM summary, not the raw thread.

Exact error strings and identifiers that used to live in the vector are gone → error-paste queries drop out of the top-10.

Why bursting helped

High-signal comments get their own embedding instead of being buried in long threads.

This is what pushed hybrid recall@10 from 0.90 → 0.94.

Remaining hard misses (even at k=10)

“Which function converts arbitrary objects into JSON-compatible data structures?” (jsonable_encoder)
“Where does FastAPI implement running background tasks after a response is returned?”

Both are natural-language → code-location lookups. Retrieval tuning alone cannot solve them.

P4 — Hybrid + LLM Rerank

Corpus: Same as P3 (16,315 docs)

Mode: hybrid + LLM reranker over the fused top-20

Final numbers

Metric	Vector	Hybrid	Hybrid + Rerank
recall@1	0.52	0.39	0.87 (27/31)
recall@3	0.71	0.65	0.94 (29/31)
recall@10	0.81	0.94	0.94 (29/31)
MRR	0.63	0.57	0.90

Headline: This is the first unqualified win in the series.

Hybrid supplies the high-recall pool (0.94 @10)
Reranker supplies the ordering (0.90 MRR)

Still missing (by design)

The same two code-location questions.

Reranking can only reorder what is already in the pool. These two never enter the top-20 → they require a different retriever (symbol / grep style), which is the subject of later posts.

Cost note

One extra LLM call per query (≈20 candidates). Retrieval legs remain LLM-free.

Summary of the arc so far

Stage	What improved	What got worse	Net effect
P1 Naive vector	Strong baseline recall@10	Weak precision on error pastes	Good starting point
P2 Hybrid	Better on sharp lexical queries	Worse on code + paraphrase	Slight regression overall
P3 Distill + Burst	Higher recall ceiling (0.94 @10)	Lower precision	Raised the ceiling for P4
P4 + Rerank	Large jump in precision (MRR 0.90)	—	Clear winner

Current default: hybrid --rerank

Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.

Rebuilding the Cerebras Knowledge Base: the web UI and a look back

fr_trenton02 — Sun, 16 Aug 2026 12:20:45 +0000

Post 6 handed the retrieval tools to agents over MCP. This last post builds the other front end (a plain web page) for the humans who don't have an MCP client sitting in their editor. And since it is the end of the series, it is also where I add up the scoreboard.

The UI is deliberately small: one FastAPI app, two JSON endpoints, and a single static HTML file with no build step. The same no-frameworks rule that governed retrieval (no LangChain, no vector-store SDK) governs the front end too (no React, no bundler, no npm).

Two endpoints, which are the two halves of the series

The whole back end is a thin wrapper over the pipeline built in posts 1–5:

def create_app(search_fn, ask_fn) -> FastAPI:
    @app.post("/api/search")   # LLM-free retrieval  (posts 1–4)
    @app.post("/api/ask")      # full pipeline       (post 5)

create_app takes the two functions as arguments rather than building them: the same seam the tests use to inject fakes, and the same seam main() uses to wire in the real hybrid_search and run_ask. The endpoints mirror the two things the series built:

/api/search runs hybrid retrieval and returns serialized hits, with no model call. This is posts 1 through 4 in one route.
/api/ask runs the whole post-5 loop (plan, retrieve, rerank, expand, synthesize) and returns the answer, the tools the planner chose, the numbered evidence, and any people. This is post 5 in one route.

The front-end toggle makes that split literal:

<label><input type="radio" name="mode" value="search" checked> search (LLM-free)</label>
<label><input type="radio" name="mode" value="ask"> ask (full pipeline)</label>

You can see the architecture from the home page: one radio button for retrieval, one for retrieval plus reasoning.

What it actually returns

Real responses from the running app against the live corpus. search mode is instant and free:

// POST /api/search {"query": "how do I add middleware", "limit": 3}
{"results": [
  {"source_id": "issue_3027", "score": 0.0164,
   "snippet": "# How to add a header field to the request in a middleware?",
   "url": "https://github.com/fastapi/fastapi/issues/3027"},
  {"source_id": "issue_10180", "score": 0.0162,
   "snippet": "# Mounting sub-applications under `APIRouter`", "url": "…/10180"}
]}

ask mode spends the LLM calls and hands back a written, cited answer:

// POST /api/ask {"question": "How do I add a custom middleware in FastAPI?"}
{
  "tools": ["search"],
  "answer": "You can add custom middleware in FastAPI in several ways.\n\n### 1. HTTP
             middleware with `@app.middleware(\"http\")` ... async def
             add_process_time_header(request, call_next): ...",
  "evidence": [{"n": 1, "source_id": "issue_5071",
                "url": "https://github.com/fastapi/fastapi/issues/5071",
                "snippet": "# Update Middleware Documentation ..."}]
}

The page renders the answer in one block and the evidence as a citation list under it: the [n] markers in the prose line up with the numbered sources, so every claim is one click from the thread it came from.

The whole front end is one file

index.html is ~125 lines: inline CSS with a prefers-color-scheme dark mode, a form, three result containers, and ~50 lines of vanilla fetch that POST to the two endpoints and render the JSON. No framework, no state library, no build. It is served as a string straight from the package:

index_html = (resources.files("knowbase") / "static" / "index.html").read_text()

That is the entire deployment story: kb-web starts uvicorn on 127.0.0.1:8000, and the one file it needs travels inside the package. For a knowledge base that a handful of people query, a single static page is not a compromise; it is the right size.

The scoreboard

Seven posts, one rule: every post had to fix a failure the previous one demonstrated. Here is the whole arc on the fixed 31-question eval (MRR, the metric that tracked the story best):

Post	Change	MRR†	What it taught
1	naive vector	0.77*	the baseline is stronger than you expect
2	+ hybrid (RRF)	0.67	fusion is confidence-blind; it regressed vs vector (0.77)
3	+ distill & burst	0.57	corpus rebuild hurt precision (vector fell to 0.63 too) but lifted recall@10 to 0.94
4	+ rerank	0.90	the reranker converts recall into precision (the win)
5	planner + synthesis	n/a	retrieval becomes cited answers; "misses" get answered
6	MCP server	n/a	the tools become an agent's, not just ours
7	web UI	n/a	…and a human's

†The MRR column traces the hybrid pipeline (posts 2–4); post 1 is its vector-only predecessor, the thing hybrid replaced. *Posts 1–2 measured on the ~3,700-doc pre-distillation corpus; 3–4 on the 16,315-doc distilled/burst corpus. Same 31 questions throughout.

The shape of that table is the honest lesson of the series. The naive baseline was good. The two changes that looked like obvious wins (hybrid search, LLM distillation) each lost in isolation, and stayed in only because they were scaffolding: hybrid supplied a recall pool and distillation/bursting raised its ceiling to 0.94, and then one LLM reranker turned that pool into MRR 0.90. If I had shipped hybrid on faith in post 2 and never measured, I would have quietly made the system worse and called it progress.

What I'd still fix

The series ends honest about its edges:

The eval grades retrieval, not answers. Every number above is "is the right document in the ranking". There is no automated score for whether the synthesized paragraph is correct or its citations are faithful (post 5's answers are demonstrated, not measured). A real answer-quality eval is the obvious next project.
search_code is only as good as the pattern it is given (post 6): a bare symbol can drown in documentation matches while def X lands the definition. It should rank indexed code above docs.
Two eval questions still "miss" on retrieval (the jsonable_encoder and background-tasks code-location lookups), even though ask answers both from issue evidence. Closing that gap means better code retrieval, not a better ranker.

The takeaway

The system that started as one embeddings table and a cosine query ended as a planner routing across three retrievers, an LLM reranker, grounded synthesis with citations, an MCP server, and a web page; and the single most valuable component was the one I almost didn't measure carefully enough to keep. Build naive first, make the eval honest before you make the system clever, and let every addition prove it earned its place. That is the whole method; the FastAPI knowledge base was just where I ran it.

Thanks for reading all seven. The retrieval numbers live in the Results Appendix; the web front end is knowbase.web plus one static index.html.
Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.

Rebuilding the Cerebras Knowledge Base: an MCP server

fr_trenton02 — Sun, 16 Aug 2026 12:20:38 +0000

Post 5 built an agent loop: a planner picks tools, the tools retrieve, a synthesizer writes a cited answer. All of that intelligence (the planning, the reranking, the synthesis) was mine, running as my code and using my API budget.

Post 6 is the same tools with the loop taken out. Instead of being the agent, the knowledge base becomes a set of tools that someone else's agent (Claude Code, Claude Desktop, any MCP client) can call. The server does the retrieval; the client is the brain. It is a small amount of code and a genuinely different posture for the whole system.

The inversion

Every LLM call in posts 1–5 lived on my side: distilling threads, reranking pools, planning, synthesizing. The MCP server flips that. It exposes raw, LLM-free retrieval and lets the connected agent do the thinking:

"""MCP server exposing raw, LLM-free retrieval tools.

The client agent (e.g. Claude Code) does the orchestration: it decides which tool to call, reads the evidence, and synthesizes its own answer.
"""

That comment is the whole design idea. The server makes zero model calls. It is fast, cheap, and deterministic; the agent on the other end supplies the judgment that posts 3–5 spent LLM calls on.

Three tools, and what they return

The server is FastMCP over stdio, exposing exactly three tools (the retrievers from post 5, minus the planner and synthesizer):

@server.tool()
def search(query: str, limit: int = 10) -> list[dict]:
    """Hybrid semantic + keyword search over FastAPI issues and code."""

@server.tool()
def search_code(pattern: str, limit: int = 10) -> list[dict]:
    """Exact-pattern grep over the indexed source tree."""

@server.tool()
def who_knows(topic: str, limit: int = 5) -> list[dict]:
    """People most involved in the issue threads matching a topic."""

Each returns lightweight JSON (a source id, a score, a one-line snippet, and a URL to follow), not walls of text. The agent scans the list and decides what to pull. Real output against the live corpus:

// search("how do I add middleware")
{"source_id": "issue_3027", "score": 0.0164,
 "snippet": "# How to add a header field to the request in a middleware?",
 "url": "https://github.com/fastapi/fastapi/issues/3027"}

// search_code("def jsonable_encoder")
{"source_id": "fastapi/encoders.py#jsonable_encoder", "score": 1.0,
 "snippet": "fastapi/encoders.py jsonable_encoder",
 "url": "fastapi/encoders.py#L129-L192"}

// who_knows("dependency injection")
{"author": "sneakers-the-rat", "score": 5.0, "issues": ["issue_13399"]}

The one deliberate omission: no rerank

Look closely at search and you will notice what is not there. In post 4 the shipped default was hybrid + rerank (MRR 0.90). The MCP search tool is hybrid without the reranker: it returns the fused pool (recall@10 0.94, ordering MRR ~0.57) and stops.

That is not an oversight; it is the point. The reranker was an LLM reading candidates and deciding which one answers the question. On the other side of MCP there is already an LLM doing exactly that: the client agent reads the snippets and decides what to open. Running my own rerank first would be paying for judgment the agent is about to apply anyway. So the server hands over recall and lets the agent supply the ordering. The division of labor from post 5 survives; the synthesizer and reranker just moved across the wire.

Where the client's judgment actually matters

search_code makes the hand-off concrete, because its quality depends entirely on the pattern the agent chooses. Under the hood it greps the cloned source, then maps hits back to indexed code chunks; and it greps the whole tree, docs included, capping at 20 files. Watch what that means:

search_code("jsonable_encoder")      -> []          # 20 doc/*.md hits, none indexed as code
search_code("def jsonable_encoder")  -> fastapi/encoders.py#jsonable_encoder
search_code("class APIKeyHeader")    -> fastapi/security/api_key.py#APIKeyHeader

A bare token floods on documentation prose and returns nothing useful; def X / class X lands the definition instantly. A capable agent naturally reaches for the precise form, which is the whole bet of this post: the server stays simple and fast, and the intelligence (including "phrase the grep like a programmer") lives in the client. (It is also a fair to-do: search_code could rank indexed code files above docs instead of relying on the caller's precision.)

Wiring it up

The server ships as a console script, kb-mcp, that connects to the same Postgres and config as the CLI. Point any MCP client at it:

{
  "mcpServers": {
    "knowbase": {
      "command": "uv",
      "args": ["run", "kb-mcp"],
      "env": { "KB_DSN": "postgresql://knowbase:knowbase@localhost:5433/knowbase" }
    }
  }
}

Now inside Claude Code you can ask a question about FastAPI and watch the agent call search, read the snippets, call search_code for the exact symbol, follow a URL, and write its own answer, with my knowledge base as the retrieval substrate and none of my LLM budget involved.

What post 6 is

No new retrieval math and no new eval number: the tools are post 5's, and the same caveat holds (the eval scores retrieval, not what an agent does with it). What changed is who runs the loop. Posts 1–5 built a self-contained question-answering system; post 6 unbundles it into composable tools and gives them to whatever agent you already use. A knowledge base is more useful as a tool your assistant can reach for than as a separate app you have to visit.

Which is the segue to the last post. Post 7 builds the other front end (a plain web UI) for the humans who don't have an MCP client sitting in their editor.

Tool output here is real kb-mcp retrieval against the corpus from
the Results Appendix; the server is knowbase.mcp_server and its three tools wrap the same hybrid_search, grep_code, and who_knows used in post 5.
Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.

Rebuilding the Cerebras Knowledge Base: planner, tools, and synthesis

fr_trenton02 — Sun, 16 Aug 2026 12:20:30 +0000

Post 4 ended on a wall: two questions missed no matter how good the reranker got.

"Which function converts arbitrary objects into JSON-compatible data structures?" (jsonable_encoder)
"Where does FastAPI implement running background tasks after a response is returned?" (background.py)

I framed them as code-location lookups that similarity search structurally can't reach, and promised post 5 would route them to a grep_code tool that could. Then I built the whole ask pipeline, ran those two questions through it, and got a surprise that changed what this post is about: both of them answer correctly, and not because grep_code rescued them. This post is that surprise, and the machinery behind it.

The "misses" weren't misses

Here is the first question, run end to end:

$ kb ask "Which function converts arbitrary objects into JSON-compatible data structures?"
tools: search

The function is `fastapi.encoders.jsonable_encoder()`. It recursively converts supported objects—such as Pydantic models, dataclasses, datetime values, enums, and standard data structures—into JSON-compatible values. [1][3]

Sources:
[1] issue_3301  https://github.com/fastapi/fastapi/issues/3301
[3] issue_4834  https://github.com/fastapi/fastapi/issues/4834
...

And the second:

$ kb ask "Where does FastAPI implement running background tasks after a response is returned?"
tools: search

FastAPI wires `BackgroundTasks` into the response in `fastapi/routing.py`, while the actual post-response execution is handled by Starlette's response/background-task machinery. Tasks are attached to the response and run after it has been sent successfully. [1][2][4]
They do not run if the endpoint raises an exception instead of returning a response. [3][4]

Both correct. Both from tools: search alone, with no grep_code involved. What happened is that the eval grades recall against one gold document per question: for the first, the encoders.py chunk that defines jsonable_encoder. That chunk never enters the retrieval pool, so recall@10 scores it a miss. But the answer doesn't need that chunk; a dozen issue threads discuss jsonable_encoder by name, and synthesizing across them produces exactly the right answer with citations.

That is the real lesson to carry out of the retrieval posts: recall@k against a fixed gold doc undersells a system that can synthesize. The document was "missing"; the answer was not.

The pipeline that makes an answer

kb ask is four stages, and only the first and last cost a model call.

1. Plan. One LLM call picks tools under a strict schema:

KNOWN_TOOLS = ("search", "grep_code", "who_knows")

search is the entire post-1-through-4 pipeline (hybrid + rerank) and is almost always on.
grep_code and who_knows switch on only when the question calls for them. If the planner call fails twice, it falls back to search-only; the planner can add precision, never subtract it.

2. Retrieve. Each selected tool runs and returns rows.

3. Build evidence. Every row becomes a numbered block [n] with a real URL (path#Lstart-Lend for code, the issue link for a thread). Outputs from multiple tools merge, deduped by source and renumbered, so nothing appears twice.

4. Synthesize. One final LLM call, handed only the numbered evidence and one instruction: answer using only these blocks, cite [n] after each claim, and if the evidence doesn't answer the question, say so rather than guess. That last clause is the difference between grounding and a plausible-sounding hallucination.

grep_code: exact lookup, when you name the thing

So what is grep_code for? It fires when the question names a concrete symbol. Ask it the same jsonable_encoder question, but by name instead of by description:

$ kb ask "Where is the jsonable_encoder function defined?"
tools: search, grep_code

The `jsonable_encoder` function is defined in `fastapi/encoders.py`, at approximately lines 129–192. [1]

Sources:
[1] fastapi/encoders.py#jsonable_encoder  fastapi/encoders.py#L129-L192
[2] fastapi/encoders.py#jsonable_encoder@2  fastapi/encoders.py#L193-L242
[3] issue_1978  https://github.com/fastapi/fastapi/issues/1978
...

Now the planner routes to grep_code, and the exact code chunk arrives at rank 1 with a precise line range. Under the hood grep_code is deliberately simple: it shells out to rg (or git grep) over the cloned source for the pattern, then maps each path:line hit back onto the code chunks in the database by file and line range, scoring a chunk by how many matched lines fall inside it. No embeddings, no fusion; if the identifier is in the tree, the chunk that contains it comes back.

The honest boundary is the routing: grep_code triggers on questions that name a symbol ("where is jsonable_encoder"), not on questions that describe one ("which function converts arbitrary objects…"), because there is nothing to grep for in a description. That is the correct behavior, and it is exactly why the two P4 eval questions went through search alone: neither names its target. The synthesizer answered them anyway.

Re-attaching the context the chunk boundary cut

Chunking splits documents mid-thought. Before synthesis, each search hit gets its neighbors re-attached, appended as a [context] block (and only ever the neighbor text, never a second copy of the hit). Two real cases from the corpus:

A burst hit for "return a custom status code from a dependency" surfaces this fragment, which is a follow-up comment that barely stands alone:

@… I would also like to know how I can implement this. I want my dependency to trigger a redirect when it can't find a certain cookie…

Expansion re-attaches the parent thread, restoring the actual question the "this" refers to:

[context] # Returning a RedirectResponse in a dependency … How can I return a Redirect in a dependency to be executed in a conditional? … raise HTTPException(status_code=303, …)

A code hit for the same query matches fastapi/applications.py#put@2, the middle of the put method, where the status_code parameter is documented. On its own it is a floating docstring. Expansion pulls in the neighboring chunk with the method's actual signature:

[context] def put(self, path: Annotated[str, Doc(…)], *, response_model: Annotated[Any, …

The reranker decides which document; expansion makes sure that document arrives with enough around it to be worth reading. It doesn't move the retrieval numbers (kb eval scores retrieval, not the synthesized answer), but it is the difference between a usable citation and a dangling fragment.

who_knows: the same search, pointed at people

The third tool falls out for free. Ask who knows about a topic and who_knows runs the same hybrid search, then credits the people on the top threads (each thread's author at full weight by rank, each commenter at half) and returns a ranked list of names with the issues that earned them the score. No extra model call; expertise is just retrieval with the scores aimed at authors instead of documents.

What post 5 is, and what it isn't

The series runs on honest framing, so: the eval measures retrieval, not answers. The recall/MRR numbers from posts 1–4 grade whether the right document is in the ranking; there is no automated grade for answer quality, citation accuracy, or the planner's routing. So post 5's evidence is demonstrative (the worked kb ask transcripts above), not a new row in the table. That is also the honest ceiling of this post: I can show these examples are right; I can't yet report a synthesis-quality number across the whole set.

What it is: the point where the knowledge base becomes usable. Posts 1–4 built a retriever that ranks the right document first for most questions. Post 5 wraps it in a planner that routes symbol questions to exact lookup, re-attaches the context chunking cut, and turns the whole thing into a cited answer, including for the two questions the retrieval metric had written off.

The last two posts leave the terminal behind: post 6 exposes these tools over MCP so an assistant can call them, and post 7 puts a web UI on top.

Transcripts here are real kb ask output against the corpus measured in the Results Appendix; the architecture is the planner, grep_code, who_knows, expand, evidence, and synthesize components.
Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.

Rebuilding the Cerebras Knowledge Base: an LLM reranker

fr_trenton02 — Sun, 16 Aug 2026 12:20:15 +0000

Three posts in, the scoreboard was not encouraging. Naive vector search (post 1) was a decent baseline. Hybrid retrieval (post 2) made it worse. Distilling and bursting the corpus (post 3) made plain vector worse still (MRR 0.77 down to 0.63), while doing one useful thing: raising hybrid's recall@10 to 0.94. The right answer was in the top 10 for 29 of 31 questions; it just wasn't at the top.

This post is where that pays off. One LLM call per query, and MRR goes from 0.57 to 0.90.

The problem was never recall; it was ordering

By the end of post 3 the diagnosis was precise. Hybrid retrieval fuses a dense-vector list and a keyword list with reciprocal-rank fusion, and RRF is confidence-blind: it knows a document ranked #1 in the vector list, but not that #1 was a cosine-0.9 lock while the
keyword competitors were noise. A confident vector hit and a lukewarm keyword hit enter the fusion as equals. That is why hybrid's recall@10 is excellent (0.94) and its recall@1 is poor (0.39): the answer is in the pool, buried under topically-related decoys that RRF had no way to rank below it.

You cannot fix ordering with more retrieval. You fix it with something that can read two candidates and say which one actually answers the question.

The reranker

After hybrid produces its fused top-20, one LLM call reranks the pool. The candidates go in (id, source, a ~1.2k-character snippet each) and the model returns a ranked list of ids with scores, under a strict JSON schema. If the call fails or returns something malformed, it falls back to the original RRF order, so the reranker can only help, never break retrieval.
It is wired in as --rerank and requires --mode hybrid (it reorders the fused pool; there is nothing to rerank without one).

No new index, no new retriever, no corpus change. Same 16,315-document P3 corpus, same 31 questions. The only new thing is one model call per query.

The results

Metric	vector	fts	hybrid	hybrid + rerank
recall@1	0.52	0.45	0.39	0.87
recall@3	0.71	0.55	0.65	0.94
recall@10	0.81	0.65	0.94	0.94
MRR	0.63	0.50	0.57	0.90

Read the last column against the one before it:

MRR 0.57 to 0.90 (+0.33 from reordering a list we already had).
recall@1 0.39 to 0.87: the reranker moves the correct document to the very top for 27 of 31 questions, more than double hybrid alone.
recall@10 stays 0.94: this is the tell that the reranker is doing exactly what it should and nothing more. It reorders the pool; it can't add a document that wasn't fused in. Post 3 built the pool; post 4 sorts it.

For the first time in the series, a change beat every alternative on every metric. Hybrid + rerank doesn't just recover plain vector's precision (MRR 0.63); it far exceeds it.

This settles the question open since post 2

Since post 2 there has been one unresolved decision: hybrid loses to plain vector on MRR, so do we keep it as the default, or revert? Post 3 confirmed the loss survives a cleaner corpus (hybrid 0.57 vs vector 0.63). Taken alone, hybrid never earned its place.

But it was never meant to stand alone. Hybrid + rerank dominates every single-retriever configuration on every metric, and it only works because hybrid supplies the high-recall pool (0.94 at recall@10) for the reranker to sort. The two are a unit:

The default is hybrid + rerank. Hybrid provides the recall; the reranker provides the ordering. Neither is sufficient alone, which is why the previous three posts each looked like a failure in isolation.

Plain vector stays as a no-LLM fallback (MRR 0.63, one embedding, zero model calls); keyword-only is there for exact-string debugging. The shipped default is the pair.

Re-attaching the context the chunk boundary cut

The other half of post 4 doesn't move the retrieval numbers (because kb eval measures retrieval, not the final answer), but it matters the moment you actually read a result. Chunking splits documents, and the split routinely lands between a function and the decorator or imports that explain it, or isolates a two-line resolving comment from the thread that gives it meaning. So before synthesis, kb ask re-attaches neighbors: adjacent code chunks for a code hit, the full raw thread for a burst hit.

A concrete case: a query about returning a custom status code from a dependency matches fastapi/applications.py#put@2 (the middle of the put method, where the status_code parameter is documented). On its own that chunk is a floating docstring. Expansion re-attaches the neighboring chunk that holds the method's actual signature (def put(self, path: Annotated[str, …], *, response_model: …), so the citation arrives with the code that gives it meaning instead of a fragment.

The reranker decides which document; expansion makes sure that document arrives with enough around it to be worth reading. Post 5 shows the mechanism in full, including a burst hit reunited with its thread.

The ceiling the reranker can't lift

Two questions miss under hybrid, and they miss under rerank too:

"Which function converts arbitrary objects into JSON-compatible data structures?" (jsonable_encoder)
"Where does FastAPI implement running background tasks after a response is returned?" (background.py)

This isn't a tuning failure, and no reranker prompt will fix it: the reranker can only reorder the fused top-20. If the correct code chunk never enters that pool, it cannot be ranked into it. Both are natural-language code-location lookups (the answer is a code chunk that shares only diffuse vocabulary with the question), so neither the vector leg nor the keyword leg surfaces it into the pool in the first place.

The fix isn't a better ranker; it is a different retriever, one that treats "where is jsonable_encoder defined" as a symbol lookup, not a similarity search. That is post 5: a planner that routes a question to grep_code / symbol retrievers when it looks like a code-location query, instead of sending everything through the same dense-plus-keyword pool. Post 4 got the ordering right; post 5 goes after the two questions that were never in the pool.

Every number here comes from uv run kb eval on the same fixed 31-question set used since post 2; the full per-mode output is in the Results Appendix, with the corpus -rebuild details in the same post.
Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.

Rebuilding the Cerebras Knowledge Base: LLM distillation and bursting

fr_trenton02 — Sun, 16 Aug 2026 12:20:06 +0000

Post 2 ended with a diagnosis and a promise.
The diagnosis: hybrid retrieval lost to plain vector search not because keyword search is useless, but because the corpus is the problem (raw issue threads bury one resolving comment under dozens of noisy ones, so topical chatter outranks real answers). The promise: post 3 rebuilds the corpus and re-runs the exact same eval to see whether a cleaner corpus flips the verdict.

I rebuilt it and re-ran the eval, and plain vector search got worse (MRR 0.77 down to 0.63). That is two "made it worse" posts in a row, but this one is worse in a useful way: the same rebuild that hurt precision raised the recall ceiling to the level post 4 needs.

What I changed in the corpus

Two things, both aimed at the "answer drowns in its own thread" problem.

Distillation. Every issue thread now goes through one LLM call that rewrites it into a clean question-and-answer document (the problem, the resolution, the relevant symbols) with a strict JSON schema and a fall back to the raw thread if the model returns nothing usable.
The distilled text becomes the document we embed. The raw thread is kept in raw_content (it still feeds keyword search), and each result is cached by a content hash so re-ingesting is idempotent and doesn't re-pay the call.

On this corpus: 2,542 of 3,002 threads distilled cleanly; ~15% fell back to raw.

Bursting. A long thread has one comment that resolves it and eighty that don't. Embedding the whole thread as one vector means the resolving comment is averaged into oblivion. So I "burst" the high-signal comments out into their own vector-only rows (issue_N#burst_i), so each resolving comment gets its own embedding. Bursts are excluded from keyword search and from IDF stats, and they canonicalize back to their parent issue at scoring time so a thread can never occupy two slots.

The corpus went from ~3,700 documents to 16,315 (3,002 issue parents, 12,626 burst rows, 687 code chunks). Same 31 questions as before.

The results

Metric	vector (P2 corpus)	vector (P3)	hybrid (P2)	hybrid (P3)
recall@1	0.68	0.52	0.61	0.39
recall@3	0.84	0.71	0.65	0.65
recall@10	0.90	0.81	0.90	0.94
MRR	0.77	0.63	0.67	0.57

Vector dropped on every metric. Hybrid still loses to vector on MRR. The verdict from post 2 did not flip. The one number that moved the right way is hybrid's recall@10 (0.90 up to 0.94). Hold onto that one.

Why distillation hurt vector search

I spent some time to correctly understand what happened.

We embed the distilled document now, not the raw thread. Distillation does its job (it produces a tidy summary), and in doing so it throws away the verbatim text: the exact tracebacks, the literal error strings, the copy-pasted identifiers. The dense vector no longer reflects any of it.

What happens to the error-paste queries, which were hybrid's showcase in post 2:

TypeError: Object of type int64 is not JSON serializable was post 2's single best hybrid win (vector rank 5, promoted to hybrid rank 1 by the exact string). On the distilled corpus it falls out of vector's top 10 entirely; the summary dropped the literal string, so even the marginal rank-5 signal is gone.
The three error pastes vector already missed in post 2 (a websocket ASGI RuntimeError, a pydantic SchemaError, a ValueError: invalid literal for int()) stay missed, and now distillation has stripped the text keyword search leaned on too.
jsonable_encoder (the "which function converts arbitrary objects…" paraphrase) went from a vector rank-3 hit to a vector miss: the summary reads further from the query's wording than the original thread did.

Distillation trades verbatim-match ability for semantic tidiness. For a pure dense retriever scoring against the summary, that is a losing trade on this eval. It is a real cost, and the honest version of this series has to name it.

Why bursting raised the ceiling anyway

Bursts are the counterweight. Giving the one resolving comment its own embedding (instead of averaging it into a 90-comment thread) is exactly the fix for "the answer drowns in its own thread". But the effect only shows up at depth: hybrid recall@10 climbed from 0.90 to 0.94. Bursts pull one more class of answer into the top 10.

They do nothing for recall@1 or @3, and that is not bursting's fault; it is the same structural weakness from post 2. Reciprocal-rank fusion is rank-based and confidence-blind: it knows a document ranked #1 in the vector list, but not that #1 was at cosine 0.9 while the keyword competitors were marginal. More candidates in the pool, same inability to order them. Bursting fills the pool; it can't sort it.

So what did P3 actually buy?

Not a better score, but a better ceiling. After distillation and bursting, the correct document is in hybrid's top-10 for 29 of 31 questions; the recall pool is as good as it is going to get. What is missing is ordering: the right answer is in there, just not at the
top.

That is a different problem from "the answer isn't retrieved at all", and it has a different fix. You don't need more recall; you need something that can look at 20 candidates and say which one answers the question. Rank-based fusion can't; an LLM can.

Which is post 4. The reranker goes over hybrid's fused top-20 and reorders it, and for the first time in this series a change is an unqualified win: MRR 0.57 to 0.90, recall@1 0.39 to 0.87, on this exact corpus and eval set. P3 didn't raise the score; it built the pile of candidates that post 4's reranker finally sorts.

The two that P3 couldn't touch

Two questions miss even at hybrid recall@10, and they miss in post 4 too:

"Which function converts arbitrary objects into JSON-compatible data structures?" (jsonable_encoder)
"Where does FastAPI implement running background tasks after a response is returned?" (background.py)

Both are code-location lookups phrased in English: the answer is a code chunk that shares only diffuse vocabulary with the question. Distillation and bursting only touch the issue side of the corpus (they never rewrite code), so these were always out of reach here. They are the clean argument for post 5's dedicated symbol retrievers. Retrieval tuning won't reach them; a different retriever will.

Every number here comes from uv run kb eval on the same fixed 31-question set used since post 2; the per-mode output and the query-by-query breakdown are in the Results Appendix.
Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.

Rebuilding the Cerebras Knowledge Base: adding hybrid search

fr_trenton02 — Sun, 16 Aug 2026 12:19:59 +0000

Post 1 ended with a clear setup. Naive vector search had solved recall@10 but leaked on a specific class of query: exact error pastes and identifier lookups, where a sharp lexical signal gets buried under semantic lookalikes. That is the canonical argument for hybrid retrieval: combine dense vectors with keyword search so the exact match can't hide.

I built it, then I measured it. On my corpus, hybrid retrieval regressed against plain vector search. This post is about why, because the failure is more useful than the feature would have been.

First, sharpen the test

Before adding anything, I had to make the eval honest. My post-1 hypothesis was about lexical queries, but my 22-question set only had a couple of those. Grading hybrid on that set would grade it on a test that can't see its main advantage or its main cost. So I expanded the eval to 31 questions:

migrated the 6 code questions from line-range ids to symbol ids (path#symbol), so they survive re-chunking;
added 5 exact error-paste questions (tracebacks that appear verbatim in exactly one thread);
added 4 exact-identifier questions (rare function/class names).

I also re-ingested with a few fixes the raw corpus needed: paginating comment threads past the 100-comment API boundary, symbol-based code chunk ids, and a stale-row sweep after a full re-walk. Then I re baselined naive vector on the new corpus and eval set:

Metric	Naive vector
recall@1	0.68 (21/31)
recall@3	0.84 (26/31)
recall@10	0.90 (28/31)
MRR	0.77

The scores dropped from post 1, not because retrieval got worse, but because the new questions are harder on purpose. And there is the signal I was looking for: 3 of the 5 error-paste questions don't appear in the top 10 at all. Strings like

ValueError: invalid literal for int() with base 10: '5XX'

are maximally distinctive lexically, and the dense embedding misses them completely (beaten by threads that are merely about similar errors). This is exactly the hole hybrid is supposed to plug.

Building the hybrid retriever

Two new pieces, both hand-rolled (no search framework).

Full-text search rides Postgres' built-in GIN index over the raw thread text:

to_tsvector('english', coalesce(raw_content, '') || ' ' || document)

I score matches by summing IDF over the query's matched lexemes (document frequencies precomputed with ts_stat). Call it BM25-lite: rare words count for more, but there is no term-frequency weighting and (keep this in mind) no length normalization.

Reciprocal-rank fusion merges the two ranked lists. Each result scores Σ 1/(k + rank) across the lists it appears in, with k = 60:

RRF_K = 60

def rrf_fuse(lists):
    scores = defaultdict(float)
    for results in lists:
        for rank, r in enumerate(results, start=1):
            scores[r.id] += 1.0 / (RRF_K + rank)
    # ... return results sorted by fused score

The full pipeline: vector top-50 and FTS top-50, RRF fuse, cap at 3 results per source file (so one chatty file can't flood the page), then a small recency tie-breaker (ε·exp(−age/τ), τ = 180 days). No LLM anywhere; deliberately mechanical.

The results

Same corpus, same 31 questions, three modes:

Metric	vector	fts	hybrid
recall@1	0.68	0.42	0.61
recall@3	0.84	0.48	0.65
recall@10	0.90	0.65	0.90
MRR	0.77	0.47	0.67

Hybrid ties vector on recall@10 and beats FTS-alone on everything, but it loses to plain vector on recall@1 (0.61 vs 0.68), recall@3 (0.65 vs 0.84), and MRR (0.67 vs 0.77). The metric that shows it most clearly is recall@3 (0.84 down to 0.65). Folding in keyword search made the top of the ranking worse.

That is not the result I wanted to write up, so I dug into individual queries, and the picture split cleanly in two.

Where hybrid won (exactly as predicted)

On sharp-lexical queries, hybrid did its job:

TypeError: Object of type int64 is not JSON serializable: vector rank 5 to hybrid rank 1. The exact string lives in the target thread and nowhere near as densely in the decoys; IDF-weighted FTS finds it and RRF promotes it to the top.
"Why does my HTTPBearer endpoint return 403 instead of 401…": vector rank 2 to hybrid rank 1. Two near-duplicate issues have almost identical titles; vector picks the wrong one, and FTS's overlap with the correct thread's exact wording flips it.
AssertionError: Status code 204 must not have a response body: vector rank 2 to hybrid rank 1. Same mechanism.

Every one of these is an error paste or a near-duplicate-title disambiguation (the case I built hybrid for). When the lexical signal is sharp, fusion works.

Where it lost (and why)

The problem is everywhere else. Two failure modes did the damage.

Code-symbol lookups drown in chatter. Ask "Where does FastAPI implement running background tasks after a response is returned?" Vector ranks the background.py module #1 (exactly right); hybrid drops it out of the top 5. Five different issue threads say "background task" over and over in prose, while the module's own code has thin, diffuse lexical overlap by comparison, so FTS floods the top with threads and RRF has no way to say "but the vector match was already confident". jsonable_encoder (rank 3 to gone) and the API-key-header class (rank 5 to gone) died the same way.

Paraphrase questions get diluted. "How do I use dependency injection outside path operations…" went from vector rank 2 to hybrid rank 5; FTS surfaced three other dependency-injection threads that share vocabulary but aren't the answer.

Both failures trace to two structural weaknesses I built right in:

RRF is rank-based and confidence-blind. It knows a document was #1 in the vector list; it has no idea that #1 was at cosine 0.9 while the FTS competitors were marginal. A confident vector hit and a lukewarm keyword hit enter the fusion as equals.
IDF-sum FTS has no length normalization. For queries built from common words ("json", "object", "header"), the score spreads thin across many long threads, and the one short code chunk that actually answers the question can't win.

On this ~3,700-document corpus, the queries where hybrid hurts outnumber the ones where it helps. The win on error pastes is real, but it is a rounding error against the damage to code lookups and paraphrase questions.

So do you ship it?

Not as the default. On this corpus I would keep plain vector as the default retriever and reach for hybrid only on obvious error-paste input. That is the honest call the numbers support.

But "revert it" is the wrong lesson. Hybrid isn't failing because keyword search is useless; it is failing for two fixable reasons, and neither is "add lexical search":

The corpus is the problem. Raw issue threads bury one resolving comment under dozens of noisy ones, which is why topical chatter outranks real answers. Post 3 attacks this directly: distill each thread with an LLM, and "burst" out the high-signal comments so the answer isn't drowned by its own thread.
The fusion is the problem. RRF can't express confidence, so post 4 puts an LLM reranker over the fused pool, which can look at a candidate and say "no, that is just topically related".

Hybrid retrieval isn't the fix; it is the scaffolding the actual fixes hang off of. Post 3 rebuilds the corpus and re-runs this exact eval to find out whether a cleaner corpus changes the verdict.

Every number here comes from uv run kb eval on a fixed 31-question set; the full per-mode output and the query-by-query breakdown are the Results Appendix.
Code forthe series: github.com/faridgnank02/cerebras_knowledge_base.

Rebuilbing the Cerebras Knowledge Base : the simple vector baseline

fr_trenton02 — Sun, 16 Aug 2026 12:19:38 +0000

After encountering a blog post on Twitter/X from Cerebras on how they build their internal knowledge base How We Built Our Knowledge Base , I tried to rebuild the architecture from from scratch (no retrieval frameworks, one component at a time). Here, I propose what I worked on in a series of 7 post. The rule for the series is simple: every post fixes a failure the previous post demonstrated. I don't add a reranker because the papers recommend it; I add it when I can show the query it gets wrong today.

Post 1 is the naive version: one embeddings table, dense vectors, cosine similarity, top-k. It is the version most people start with. The useful question is how far it gets, and where it doesn't.

The corpus

I needed a body of knowledge with the same shape as an internal company one: an authoritative source of truth, plus a messy running conversation where the real answers live. Cerebras had their docs and their Slack. I used a single open-source project with two views:

Code: the FastAPI source, chunked.
Issues: the 3,000 most recent FastAPI issue threads (issue plus all comments), pulled raw from the GitHub API.

The issues are the Slack analog: someone pastes a traceback, a few people discuss it, and a maintainer leaves the one comment that resolves it. If retrieval can't find that comment, it can't do the job.

For post 1 the threads go in raw (no summarization, no cleanup). That is deliberate: I want to see what naive retrieval does with real, noisy text before changing anything.

The naive design

The whole thing is one Postgres table with pgvector:

CREATE TABLE embeddings (
    id          BIGSERIAL PRIMARY KEY,
    source      TEXT NOT NULL,     -- 'github_issue' | 'github_code'
    source_id   TEXT NOT NULL,     -- 'issue_1234', 'fastapi/encoders.py#jsonable_encoder'
    document    TEXT NOT NULL,     -- the text that gets embedded
    raw_content TEXT,              -- original thread/chunk, for display
    embedding   VECTOR(1024),      -- BGE-M3, HNSW cosine index
    metadata    JSONB NOT NULL,
    updated_at  TIMESTAMPTZ,
    UNIQUE (source, source_id)
);

Embeddings are BGE-M3 (1024 dims) run locally through sentence-transformers (no embedding API). Code is split with a small language-aware recursive chunker (split on class/function boundaries, not arbitrary line counts). Search is straightforward:

SELECT id, source, source_id, document, metadata,
       1 - (embedding <=> %s) AS score, updated_at
FROM embeddings
ORDER BY embedding <=> %s
LIMIT %s;

Embed the query, order by cosine distance, return the top k. No FTS, no reranking, no query planning. That is the entire retriever.

That gave me 3,000 issue threads plus 687 code chunks in the table.

Measuring it

You can't claim a retriever is good by eyeballing three queries, and you can't claim the next version is better without a fixed yardstick. So before writing any improvement, I wrote an eval harness: 22 hand-written questions, each tagged with the source_id that answers it. The metric is recall@k (is the right document in the top k?), reported at k = 1, 3, 10.

I chose recall@k because it maps to how the tool gets used: k=10 is "is the answer somewhere on the results page"; k=1 is "did we lead with it". The gap between them is the story.

The numbers

Metric	Naive vector
recall@10	1.00 (22/22)
recall@3	0.95
recall@1	0.77 (17/22)

On a ~3,700-document corpus, naive cosine similarity put the correct document in the top 10 every time. The "you need a sophisticated hybrid pipeline" narrative undersells where a plain vector index starts.

So if you're building internal search over a few thousand documents and skip straight to a multi-stage pipeline, you may be solving a problem you don't have yet. recall@10 was never the problem; precision was. recall@1 sits at 0.77, meaning one query in four doesn't lead with the right answer.

Where it actually fails

The interesting part isn't the score; it's the shape of the 23% that miss at k=1. A few representative cases:

TypeError: Object of type int64 is not JSON serializable (a verbatim error paste). The exact string appears in one thread's comments. Naive vector ranks that thread 5th, behind threads that are about JSON serialization in general. The dense embedding treats "semantically nearby" and "contains this exact string" as the same thing, and there are many nearby decoys.
AttributeError: 'Depends' object has no attribute ... (same story). The canonical thread ranks 4th behind lookalikes. An exact lexical match arguably should win, and cosine similarity has no notion of "exact".
"Which function converts arbitrary objects into JSON-compatible data structures?" The answer is the jsonable_encoder code chunk, but issue threads discussing jsonable_encoder outrank the function's own definition. This "chatter about the code beats the code" pattern recurs.

There is a pattern here: naive vector search struggles where the query has a sharp lexical signal (an exact error string, a specific identifier) that the dense embedding smears out. That is a nameable weakness, and it points at an obvious next move: add keyword search back in.

I'll note a counter-example, because it changes the plan. For a query like "allow_inf_nan=False is not enforced..." (a rare token, exactly what keyword search is supposed to own) naive vector already ranks the right issue #1. At this corpus size, dense search is stronger than the hybrid pitch assumes. So whatever I add next has to be argued at k=1 precision on error-paste queries, not accepted on general principle. That is the bar for post 2.

Two bugs the first real run found

A note for anyone doing this locally, because both cost me time:

BGE-M3 ran out of memory on Apple Silicon. Its default max_seq_length is 8192; encoding real issue threads at that length exhausts MPS memory. Capping the sequence length to 1024 and lowering the encode batch size fixed it with no measurable quality loss on this corpus.
Python 3.13 silently ignored my editable install. On macOS the uv editable .pth file gets the UF_HIDDEN flag, and Python 3.13 skips hidden .pth files without a word, so imports failed. Pinning to Python 3.12 fixed it.

One thing that worked on the first try: incremental ingest. Each connector keeps a watermark, so re-running the ingest after it has caught up writes ~0 rows. Boring and correct.

What's next

Naive vector-only search solved recall@10 and left a specific hole: exact-string and error-paste queries where a lexical match is buried under semantic lookalikes. That is the textbook motivation for hybrid retrieval (dense vectors and keyword search, fused).

So in post 2 I add full-text search and reciprocal-rank fusion, re-run the same eval, and find out whether hybrid fixes those queries. It is not a clean win, and the way it fails is more instructive than a win would have been.

The full 22-question eval and the k=1 miss list are in the Results Appendix
Code for the series: github.com/faridgnank02/cerebras_knowledge_base.