We Maxed Out the Reranker. The Expired Policy Still Won.

Building a Knowledge Base from Scratch, EP04

The setup

Last episode I polluted my knowledge base on purpose: an expired 2023 expense policy next to the 2026 rewrite, a broken spreadsheet that shattered into 11 of the library's 27 slices. The old policy outranked the current one at 0.7226 versus 0.7055. The standard advice for this failure mode is the same everywhere you read: turn on the reranker.

This episode answers it with twenty-two commands, not opinions. Same library (zj0knmrbye), baseline re-run first: five scores identical to the EP03 verification file, digit for digit. The library hadn't drifted. Everything below stands on that.

Discovery 1: reranking was already on

Before touching any parameter, I read the library's configuration:

bl knowledge info --index-id zj0knmrbye

There in the output: rerankModelName: qwen3-rerank. If you picked a ranking model when creating the library, it has been scoring every query since day one. So adding --rerank to the command changed nothing, all five results identical. The flag confirms the status quo. It doesn't start anything.

The proof that parameters do reach the server: swapping the model.

bl knowledge retrieve --index-id zj0knmrbye --query "报销发票怎么提交" --rerank --rerank-model qwen3-rerank-hybrid

All five scores shifted, from the 0.72 range to the 0.68 range. Different model, different scoring scale. But note the ordering didn't budge. Hold that thought.

The mode trap

--rerank-mode sits right there in the parameter table with three values: qa, similar, custom. I tried the aggressive version, custom mode with the instruction "优先返回2026年现行版制度", prefer the policy currently in force.

Results identical to baseline. Then I filled the mode field with a value that doesn't exist, xxx. The command ran fine. No error. When an invalid value produces no complaint, the parameter isn't being consumed at query time at all. The official docs have the matching sentence: the ranking model's mode is selectable only at library creation and cannot be changed afterward.

A flag being in the parameter table is not a promise you can set it at query time. This one is a library-level setting, locked at birth.

The arithmetic constraint

The recall parameters, --dense-similarity-top-k and --sparse-similarity-top-k, control how many candidates each lane feeds into the reranker's pool. Tested one lane at a time: nothing changes, this library's default is hybrid recall, both lanes or nothing. Both lanes at 5+5: the total field in the JSON drops from 27 (the whole library) to 6 (the merged candidate pool), and from third place down the cast changes.

Then the extreme values. Recall lanes at 1 and 1, rerank window at 10. The server refused it outright, HTTP 400:

{"message": "dense_similarity_top_k + sparse_similarity_top_k >= rerank_top_n", "api_code": "Index.InvalidParameter"}

The two recall lanes combined must be able to fill the rerank window. One plus one is two; two can't fill ten; rejected. The server states the pipeline's arithmetic constraint in the error itself. Honestly the best architecture documentation I've read all week.

One more behavior worth knowing: rerank window at 100 with both recall lanes at 200 returned only 17 of the library's 27 slices. The missing ten were dropped by the rerankMinScore 0.2 gate. What you ask for is a ceiling. What clears the bar is what you get.

The main event: rerank versus dirty data

Four controlled runs against the EP03 failure cases. --rerank alone. --rerank-mode similar. Custom mode with the explicit instruction to prefer the 2026 policy. The fragment question with --rerank. Every single one returned results identical to baseline, digit for digit. The expired policy held first at 0.7226. The broken-spreadsheet fragment held first at 0.7982.

The mechanism is not mysterious. A reranker scores candidate chunks by textual relevance to the query. The old policy's line, "staple the reimbursement form with original invoices and hand them to the accountant", reads exactly like an answer to "how do I submit invoices". A reranker scoring accurately still operates inside the frame of "more relevant goes first". It doesn't know which policy was superseded. That was never its job.

The model swap confirms it from the other side: qwen3-rerank-hybrid changed every score and moved zero rankings. Ordering is decided by textual relevance. The model choice sets the scale.

The phrasing gap

Parameters can't rescue dirty data, and it turns out they can't rescue phrasing either. Same fact, two askings. Literal: "报销发票怎么提交", the two policy versions neck and neck. Colloquial: "发票交了之后多久能报下来", how long until the money lands. The top five collapsed to the old policy at 0.4700 followed by four broken-spreadsheet fragments. The current policy fell out of the top five entirely.

Same pattern on the freight question. "新疆买沙发能发货吗" finds the answer fragment at 0.7982. Rephrase as "偏远地区买大件家具送不送", do you deliver big furniture to remote areas, and the winner becomes a fragment about free delivery in East China, 0.7420, purely on the literal match of the words for "large items". The actual answer fell to third.

Retrieval matches wording, not intent. Your users don't phrase questions the way your documents phrase answers. The enableRewrite setting in the library config is the platform's library-level compensation for exactly this.

A correction I owe

EP02 and EP03 both claimed knowledge base ingestion has no CLI command, console only. Wrong. bl knowledge --help lists 31 knowledge subcommands, and doc upload is one of them. I ran the full loop this episode: upload a temporary markdown file, watch the status come back PARSE_SUCCESS, clean up with file delete. With --index-id, upload and ingest complete in one step.

How the error happened: the local reference file with all 31 commands already existed at EP02 verification time. I checked the few I needed that day and treated "what I checked" as "what exists". A partial audit is not a full conclusion. Second episode in a row in that pit.

Two exhaust pipes, one engine

retrieve is now marked deprecated, with a successor called search that routes through a deployed retrieval service. Same query on both paths: five scores identical digit for digit. The differences are all in shape. search returns text as structured blocks, document name, title, body, ready for an agent's context. Its metadata carries the rerank score, the query score, and a cost_time of 321ms. And its total field counts actual results (5) instead of the candidate pool (27).

Tuning and experiments: retrieve, the full parameter surface is in your hands. Agent integration: search. Same engine underneath, so the twenty-two findings apply to both.

The triage order

Twenty-two commands compressed into a checklist, walked from the top:

1. Is the evidence present at all? Correct answer missing from the top five means corpus or phrasing problems. No parameter helps.
2. Run the two-phrasing check. Same fact, literal and colloquial. A drifting top result tells you more than any single high score.
3. Only then, parameters. top-k sizes the pool, top-n sizes the plate, and the arithmetic constraint is enforced with a 400.
4. Keep expectations for rerank level. It puts the more relevant first. It doesn't manufacture correct answers.

Try it

npm install -g bailian-cli
bl auth login --api-key sk-xxxxx
bl knowledge info --index-id your-library-id
bl knowledge retrieve --index-id your-library-id --query "how do i submit invoices"

Take the rule you trust least in your own documents, ask about its procedure in two different phrasings, and watch whether the top result survives the rephrase. That stability, or lack of it, is the honest health check of your library.

---

All twenty-two retrievals ran for real on the Bailian CLI against the zj0knmrbye knowledge base; request IDs are kept in the project repo. CLI install: Bailian CLI docs. API key: get one free, new accounts include free quota for 90 days.