Building a Knowledge Base from Scratch, EP05. The advanced arc begins.
The setup
Last episode closed the fundamentals arc with a cliffhanger: my knowledge base holds an expired 2023 expense policy, the 2026 rewrite that contradicts it, and eleven broken-spreadsheet fragments. Fixed retrieval ranks the expired policy first, 0.7226 to 0.7055, and no reranker setting moves it. The colloquial version of the invoice question drops the current policy out of the top five entirely.
This episode wires the library into an agent and watches what it does. Eight controlled runs, same library both sides. The control group is EP04's fixed retrieval. The experiment group is a new agent application in the console: pick a model (qwen3.8-max, chosen from bl model list --capability TG, which lays out fifty-four conversational models with context windows and prices), associate the knowledge base, switch on the thinking process, publish.
One command, and the agent's whole reasoning comes out:
bl app call --app-id 32d619eeea744a31ad892e3873afe487 --prompt "报销发票怎么提交" --has-thoughts --stream
The --stream flag matters. --has-thoughts alone returns only the answer. With both flags, the output arrives in three marked segments: [Thinking] (the judgment), [Action] (the tool call, search_knowledgebase in every run this episode), and [Observation] (the tool's return, embedding the full retrieval JSON: scores, the rewrite field, the reranker name, total 27). Cheapest agent-audit mechanism I know of.
A trap before the start
My first instinct was to skip the console step and call the retrieval service that the platform auto-created back in EP02. Its ID was on file.
bl app call --app-id aid-4d6b2f2fddd047c794a5f88963ef397b --prompt "报销发票怎么提交" --has-thoughts
HTTP 403, App.AccessDenied. The API key was fine. That resource is a retrieval pipeline built for bl knowledge search --agent-id, not an agent application. The official docs seal it: the old RAG-augmented applications stopped accepting new instances in May 2024, and the official path is associating your knowledge base with an agent application. Two kinds of resources, two roads. The 403 is the border post.
One design note on the application: the system prompt is a single sentence, "you are the customer-service assistant for Nuanwu Home Furnishing." Deliberately minimal. The exam questions are whether the agent notices the version conflict, rewrites colloquial questions, and admits ignorance on its own. Write those hints into the prompt and you're grading the prompt, not the agent.
Discovery 1: the expired policy still ranked first. The answer used the current one.
The agent's retrieval results had the old policy first, 0.5655 to 0.4713. The answer was pure 2026 policy: electronic invoices, thirty days to upload through Enterprise WeChat, five working days for review. The old policy's fifteen-working-days-and-staple-paper routine never appeared.
The thinking trace holds the pivot line: "我应该以现行版为准回答" (I should answer based on the version currently in force). The agent had read the current policy's opening sentence, "the 2023 version is hereby superseded." That sentence is metadata the document carries about itself. The retrieval layer sorts by textual relevance. The decision layer reads context and picks.
The reverse check made it solid. Ask "住宿标准是多少" (what's the lodging cap) and this query's ordering flips, current policy first at 0.8512. Same answer either way. The ranking swings with the query. The answer doesn't swing with the ranking.
Discovery 2: the colloquial question got rescued, and the rescue is visible
EP04's most painful result: ask "发票交了之后多久能报下来" (how long until the money lands) and the current policy falls out of the top five, replaced by the old policy at 0.4700 plus four spreadsheet fragments.
The agent answered correctly. Five working days, citing the current policy. The interesting part was in the Observation metadata, a rewrite field:
"rewrite": "发票报销多久能报下来 报销周期"
The server had rewritten the colloquial question into document-style phrasing before retrieval. EP04 mentioned the enableRewrite config with no behavioral evidence. Now the evidence exists: control group direct search on the same question produces the 0.47-range fragment pileup; the agent channel has the current policy comfortably present. Two layers in relay, both on by default, zero code.
Discovery 3: the broken fragment got used as evidence
Ask "新疆买沙发能发货吗" (can a sofa ship to Xinjiang). The agent's retrieval: the broken-spreadsheet fragment first at 0.8305, the clean document second at 0.7245. Then my favorite passage of the episode, the agent comparing sources:
The clean version provides general information about large-furniture delivery but doesn't specifically exclude remote areas.
It noticed the granularity gap. The fragment has rotten formatting but is an explicit clause naming remote regions. The clean document is complete but vague on this specific question. The final answer is stitched: the no-large-items conclusion from the fragment, the 6-8 day window and 15 yuan surcharge context from the clean document. The eleven fragments went from retrieval killers to the decision layer's ugly-but-usable evidence.
Discovery 4: it decided not to search at all
Second turn, with --session-id attached, asking what the 2023 policy said. No tool call this turn. The trace: "based on the previously retrieved results, the 2023 policy states: review takes about 10 working days." It reused the previous turn's slices, answered correctly, and volunteered a comparison with the current process. "Who decides what to search" includes the third case: who decides not to. Sessions expire after one hour without requests, so pace your experiments.
Discovery 5: honest refusal
Ask it to compare shipping policies with IKEA. The library holds nothing about IKEA. The trace: "I should not compare ourselves with other companies in a way that fabricates." The answer states its own policy in full (all library-sourced), declines to speak for IKEA, and redirects: tell me what you care about and I'll walk you through our terms. A move straight out of human customer-service training.
The turn: it rescued the version conflict, and silently fumbled the threshold conflict
Then the cold water, from an unplanned discovery. The shipping answer contained "in-stock items ship within 48 hours of payment." Suspicious, since neither shipping document seemed to say that. I searched the library specifically. Real content, document 01, retrieval score 0.9456. False alarm on the hallucination, but the search surfaced the real problem: the library holds two free-shipping thresholds. Document 01 says free shipping over 59 yuan. Document 13, the clean table, says free over 99. Both were in the agent's retrieval results that turn, 01 scoring higher at 0.6557 to 0.5755.
The agent took the 48-hour timeframe from 01 and the free-shipping-over-99 rule from 13. By topic, from each what it needed. Not one word about the 59-versus-99 conflict. It silently output one of two contradictory policies.
Compare with the version conflict and the boundary is exact. The expense policies carry their arbitration basis in plain text: "the 2023 version is hereby superseded." The agent reads that and arbitrates. The shipping documents carry nothing. The agent can't arbitrate, and doesn't even register that arbitration is needed; from its seat these look like two topics, not one conflict.
The deepest finding of the episode: the decision layer rescues conflicts that carry evidence chains and cannot rescue conflicts missing metadata. Gartner's much-quoted number says 80% of enterprise agent projects never reach production over hallucination and consistency problems. This experiment is a micro-footnote: the agent's judgment filled real retrieval-layer holes, and the metadata debt in the library it cannot carry for you.
Command surface notes
--pipeline-ids: with the application already associated to a library, passing the correct ID changes nothing (redundant), and passing a fake one (fakeid123) also changes nothing, silently, no error, still the mounted library. You think you switched. You didn't. Same pattern as EP04's --rerank-mode accepting garbage values. A parameter's presence in the CLI table is not a promise the server validates it.
--session-id: first call's response carries the ID, pass it back for automatic context, cloud storage expires after one idle hour.
Try it
npm install -g bailian-cli
bl auth login --api-key sk-xxxxx
bl app call --app-id your-app-id --prompt "your policy question" --has-thoughts --stream
Before you ship an agent on top of a real document library, take your most conflict-prone business rules, run them with --has-thoughts --stream, and read the Observation sections. Two documents present, two different numbers, one silent answer: that's the failure you want to find before your users do.
All eight agent calls and retrieval controls ran for real on the Bailian CLI; full thinking traces and request IDs are kept in the project repo. CLI install: Bailian CLI docs. API key: get one free, new accounts include free quota.
Top comments (0)