Building a Knowledge Base from Scratch, EP01
Everyone worries about LLM hallucinations on company-internal questions. In 2026, the standard move is pasting a policy file into the chat first, so hallucination feels handled. I ran three controlled experiments on that move itself, and found three failure modes that have nothing to do with hallucination: it improvises around missing files, it never questions the version you hand it, and it forgets everything by the next session.
This post walks through all three with reproducible commands. It's the first episode of a series that builds a knowledge base from zero.
The setup
- Tool: Bailian CLI (
bl), the CLI for Alibaba Cloud's Model Studio - Model: Qwen3.8-Max
- Question domain: a company travel policy. A real reimbursement answer spans the policy PDF, an OA announcement, a shared-drive form template, and a group-chat notice. No one pastes all four every time.
Install and authenticate if you want to follow along:
npm install -g bailian-cli
bl auth login --api-key sk-xxxxx
An API key with free quota is available in the Model Studio console.
Experiment 1: paste half the policy
I pasted only the standards section (450 RMB cap for first-tier cities, over-cap not reimbursed, 30-day filing window) and left out the approval-process section, then asked whether a 550 booking could be reimbursed and what the process was.
bl text chat --model qwen3.8-max --message "根据以下公司制度回答问题：《星野科技差旅报销制度（2026年3月修订）》（一）住宿标准：一线城市450元/晚，二线城市350元/晚，其他城市300元/晚。单人单晚超出标准部分原则上不予报销。（二）报销时限：出差返回后30日内在财务系统提交，逾期视为放弃。问题：我下周去上海出差，订了550元一晚的酒店，能报销吗？具体流程是什么？"
The standards half was flawless: 450 cap, 100 over, reimbursed at 450. The process half started leaking. It admitted the document "doesn't specify an exception process," then filled the gap with industry common sense: invoice, payment record, trip approval form. None of that was in the text I gave it. Its reasoning log said it plainly: the policy doesn't detail this, but suggestions can still be offered.
The control run with the full policy produced the complete chain: OA approval form, two director signatures, four evidence types, 30-day deadline. One file of distance between the two answers.
Experiment 2: paste the outdated version
I swapped in the 2023 edition (cap 600, every other clause identical) and asked: booking at 600 gets fully reimbursed, right? If so I'll place the order.
bl text chat --model qwen3.8-max --message "根据以下公司制度回答问题：《星野科技差旅报销制度（2023年1月修订）》（一）住宿标准：一线城市600元/晚，二线城市450元/晚，其他城市350元/晚。单人单晚超出标准部分原则上不予报销；因会议安排等特殊情况超标的，须出差前在OA系统提交《超标住宿审批单》，经部门总监与财务总监双签后方可报销。（二）报销时限：出差返回后30日内在财务系统提交，逾期视为放弃。问题：我下周去上海出差，准备订600元一晚的酒店，按这个制度能全额报销吧？没问题我就下单了。"
It ran a diligent checklist: city tier, tax-inclusive pricing, single occupancy, the 30-day window. It even warned that 601 would tip it over. Then it waved me through: no over-cap, fully reimbursable, no approval form needed.
Not once did it ask whether a 2023 policy is still in effect. The 2026 revision caps it at 450. Booking at 600 on the old file means paying 150 a night out of pocket. The AI said nothing wrong; the file was wrong, and the model has neither the duty nor the ability to suspect your file.
Experiment 3: new window, no file
bl text chat --model qwen3.8-max --message "我们公司规定一线城市出差住宿费每晚报销上限是多少？"
The reply: I don't have your company's internal policy. Everything you pasted yesterday is gone.
Why this argues for a knowledge base
The manual-paste move has a structural flaw beyond inconvenience: you usually don't know which file the answer lives in. Reimbursement spans the policy, the OA notice, the group announcement. Support spans the product page, the after-sales doc, the latest Excel from ops. You can't paste what you can't find.
RAG (retrieval-augmented generation) fixes all four gaps at once: ingest every source in every format once, retrieve across files at query time, serve everyone the same currently-effective version, retire old versions at the document layer. Store, retrieve, answer. That's the architecture behind most internal Q&A systems since 2023, and the frontier keeps moving: agents that decide what to retrieve (Agentic RAG), Karpathy's compile-don't-retrieve LLM Wiki proposal from this April, and PixelRAG's screenshot-based retrieval.
The series continues with EP02, where we build an actual knowledge base in the Model Studio console, load the policy, announcements, and FAQ together, and query it with one command. Fifteen minutes, end to end.
All commands and raw responses are kept in the project repo for verification. CLI docs: Bailian CLI.
Top comments (0)