It Found a Contradiction I Didn't Plant. Then It Answered as if Nothing Was Wrong.

Building a Knowledge Base from Scratch, EP06. The paradigm arc starts here.

Where EP05 left off

My library holds two shipping documents. One says free shipping over ¥59, the other says over ¥99. Nothing anywhere says which is current. Both came back in the same retrieval call, the ¥59 one actually scoring higher at 0.6557 to 0.5755, and the agent quietly picked one. It never mentioned a second number existed.

Version conflicts it handled fine, because the 2026 expense policy contains the sentence "the 2023 edition is hereby superseded." Metadata sitting in the corpus, readable by anyone with context. The threshold documents carry nothing, so there was nothing to read.

Five episodes of retrieval hit that wall. This episode changes the paradigm instead of the plumbing: have the model read the entire library once, write it up as structured knowledge pages, then answer from those pages with no retrieval at all.

Karpathy's LLM Wiki gist frames the two as interpreter versus compiler. RAG interprets your library at question time, every time. LLM Wiki compiles once and then executes. Three directories (raw/ read-only sources, wiki/ pages with [[wikilink]] cross-references, schema/ conventions), three operations (Ingest, Query, Lint).

One thing to flag before anything else. That gist carries its own disclaimer: "This document is intentionally abstract. It describes the idea, not a specific implementation." So the secondhand write-ups claiming "Karpathy's research shows accuracy falls off a cliff past a certain document count" are straightforwardly false. There is no experiment in that document. Somebody attached fabricated numbers to a design note and the numbers spread.

This episode runs the idea against a real library: zj0knmrbye from EP02, eight documents, 27 chunks, with the dirty data I planted in EP03 still in place. Not a single character changed.

The command surface

One entry point, bl text chat with --messages-file:

bl text chat --model qwen3.8-max --messages-file messages.json \
  --max-tokens 8000 --enable-thinking --thinking-budget 3000 --output json

messages.json is a plain messages array, one system turn and one user turn carrying the full text of all eight documents merged, 8,811 characters.

Two practical notes if you're on Windows. Put non-ASCII system prompts inside the messages file, never on the command line, because PowerShell mangles the encoding on the way through. And write results to a UTF-8 file instead of printing them: one emoji in the output is enough to make print raise UnicodeEncodeError against the console's GBK codec.

Thinking stays on deliberately. The EP05 agent app runs with thinking enabled, so compiling with it off would have made the comparison meaningless.

The messy spreadsheet got the least dignified treatment available. openpyxl dumps every cell to text, and the empty columns left by merged cells, the misaligned headers and the notes stranded on a second worksheet all survive verbatim:

Shipping zone |  | Lead time and fee

Why I ran two nearly identical compiles

Karpathy's Ingest step includes an instruction to "flag contradictions between new and existing content." If I copy that in and the model duly flags the ¥59/¥99 conflict, the result tells me nothing. I can't separate "giving the model global visibility worked" from "telling the model to hunt for contradictions worked." The EP05 agent's prompt contained nothing like that instruction.

So, two runs differing by one line:

• A1a, neutral. Build topic pages, merge same-topic content, produce an index.md, cross-link with [[page name]], cite the source document for every fact, add nothing the documents don't state. The words "contradiction," "conflict," "59" and "99" appear nowhere.
• A1b, Karpathy's version. Same text plus one line: "5. While ingesting, flag contradictions between new and existing content."

Reading rule fixed in advance: if A1a finds the conflict, credit goes to global visibility. If only A1b finds it, credit goes to the prompt and the conclusion shrinks accordingly.

Measured usage from --output json:

Run	Input	Output	of which reasoning	Wall time	Cost
A1a neutral	5,905	9,594	1,725	179.7s	¥0.4162 (~$0.059)
A1b Karpathy	5,918	8,757	2,100	175.8s	¥0.3863 (~$0.054)

Pricing from bl model list --model qwen3.8-max, which needs no authentication: ¥12 per million input, ¥36 per million output. USD figures use 7.1 CNY/USD on the run date and are there for scale, not accounting. finish_reason was stop on both, so the coverage gaps below are choices, not truncation.

Finding 1: the neutral run built a conflict table on its own

A1a produced a section I never asked for:

## Free shipping threshold and fees

> ⚠️ Document conflict: 01-shipping-and-logistics and 13-shipping-fee-table-clean
>   disagree on the free shipping threshold and the fee amount.
>   Treat the amount shown at checkout as authoritative.

| Item | 01-shipping-and-logistics | 13-clean / 14-messy |
| Standard zone threshold | ¥59 | ¥99 |
| Fee below threshold | ¥10 | ¥8 |
| Remote zone threshold | free over ¥129 | excluded from promo, ¥15 surcharge |

Same two documents as last episode, same conflict, and the retrieval side picked one silently. Nobody told this run to look for contradictions. It put both versions side by side and volunteered a resolution, defer to checkout, that it lifted from a throwaway line buried in the messy spreadsheet.

By the rule I fixed in advance, the conclusion stands: what did the work was global visibility, not the instruction.

A1b flagged more explicitly, three conflicts under a 🔴 marker with source columns. It also declined a fake one: document 02 gives a one-year appliance warranty, document 15 lists warranty periods for wood furniture, hardware and mattresses and says nothing about appliances. Its note reads "different product categories; no direct conflict, but the information is complementary." Refusing to pad the count tells you more than an extra flag would.

Finding 2: the second conflict wasn't on my list

When I seeded dirty data in EP03 I designed exactly two conflicts: the shipping threshold and the policy version pair. The third one A1b found by itself, in how the two documents define remote zones:

• 01-shipping-and-logistics: Xinjiang, Tibet, Inner Mongolia, Qinghai, Ningxia, Hainan
• 13-shipping-fee-table-clean: Xinjiang, Tibet, Gansu, Qinghai, Inner Mongolia, Ningxia

One list has Hainan and no Gansu. The other has Gansu and no Hainan. I checked the sources. The model was right.

I built this library and ran five episodes of experiments against it. That mismatch sat there the whole time and I never noticed. This carries more weight than finding the traps I set on purpose: with a planted trap you can reasonably suspect I hinted at it somewhere. I couldn't have hinted at this one.

The academic name is inter-context conflict, one of three categories in the EMNLP 2024 survey Knowledge Conflicts for LLMs. Retrieval rarely surfaces this class, because two chunks that disagree usually don't get read together and nothing in the pipeline is responsible for comparing them.

Finding 3: the sharpest objection didn't reproduce

The strongest critique I found is about overwrite behavior: during compilation the new revision silently replaces the old one, even though somebody may still be running the old value. The example is a hardware datasheet where Rev 6 specifies 100nF and Rev 7 changes it to 220nF, so the engineer holding a Rev 6 board can no longer look up the value that applies to them.

My two expense policies are structurally the same case. Neither run overwrote anything. A1a kept both editions as separate index entries, gave each a standalone page with its full allowances, put a banner on the old one (⚠️ Superseded. Effective 2023-04-01, replaced 2026-02-01. Historical reference only.), and [[link]]ed the two pages to each other.

The boundary matters though: both documents state their effective dates and supersession relationship in their own body text. The metadata was there to read. EP05's line about a decision layer rescuing conflicts that carry evidence holds in this paradigm too.

Finding 4: what verification caught

Fabricated values, orphaned pages and dropped information can't be assessed by reading output and nodding. I scripted it: pull every numeric token out of the artifact and look for it in the source corpus, then check every [[link]] against the set of headings.

Check	A1a neutral	A1b Karpathy	Objection tested
Artifact size	12,089 chars / 578 lines	10,089 chars / 433 lines	reference
Numbers traced to source	73 distinct, 0 unfound	71 distinct, 0 unfound	fabrication: didn't occur
[[link]] integrity	42 links / 11 targets, 0 broken	7 links / 7 targets, 0 broken	orphans: didn't occur
18 key facts spot-checked	18/18	17/18	dropped info: partly occurred

Both runs lost the same thing: the timestamp on the messy spreadsheet's footer, "subject to change, latest support response governs (updated 2024.6)."

That is not decoration. Document 01, the other side of the threshold conflict, carries no date at all. So "which of these two is more recent" has exactly one clue in the entire library, and compilation dropped it. What compilation loses is precisely the metadata arbitration depends on.

A1b dropped three more things A1a kept, including the manual's own note that "version v3.1 (March 2026) is superseded." Sit with that one: A1b flagged the expense policy's version relationship more explicitly and simultaneously lost the manual's. It also came in 2,000 characters shorter with [[link]] count down from 42 to 7. Attention is zero-sum inside a fixed output budget. Copying Karpathy's instruction in isn't a free upgrade, it's a trade.

The cold water: the artifact says "conflict," the answer doesn't

A compiled artifact isn't for human reading, it's context for the model. So the real test is loading it as context and asking the exact EP05 questions, side by side. Terms: the compiled side's system prompt is one sentence naming its role, matching the EP05 app. The artifact used is A1a, the neutral one, because using the more explicitly annotated A1b would be stacking the deck.

Shipping threshold question, compiled side:

Standard zones: free over ¥99 / ¥8 fee below ¥99
Remote zones (Xinjiang, Tibet, Gansu, Qinghai, Inner Mongolia, Ningxia):
  excluded from promotional free shipping, ¥15 surcharge applies

> Note: our rules changed recently, so please treat the shipping
>   amount shown at checkout as authoritative

Retrieval side: "Orders over ¥99 ship free (remote zones excepted). Below ¥99, an ¥8 fee applies."

Both report one number. Neither mentions ¥59. And the compiled side does something slightly worse: the artifact it's reading says "document conflict" in plain text, and the answer rephrases that as "our rules changed recently." Nothing in the library says the rules ever changed. That framing is invented, and it converts uncertainty into false confidence.

So the accurate claim isn't that compilation surfaces conflicts to users. It's that compilation solves finding the conflict and does nothing about telling anyone. Two separate jobs, and that gist only covers the first.

I reran the version questions too, formal and colloquial phrasing. Both sides answered correctly from the 2026 edition, 30 days to submit, five business days to review. Switching paradigms bought nothing there, for the same reason as before: the metadata was intact and retrieval was already handling it.

Where they actually diverge

The fourth question got it out: a customer in Xinjiang wants a sofa, can you ship it, is cash on delivery available, how is shipping calculated? Answering forces the remote-zone rules from both documents into one answer.

Retrieval side:

- Remote zone orders ship free over ¥129; below that, an ¥8 remote surcharge
  applies on top of the standard fee.
- Remote zones are excluded from promotional free shipping and require
  a ¥15 fee adjustment.

Those two bullets are mutually exclusive. Presented as a numbered list, they read like two rules that both apply. A support agent reading this to a customer contradicts themselves mid-sentence.

Compiled side, same question:

| Source document | Rule |
| 01-shipping-and-logistics | Free over ¥129 in remote zones, ¥8 surcharge below |
| 13/14-shipping-fee-table | Remote zones excluded from promo, ¥15 adjustment |

> ⚠️ The two documents conflict. Treat the checkout amount as authoritative.

Split by source, with a warning. Cleanest divergence in the episode, and the mechanism is easy to state: the compiled side put those two rules next to each other once, at compile time, judged them incompatible, and that judgment persisted in the artifact. The retrieval side never saw the library whole. It got two chunks and had no basis for deciding they were incompatible.

Put that with the previous section and the gain is narrow and specific: compilation pays off when two conflicting rules have to enter the same answer, because it knows they conflict. On simple single-fact questions it picks one silently, exactly like retrieval.

Cost went the opposite way from what I expected

I assumed this section would be where compilation died. Stuffing an entire library into context on every question has to cost more than retrieving a few chunks. It doesn't.

The four compiled-side questions ran back to back, and from the second one on prompt_tokens_details.cached_tokens had a value:

Call	Input	Cached	Output	Cost
Q1	7,929	0	370	¥0.1085
Q2	7,935	7,168	388	¥0.0339
Q3	7,932	7,168	287	¥0.0303
Q4	7,942	7,168	665	¥0.0440

The artifact prefix bills at the cache-hit rate, ¥1.5 per million, 12.5% of standard input. Only the few hundred tokens that change pay full price. Input cost per question fell from ¥0.0951 to ¥0.0200, and I configured nothing. The server matched the prefix implicitly.

Head to head:

Same question	Compiled (cache hit)	Retrieval	Delta
Shipping threshold	¥0.0339	¥0.0753	compiled 55% cheaper
Xinjiang sofa (cross-document)	¥0.0440	¥0.1431	compiled 69% cheaper

Two reasons. The retrieval side's 4,739 input tokens get no discount at all, everything at ¥12. And retrieval generates more output, because it emits tool calls and restates what it retrieved: 1,510 tokens against 665 on the cross-document question, and output is the expensive line item at ¥36 per million.

Compiling costs ¥0.4162 once. Steady-state savings averaged ¥0.0703 per question, so ¥0.4162 ÷ ¥0.0703 ≈ 6. Roughly six questions in, the compile pays for itself.

That 6 rests entirely on the cache. Look at Q1 again: with no cache hit, compiled costs ¥0.1085 against retrieval's ¥0.0753, which is 44% more. Heavy consecutive use breaks even at six and then runs half to two thirds cheaper. Sparse occasional use pays a cold start every single time and never breaks even.

My four questions ran inside a minute of each other, so the second one hit. How long the prefix survives is a server-side policy question, and you'll want to measure it against your own traffic. Which lines up with the boundary Karpathy drew himself: if you just need to look one thing up, RAG is fine. He offered that as intuition. This is the invoice.

Writing a human decision back in

The other RAG criticism in that gist is the absence of accumulation. Every question starts from zero, so yesterday's conclusion doesn't carry. A compiled artifact is a file, so you can write conclusions into it.

I simulated what an operations team would do, appended to the end of A1a: operations confirms ¥99 and ¥8 are current, the ¥59 and ¥10 figures are pre-2024 and must not be used, agents must also tell the customer the threshold changed, and the remote-zone conflict is not yet decided so escalate to a team lead. That last line is deliberate: one conflict resolved, one explicitly left open, so I could see whether it distinguishes them.

Same shipping question, new answer:

Current rule: free shipping over ¥99 / ¥8 fee below ¥99

⚠️ Heads up: this threshold changed recently. If you've seen "free over ¥59"
   somewhere, that's the old rule and no longer applies. ¥99 is current.

One more thing: remote zones have their own rules. If your delivery address
   is in one of them, let me know and I'll confirm the details for you.

¥59 appears in a customer-facing answer for the first time, with the correct disposition attached. On the conflict I left undecided it didn't force an answer, it asked for the delivery address and offered human follow-up. And the internal instruction to escalate didn't leak.

There's a practical finding buried in this call. cached_tokens was still 7,168. The arbitration log went at the end, so the prefix didn't change and the cache survived. Append rather than insert, and incremental maintenance stays cheap.

The cost of all this is that a human made the call. Compilation didn't tell me whether ¥59 or ¥99 was right. It put the disagreement in front of me.

Limits

The context window is not the constraint here. bl model list --model qwen3.8-max reports a 1,000,000 window and 991,808 max input tokens. Those 8,811 characters compile to 5,905 input tokens, so 991,808 ÷ 5,905 ≈ 168. The library can grow 168× before the window matters.

One asymmetry belongs on the table. The compiled side read the full text dump of the messy spreadsheet; the retrieval side read the 11 chunks it was split into. The compiled side genuinely saw a more complete table. That follows from the definitions, compilation eats documents and retrieval eats chunks, but part of the divergence on the cross-document question comes from seeing more, and that part shouldn't be credited to the paradigm.

Update costs are lopsided. Change one document and the compiled side re-runs the whole compile, ¥0.4162 and about three minutes, and the new artifact invalidates the cache prefix so the next question pays ¥0.1085 instead of ¥0.0339. The retrieval side re-uploads and re-indexes that one document at effectively zero cost.

Cases where I'd skip compilation, in priority order:

1. Libraries needing per-user access control. A compiled artifact merges cross-department knowledge onto shared pages, which breaks permission boundaries by construction. A retrieval pipeline can pin tenant_id and filter before search. I haven't seen this objection raised publicly.
2. Libraries that change often, per the update cost above.
3. Sparse, occasional questions, where no cache hit means every question costs more.
4. One-off lookups. Karpathy's own boundary.
5. Personal knowledge management, which inverts the enterprise case. Organizing material is itself how learning happens. A company library exists for other people to query, so automating it costs nothing. Automate your own study notes and the learning stops.

Try it

npm install -g bailian-cli
bl auth login --api-key sk-xxxxx
bl model list --model qwen3.8-max
bl text chat --model qwen3.8-max --messages-file messages.json --output json

Ten billed calls, ¥1.27 total, about eighteen cents.

My read: this isn't a choice between two options. On this library, compilation delivered visible conflicts and durable decisions. It did not deliver disclosure to the end user, so you'll need prompt or process work to cover that gap. And the economics rest entirely on cache hits, so a change in question cadence flips the math.

Last episode I said the curation debt comes due no matter what. Switching paradigms changes one word: compilation will itemize the debt for you, but you still pay it. It put a mismatch that had been sitting in my library for five episodes right in front of me, and in the same pass it threw away the timestamp needed to settle it.

Next episode continues the paradigm arc. I fed the artifact in as one block and got implicit cache hits from the second question on, which saved real money, but I have no idea when that prefix expires because it's entirely server-side. Model Studio also offers explicit caching, ¥15 per million to create and ¥1 per million on hits, with rules that work nothing like the implicit path. EP07 runs both against this same artifact.

---

Everything here was measured with Model Studio's CLI (bl). Full usage figures, reasoning traces and compiled artifacts from all 10 calls are archived in the project repo. Command signatures may shift between versions, so check the official docs. API keys are free to obtain, with a free tier for new accounts.