A knowledge base built for people and a knowledge base built for AI agents look similar from the outside and behave very differently in use. People navigate: they remember roughly where something lives, open the page and read around the part they need. An agent does none of that. It issues a query, receives some passages, and answers from whatever came back. Everything that makes the human experience tolerable — browsing, skimming, asking a colleague which page is the current one — is unavailable to it. This is a practical checklist of what a knowledge base has to provide before connecting an agent to it produces answers you would let a customer see.
Agent Access Is Retrieval, Not Search
A search box returns a ranked list and lets a human decide which result is relevant. That last step is doing enormous work, and it disappears when an agent is the reader. The agent takes the passages it receives as the truth of the matter, so anything ranked third that should have been first becomes a wrong answer rather than a slightly inconvenient click.
This shifts where quality has to live. For humans, a good knowledge base needs decent titles and a search that surfaces plausible candidates. For agents, it needs retrieval that is right on the first attempt, because there is no second attempt and no judgement applied to what comes back.
The implication for evaluation is direct: judge a tool by the answers it produces to your real questions, not by how its search results page looks. A demo that returns five relevant-looking documents tells you very little. A demo that answers 'what is our refund window for annual plans' correctly, with the source attached, tells you everything.
Five Requirements Worth Checking First
First, live retrieval rather than a copy. If the integration works by exporting documents into another tool, the export is stale from the moment it finishes and someone owns the job of refreshing it. Second, citations on every answer, naming the document the content came from, so a wrong answer can be traced to either bad retrieval or a bad source.
Third, a clean failure mode. The knowledge base must be able to say it does not know. An agent that improvises when retrieval returns nothing is worse than no agent, because the failure is invisible in the output. Fourth, permissions that follow the person asking, not a single service account with access to everything — otherwise the first sensitive document anyone adds turns the whole thing into a leak waiting to happen.
Fifth, a standard connection interface. The Model Context Protocol exists precisely so the knowledge base can expose one read interface that any compatible assistant uses, instead of a bespoke integration per tool that gets rebuilt every time the team changes assistants. These five are not a feature wish list; each one maps to a specific way the arrangement fails without it.
Freshness Matters More Once Agents Read It
Stale documentation has always been a problem, but the failure was self-limiting. A human reading a page from two years ago usually notices: the screenshots look wrong, the named owner left, the tone of the process no longer matches how things are done. Suspicion kicks in and they go ask someone.
An agent has no such instinct. It will quote a superseded policy with exactly the same confidence as a current one, and the answer it produces reads perfectly. The cost of staleness therefore rises sharply the moment agents start reading, which is worth knowing before rather than after.
The mitigation is cheap if it is built in: show when a document was last updated and who owns it, surface that alongside the answer, and treat unanswered questions as a backlog rather than an error. A lookup that returns nothing is a timestamped signal that a specific gap exists and someone needed it filled — the most useful prompt for writing a missing page you will ever get.
How To Evaluate A Tool In One Afternoon
Collect twenty questions your team has actually asked in the last month. Take them from chat history rather than inventing them; real questions are messier and more specific than the ones you would write, which is the point. Load three to five documents that should contain the answers.
Run all twenty and score each answer three ways: correct, cited to the right document, and honest when the answer is not present. Deliberately include four or five questions whose answers you have not documented — the handling of those tells you more than the handling of the ones you have.
A tool that answers fifteen of twenty correctly with citations, and declines the other five cleanly, is working. A tool that answers all twenty, including the five with no source, has told you the most important thing about itself in under an hour. This test costs an afternoon and settles a decision that otherwise takes weeks of demos.
Frequently asked questions
What makes a knowledge base suitable for AI agents?
Live retrieval instead of stale exports, citations on every answer, an honest response when nothing is found, permissions that follow the person asking, and a standard read interface such as MCP so any assistant can connect without a custom build.
Do we need a separate knowledge base for agents?
No. The same documents should serve both, because two sources diverge immediately. What changes is the retrieval layer on top: agents need the right passage returned first, where people can browse and recover from an imperfect result themselves.
How many documents before this is worth doing?
Three to five covering your most repeated questions are enough to test the idea. Coverage matters far less than hit rate, and writing comprehensively up front usually produces volume without improving the answers people actually receive.
What happens when the answer is not documented?
The knowledge base should say so plainly rather than improvise. A clean not-found is useful: it records that a specific question was asked and went unanswered, which is the best possible prompt for writing the page that was missing.
Top comments (0)