When an engineering team changes a search index, they run a regression suite. When most teams update an AI agent's knowledge base, they paste the new content in and hope.
That asymmetry is the single biggest quality risk in agent deployments, and it is easy to fix.
Why a knowledge update is a deploy, not an edit
A knowledge base change does three things at once:
- Adds new answers, which may now out-compete existing ones
- Removes or replaces old answers, which other answers may have depended on
- Shifts retrieval — adding a long document about refunds can change which document gets selected for a question about cancellations
The third effect is the one nobody expects. You add a document to fix one question, and a different question starts answering from the new document. Nothing "broke" — the corpus shifted underneath.
The three ways an update goes wrong
1. The new answer wins by accident. A new policy document is longer and covers more keywords, so it now answers questions that used to be answered by a more specific document.
2. An old answer disappears. You replaced a document rather than adding one, and it contained two facts — one you meant to change, one you did not.
3. A contradiction is introduced. The new document says one thing, an older one says another, and neither is marked as authoritative.
All three are invisible to the person who made the change, because they knew what they intended. They are obvious to a test.
The minimum regression check
You do not need a test framework. You need a spreadsheet and twenty minutes.
Step 1 — build the question set. Take the twenty questions you are actually asked. For each, write the expected answer in your own words, and note which document it should come from.
Step 2 — capture the answers before the change. Ask all twenty, record the responses verbatim. This is your baseline. Do it before your first change, or you will never have one.
Step 3 — make the change.
Step 4 — ask the same twenty, in the same order, and diff.
Three columns are enough: question, before, after. You are looking for two things:
- questions that were right and are now wrong
- questions that now hedge where they used to answer, or vice versa
Step 5 — add the new question. Whatever prompted the update, add it to the set permanently. The set should grow, never shrink.
What "proven" actually means
A vendor who cannot show you a before/after on a fixed question set is not verifying anything. This is a reasonable thing to ask during evaluation, and the answer is informative:
- "We have a comparison view" — good, ask to see it used on a real change.
- "We test with a set of questions internally" — acceptable, ask what happens when an answer regresses.
- "The model handles it" — this means nothing is being checked.
The same applies internally: if your team cannot produce a before/after, the update is unverified by definition, regardless of how careful the editor was.
The part that is not about testing
A test catches regressions. It does not prevent them. Two habits do most of the preventing:
Retire, do not duplicate. When a policy changes, remove the old document rather than leaving both. A corpus with two versions of the truth will answer inconsistently and no amount of testing will make it stable.
Change one thing at a time. Updating prices, hours, and a refund policy in the same edit means a regression tells you nothing about which change caused it.
If it does go wrong
The other half of safety is recovery. Before any substantive change, confirm that you can get back to the previous state — and that "back" includes the answers, not just the documents.
If a platform offers version history, check what a rollback actually restores. Rolling back a document is not the same as rolling back an agent's behaviour if the agent configuration also changed.
Practically: make content changes and configuration changes on different days. When something breaks, you want only one variable in play.
A note on cadence
Most teams update the knowledge base reactively, after a customer complains. That guarantees every update is made under time pressure, by whoever is available, with no baseline.
A better rhythm, for a small business:
- Weekly: prices, hours, anything that changes often
- Monthly: policies, FAQs, service descriptions
- Immediately: anything a customer got wrong this week — with the question added to the test set
The cadence matters more than the tooling. An agent with a maintained corpus and no test suite beats a perfect test suite on a stale corpus.
Disclosure: I work on RingBot, an Agent-as-a-Service platform where knowledge can be written or uploaded, bound to an agent, and versioned so an update can be rolled back. The regression discipline above is what I would apply to any agent platform, including ones with better tooling than ours.
Related: why agents give wrong answers, how to run the update, flow regressions, and the cost comparison of the four options.
Top comments (0)