Your AI chatbot isn't hallucinating. It's reading your outdated docs.

sopkits — Sun, 04 Oct 2026 03:47:00 +0000

When a support chatbot gives a wrong answer, the first instinct is to blame the model. But when I looked at how these failures are described in support and operations communities, a different pattern kept showing up: the model was faithfully repeating the documents it was given. The documents were the problem.

Typical examples:

Two FAQ pages with different refund windows (7 days vs 14 days)
Two different support email addresses, both "official"
A policy last reviewed two years ago
An answer that literally says "TBD" - and still gets retrieved

A chatbot can't resolve a contradiction. It picks whichever page it finds first, so the answer becomes a coin flip. New employees fail in exactly the same way.

A small, free Claude skill for this

I built a Claude skill called Knowledge Freshness Audit. You attach your SOPs, policies, FAQs, or help articles, and it returns a prioritised fix list:

Contradictions - the same topic with different facts, quoted side by side with file names. Listed first, because they're the most dangerous.
Outdated or undated documents
Duplicates - with a recommendation on which one to keep
No owner / unfinished documents (drafts, TBD answers, empty sections)
Gaps - obvious questions no document answers

Two design decisions

It never decides which fact is correct. If one page says 7 days and another says 14, the skill flags it and asks. Only the business knows its current rule; an AI guessing here would just create a third version of the truth.

It keeps personal data out of the report. It refers to files and lines instead of copying customer names, phone numbers, or account details.

Test run

I tested it on four small documents with planted problems: two contradictions (refund window and support email), three undated files, one duplicate, three files with no owner, one "TBD" answer, and one unanswered question. The skill found all of them. The test documents and the full report are in the repo's example/ folder, so you can check the output yourself.

Try it

It's free and MIT-licensed:
https://github.com/sopkits/knowledge-freshness-audit

Install the .skill file in Claude (requires a plan that supports Skills), attach your docs, and say: "Audit these help docs - our chatbot keeps giving wrong answers."

I'd love feedback - especially on problems it misses.

This article was written with AI assistance and reviewed by me before publishing.

DEV Community: sopkits

Your AI chatbot isn't hallucinating. It's reading your outdated docs.

A small, free Claude skill for this

Two design decisions

Test run

Try it