DEV Community

Cover image for I built CI for the facts in your docs (so your AI assistant stops quoting old pages)
YASH THOKAL
YASH THOKAL

Posted on

I built CI for the facts in your docs (so your AI assistant stops quoting old pages)

#ai #documentation #opensource #devops

Your AI docs assistant can give a wrong answer and cite a real page while doing it.

Not because the AI is bad. Because one of your pages is out of date, and the assistant has no way to know.

This post explains how that happens, why the usual docs checks miss it, and the open-source tool I built to catch it: TruthLayer.

How a docs assistant gives a wrong answer

Most "chat with your docs" assistants work like this:

  1. Split every docs page into chunks.
  2. When a question comes in, find the chunks that best match it.
  3. Send those chunks to an AI and ask it to answer, with citations.

That works well, as long as every page is correct.

But the same fact usually lives in many places. Say your Pro plan costs $29 per month. That price is on:

  • the pricing page
  • the FAQ
  • an "Upgrade to Pro" guide
  • the changelog

Then you raise the price to $39. You update the pricing page and the changelog. The FAQ and the guide still say $29.

A customer asks "How much is Pro?" The guide happens to match the question best, so the assistant answers:

The Pro plan costs $29 per month. [1] guides/upgrade-to-pro.md

It has a citation. It looks right. The customer signs up, gets charged $39, and opens a support ticket. And nobody on your team knows the guide is wrong.

Why spell checkers and link checkers don't catch it

A typical docs pipeline checks:

  • spelling and style (Vale, cspell)
  • broken links
  • that the site builds

The stale guide passes all of them. The words are fine. The links work. The fact is wrong, and nothing checks facts.

It happens to real companies

While testing this idea on public docs, I found two pages in Mintlify's docs that give different Node.js requirements for their CLI: one says Node.js v20.17.0+, another says Node.js 22.20 or later for mint agent setup (reported as mintlify/docs#8358).

If it happens in the docs of a company that builds documentation tools, it happens everywhere.

TruthLayer: CI for the facts in your docs

TruthLayer treats your OpenAPI spec and your changelog as the source of truth, and checks your Markdown docs against them.

npx truthlayer init     # creates truthlayer.yml
npx truthlayer check    # finds outdated and conflicting facts
Enter fullscreen mode Exit fullscreen mode

On the example project in the repo, the output looks like this:

TruthLayer · Acme Assistant
Scanned 6 pages, 1 OpenAPI spec, 1 changelog · 23 facts

  ✖ 4 outdated   ⚠ 1 conflict   ⚠ 1 endpoint warning   ℹ 0 old pages

✖ OUTDATED  API rate limit (per minute)
  1 page says something different from the source of truth.
    ✓ source of truth openapi.yaml:10            x-rate-limit: 200 per minute
    ✓ source of truth CHANGELOG.md:7             200 requests per minute
    ✗ outdated        docs/api/rate-limits.md:3  100 requests per minute
  → Changelog 2.0.0 changed this from 100 to 200. The page still shows the old value.

⚠ CONFLICT  Minimum node version
  2 pages give different values. Your assistant may answer either one.
    • says            docs/faq.md:5              Node.js 20 or later (newest)
    • says            docs/getting-started.md:7  Node.js 18 or later
Enter fullscreen mode Exit fullscreen mode

Each finding tells you the page, the line, what it says, and what the source of truth says.

What it checks

Fact Example sentence
Prices "The Pro plan costs $39 per month."
Plan limits "The Free plan includes up to 5 projects."
Rate limits "Each API key can make 100 requests per minute."
Storage and file sizes "The maximum file size is 25 MB."
Pagination "up to 50 items per page" vs. the spec's maximum
Timeouts and retention "Logs are kept for 30 days."
Required versions "Requires Node.js 18 or later."
Latest version "The latest version is 2.0." vs. the newest changelog entry
Endpoints a page mentions an endpoint that was removed, is deprecated, or isn't in the spec

How it avoids false alarms

The obvious approach would be to ask an AI "do these two pages contradict each other?". Research shows that gives too many false alarms, and a CI check that cries wolf gets turned off.

So TruthLayer finds facts with patterns, not AI, and only compares facts that mean the same thing:

  • same kind (a price is only compared with a price)
  • same plan ("Free: 5 projects" is never compared with "Pro: 50 projects")
  • same endpoint
  • same docs version (docs/v1/… is only compared with docs/v1/…)

Code blocks are ignored, values inside one page (like a pricing table) are not compared with each other, and you can silence a sentence that's correct on purpose with <!-- truthlayer-ignore -->.

It runs on every pull request

Add one workflow (or run npx truthlayer init --github-action):

name: TruthLayer
on:
  pull_request:
permissions:
  contents: read
  pull-requests: write
jobs:
  knowledge-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: YashThokal777/truthlayer@main
        with:
          strict: 'false'   # 'true' fails the PR when errors are found
Enter fullscreen mode Exit fullscreen mode

On each PR it only reports findings that involve files changed in that PR, and posts (or updates) one comment. In the demo PR, someone adds an "Upgrade to Pro" guide with the old price, and TruthLayer comments:

Fact Source of truth New page
Price · Pro plan CHANGELOG.md:9 ✓ $39 per month upgrade-to-pro.md:7 ✗ $29
Free plan projects CHANGELOG.md:8 ✓ 5 projects upgrade-to-pro.md:8 ✗ 10
Rate limit openapi.yaml:10 ✓ 200 / min upgrade-to-pro.md:13 ✗ 100

By default it never blocks a merge. You can turn on strict once you trust it.

A chat assistant that knows when a page might be wrong

TruthLayer also ships a local chat playground and an embeddable widget:

npx truthlayer dev   # http://127.0.0.1:4141
Enter fullscreen mode Exit fullscreen mode

Every answer cites its sources and shows when each source was last updated. If a cited page has a known problem, the answer shows a warning, so the assistant answers "$39 per month" and tells you the guide may be outdated.

The chat widget answering with sources and flagging an outdated pricing page

It works with a free Google Gemini key, Groq, any OpenAI-compatible API, or fully offline with Ollama. With no key at all, it runs in search-only mode and shows the matching parts of your docs.

Honest limits

  • Facts are found with patterns, so sentences written in unusual ways can be missed. npx truthlayer facts shows everything it found.
  • It's best for SaaS and API product docs (limits, prices, plans, endpoints). Reference docs without such facts will show few findings.
  • Search uses keywords (BM25), not embeddings, so it works offline with no downloads, but questions phrased very differently from the docs may find weaker sources.
  • It reads Markdown source files; crawling a docs website isn't supported yet.

Try it

It's free, MIT-licensed, local-first, and needs Node.js 20.12+. No database, no server to rent.

npx truthlayer check
Enter fullscreen mode Exit fullscreen mode

⭐ GitHub: https://github.com/YashThokal777/truthlayer
📦 npm: https://www.npmjs.com/package/truthlayer

I'd really like feedback, especially a sentence from your own docs that TruthLayer misses or flags wrongly. Open an issue or drop it in the comments.

Top comments (0)