I built an eval tool for docs, then I ran it on PostHog's own docs

Every LLM eval tool I could find measures the output. Was the model's answer correct? Was it hallucinated? Did it call the right tool? All important questions, all measuring the wrong end of the pipeline.

The other end is the input. If your RAG system pulls a chunk from bad docs, it produces bad answers, and nothing you do in the pipeline fixes that. The fix has to happen upstream, in the docs themselves. And to fix docs at that layer, you first have to measure them.

That's what I built. It's called docfit and it scores documentation across six dimensions.

The two-audience problem

Docs used to have one audience: humans, reading top to bottom. Now they have two. Humans (still reading top to bottom) and AI agents (grabbing individual chunks to answer a specific question). The two audiences want subtly different things.

A doc that reads well for humans might use "this" and "that" liberally, with clear antecedents from earlier paragraphs. An LLM retrieving one chunk in isolation doesn't have those antecedents. The doc breaks silently.

A doc might describe at a high level what a feature IS. Humans reading top to bottom can piece it together from surrounding context. An agent retrieving that chunk to answer "how do I install X?" has nothing concrete to work with. It either invents details or refuses.

I wanted a tool that surfaces these gaps without me reading every page. Nothing did what I wanted, so I built one.

What docfit does

Six evaluators. Each one asks a specific question about a doc, produces a 0-100 score, and (optionally) generates concrete rewrites.

Chunkability looks at header structure, paragraph shape, sentence boundaries. Can the doc even be split into sensible pieces? Fully deterministic, no LLM needed.

Retrievability generates realistic questions for each chunk, embeds them locally, measures where the source chunk ranks in a semantic search. Direct measure of RAG quality.

Answerability asks the LLM: "given only this chunk, can you answer questions about it without inventing missing context?" Predicts hallucination risk.

Code quality parses code blocks. Are they tagged with a language? Syntactically valid? Complete enough to run standalone? Auto-recognises install commands and doesn't false-positive on them.

Ambiguity flags dangling pronouns, undefined jargon, unstated context. This is where the rewrite suggestions shine.

Task orientation judges whether the doc is structured around what users want to DO versus what things ARE. Task-oriented docs match user intent better.

The PostHog run

Here's what a run looks like on PostHog's product-analytics page:

The overall 84.3 is fine. The interesting thing is the shape. Answerability at 62.5 says "this page reads at a high level but doesn't fully answer specific questions from a single chunk." Ambiguity at 80 caught 11 issues, mostly jargon: funnels, stickiness, Self-driving used without inline definitions.

The bit I'm most proud of: run it with --suggest and it doesn't just complain. It hands you the fix:

Before: Build trends, funnels, retention, paths, stickiness, and lifecycle insights on the events you already send to PostHog.

After: Build trends, funnels (a series of steps users take to complete a task), retention (the ability to keep users engaged over time), paths, stickiness (how often users return), and lifecycle insights on the events you already send to PostHog.

That rewrite was generated by the tool. You could paste it directly into the docs and merge. That's the difference between an eval tool that tells you what's wrong and one that helps you fix it.

What I found across nine PostHog pages

I audited nine pages while building the tool. Some patterns came out:

• Answerability was the weakest dimension almost everywhere (median 50). Pages describe features at a high level but often lack the concrete step-by-step that lets a single chunk answer a specific question.
• The install guide scored highest overall (83.4). Task-oriented, well-structured, concrete. Installation content wins because it's inherently doing-shaped rather than being-shaped.
• The landing page scored 62. Correct behaviour, not a bug. Landing pages are navigation grids, not narrative content, and the tool rightly reflects that.
• The data-warehouse page got 15 on chunkability. Almost no header structure, which hurts both human skimming and agent retrieval.

None of these findings would surprise a docs team. What matters is that a tool can surface them systematically, at scale, without a human reading every page.

The unfair advantage of a SDET brain

I've spent six years as a SDET engineer. That background changes how I approach LLM systems.

Most ML tooling treats evaluation as an afterthought. QA people know that if you don't measure, you can't improve. Writing an LLM eval tool with a SDET hat on means you obsess over the failure modes. What happens when the LLM returns malformed JSON? (It happens. Handle it.) What happens when the daily token budget runs out mid-run? (Saw it live during testing. Now the tool bails cleanly instead of hammering the API.) What happens on the empty chunk that trafilatura sometimes produces? (Filter it before it drags every score down.)

Every one of those was a bug I hit while dogfooding on real docs. The 106 tests grew organically.

Track it like any other product metric

Since I was running docfit against PostHog's docs anyway, I wired up a PostHog integration:

docfit report ./docs.md --posthog-key phc_your_key

Each run sends a docfit_run event with all the scores as properties. Point at a PostHog project and you get a dashboard.

The idea: wire docfit into your CI. Every commit to your docs repo triggers a run. PostHog shows the trend. If a PR drops the average score, you know. If a specific page is consistently your worst, you can prioritise fixing it.

What docfit doesn't do yet

Being upfront about the sharp edges:

• English only. All evaluator prompts are English.
• Batch runs sequentially. A 20-page audit takes a while.
• Only markdown files supported for directory input. HTML, reStructuredText, and Jupyter notebooks are on the roadmap.
• The --suggest mode roughly doubles token usage. That's the tradeoff for actionable output.
• Small local models sometimes return malformed JSON on judge prompts. The tool handles it gracefully but you'll see lower coverage than with a bigger model.

None of these are showstoppers. They're the honest limitations of a v0.1.

Try it

pip install "git+https://github.com/bewanderer/docfit.git#egg=docfit[embeddings]"

Grab a free Groq API key at console.groq.com, export it, and:

docfit report https://your-docs.example.com

Full docs, install guide, and CLI reference at github.com/bewanderer/docfit.

If you're on a docs team and you've thought about the agent-readability problem, I'd love to hear what you're doing about it. Or run docfit against your own docs and tell me what surprised you. Issues and PRs welcome on the repo.