Here's an uncomfortable truth about documentation in most engineering orgs: code gets reviewed, docs don't. A pull request with code changes gets two approvals and a CI run. A pull request with docs changes gets a "LGTM" from someone who scrolled through it in eight seconds — if it gets reviewed at all.
I was the sole owner of documentation for a set of AI platforms at a large tech company in Eastern Europe. Sole owner means exactly what it sounds like: there was no second pair of eyes. Every broken code example, every outdated parameter, every style inconsistency was going to ship unless I caught it myself. At 30–40 docs merge requests a week — spiking toward 50 in release weeks — from a hundred engineers across three departments, review alone was eating half my day. I was the bottleneck and the single point of failure at the same time.
So I did what any engineer would do with a bottleneck: I automated it. I built an LLM reviewer into the docs pipeline. Every merge request now gets an automatic review before a human ever looks at it.
What it actually catches
Not what I expected, honestly. I assumed the main value would be style — and it does enforce style. But the real catches, ranked by how much pain they prevent:
Broken and outdated examples. The classic failure mode of platform docs: the API changed three releases ago, the example didn't. The reviewer cross-checks examples against the current system state and flags mismatches. This alone would justify the whole setup — a broken quickstart example is the single fastest way to lose a developer.
Contradictions between pages. Page A says the limit is 100 requests per minute, page B says 60. No human reviewer catches this, because no human reviewer holds the whole corpus in their head. The LLM does.
Silent scope creep. A writer (or a developer editing docs) quietly documents behavior that doesn't exist yet, or removes a warning that still applies. The reviewer flags claims that changed without a corresponding release reference.
Style violations — last on the list, and that's the point. Style is what people think docs review is about. It's the least valuable part.
And the meta-effect that mattered most: merge requests stopped queuing on me. Before, every docs change sat in the bottleneck of a single human reviewer — my calendar was the pipeline's latency. Now the machine does the first pass in minutes, authors fix the mechanical findings themselves, and what reaches me is already clean. Time-in-review dropped from a 2–3 day queue to same-day, and I stopped being the reason a finished doc wasn't live.
What it doesn't do
It doesn't write the docs. It doesn't know what the reader needs, what to explain and what to leave out, how the information should be structured. It reviews — against the spec, against the corpus, against the style guide. The judgment stays human.
My favorite false positive makes the point better than any argument. We were documenting a custom no-code builder for feature extractors — a genuinely bespoke piece of our platform, built in-house, existing nowhere else. The LLM reviewer flagged our description as "inaccurate" and helpfully suggested a correction… describing how feature extraction works in the generic ML tooling it had seen in its training data. The feature was too new and too ours to exist in the model's world — so the model concluded the docs were wrong, not that the world had something it hadn't seen. It reviewed our product against its memory of other products, with total confidence.
That's the boundary in one anecdote: an LLM reviewer is superb at checking your docs against things that are written down — the spec, the corpus, the style guide. For anything genuinely novel, the human who built it is the only source of truth, and the AI-reviewer's confident "correction" is exactly the hallucination you built the pipeline to catch. Which is why the reviewer proposes and a human disposes — never the other way around.
The part nobody tells you
The hard part wasn't the LLM. The hard part was that an LLM reviewer is only useful if your docs live in a pipeline at all: docs-as-code, version control, CI, a reference generated from a spec instead of written by hand. If your documentation is a pile of pages in a wiki, there is nothing to hook a reviewer into. The AI is the last 10%. The system is the other 90%.
Which is, in my experience, the actual state of documentation at most companies: not "we need AI," but "we need the system that AI could then plug into."
I build these systems for API-first companies — docs-as-code pipelines, references generated from OpenAPI, LLM review on every change. If your docs are a pile of pages in a wiki and your integrations are slower than they should be, that's exactly what my documentation audit finds and fixes. DM me or find me on LinkedIn.
Top comments (0)