When a coding agent gets something wrong, you find out fast. The build fails, a test goes red, the page renders blank, or something else falls over loudly enough that you can't miss it. When a research agent gets something wrong, you get a beautifully formatted brief filled with wrong facts.
And documentation work runs on research.
The one sentence I gave this in February
Back in February I wrote about a command I'd built to do the research phase of a documentation ticket. You handed it a ticket ID and it pulled the ticket with every comment on it, chased the linked engineering issues, went hunting through the internal wiki for a PRD or a spec, read the pages that were already published, and gave back a structured brief with open questions at the bottom.
I was pleased with it. It saved hours of clicking between tools, and it surfaced things I'd have missed, because our internal wiki had a terrible search engine.
Rereading that post, the entire section on what could go wrong is one sentence: you still need to review the information and check with stakeholders to make sure the draft is accurate. That's true, and nowhere near enough. If you're building something like this, that sentence is where most of your risk lives.
The brief is the most dangerous thing in your pipeline
Everything downstream is built on it. You read the brief, you write the draft from it, and the draft goes to an engineer who reads it as your understanding of their feature. They're reviewing your prose, not the retrieval that produced it.
So an error in the brief doesn't sit still waiting to be found. It gets written into a page and approved by someone who reasonably assumes you did the reading. Somewhere in that chain it stops being the tool's mistake and becomes your judgement.
Reading it carefully won't save you
Prose has no failing state. A citation to a PRD that doesn't exist is formatted exactly like a citation to one that does. A stakeholder attributed to the wrong team reads perfectly well. Nothing goes red.
And you're reading the brief because you don't know the material yet, which was the whole reason you asked for it. So you're checking a summary before you've read the sources behind it, about a system you're still forming a picture of, at the exact moment you're least equipped to notice something missing.
I don't know about you, but I tend to trust a tidy document more than a messy one. Structure is what these tools are genuinely good at. Accuracy is what they're worst at. Which means the instinct you'd normally use to catch sloppy work is aimed at the wrong signal.
Put the skeptic in front of the reader
If you already end agent work with a skeptical pass, handing the output to a second agent whose only job is to attack it, you're probably running it last. For research, that's too late. Set it up so the research agent spawns a fresh adversarial one as soon as the brief exists, and have it fact-check the brief before anything reaches you.
Keep the pass narrow. Not "is this any good", but: does each claim trace back to something actually retrieved, does anything here contradict anything else here, is the information up to date or based on a dusty page abandoned years ago, and are the gaps marked as gaps?
That last one carries most of the weight. Ask an agent for a research brief and you will get a research brief. It won't hand you three quarters of one and tell you the rest isn't there. The failure worth guarding against is the smooth, complete-looking answer where the honest output was "the wiki has nothing on this, go ask a person."
What it costs
It's slower, which might seem absurd for a workflow built to go faster. That's still far quicker than finding those forty wiki pages yourself, and you get something back for the wait.
It won't make the brief true. What a fresh pair of eyes catches is facts that trace nowhere, claims resting on thin evidence, and gaps that got smoothed over. Those are the ones I kept running into.
What you write still goes to someone who knows the system better than you do, and that review isn't optional. The skeptical pass just means you arrive at that conversation believing fewer things for no reason.
Top comments (0)