When I started my career in technical writing, the requirements and expectations were simple: learn the product, talk to some SMEs or engineers, get the job done by completing the documentation work, and work on the collected feedback.
But with AI coming in, it doesn’t feel the way it used to. The role is shifting dramatically, and so are expectations. But why?
If you maintain a dashboard for your documentation insights, you must have noticed that AI agents now read your docs more than real humans do. This helps us understand the shift in the role. AI has changed the audience for our documentation. Real people still read docs, but agents read them more, mix them with chat history, and answer in your product’s voice. So “is this page clear?” is only half the review and feedback you get.
Let’s understand this better.
Last month, I noticed something in my bot’s chat history. The Docs AI answered all questions for a user’s query, but some of the links it shared back were wrong. The links did not belong to the context the user asked about. What I understood after my initial investigation was that the AI got stuck in the wrong documentation context, and the carried-over long chat history made it share the wrong link.
If a Docs bot answers incorrectly from your documentation, someone has to own the answer it responded with. That someone is the technical writer now. This is where the role is expanding now, and I call it docs-as-evals.
What docs-as-evals means for technical writers
If I explain this in simple language, it is the process of treating documentation quality as something you can test by asking real user questions against your docs, then testing whether the AI answer is correct or not.
With this process, we are trying to evaluate how the bot behaves in a certain way; for instance, we want to know:
- Did it stay on the right product?
- Are citations correct?
- Did it keep conversation history as context when the user followed up
- Did it drop the history when the user switched products?
These are important parameters to know, and technical writers are good at evaluating these things. We understand the mess and have data to check: which words users actually typed, which page is the real source of truth, etc. As tech writers, we always used to map these things in our heads and in the style guide. But now it needs to be used in evaluating AI answers.
Problems with RAG in developer documentation
Most of the product tech writers document are multi-product systems. The product that I document is also multi-product, and I have noticed that on a multi-product developer site like mine, RAG does not usually fail with a blank answer. It fails with an answer that reads correct but makes no sense.
One of the core problems that my docs assistant had was that it used to send a large conversation window on every turn. Great for follow-ups. Bad for product switches in multi-product documentation.
Let me make you understand with an example from my docs bot.
In my docs bot chat history, I saw a user ask questions about Ambient APIs, followed by how to create a session, then check headers, and status. Everything was fine until they asked something like, “How do I quick-start with Mobile SDK?”
For a human, that is a clear example of a new topic shift, but for the docs bot, it was still one long chat. So the old product messages stayed in the API request it sent, and the answer started mixing things up and producing wrong links.
In one real test I ran, that product-switch question was still carrying about nine earlier turns. After we changed how context was handled, the same question only carried about two turns, and it cleared the old thread. Same question. Better result that I will share with you at the end of this blog.
A few other problems around this that I noticed were:
Users do not ask with perfect product names. They say things like “the SDK” or “how do I authenticate?”
It's common in any product that different products share the same words, so a RAG-based chatbot usually gets confused. Handling this is a challenge because the current page, the chat history, and the new question can all pull the answer in different directions and scenarios.
So the issue was not only “write clearer docs.” The issue was also: when should the bot remember the old chat, and when should it forget it?
That is where docs-as-evals helped me.
How I started testing this
I made a simple list of real user questions from the bot history that I collected from my docs dashboard. For each question, I wrote what “correct” meant for me as a tech writer. My idea was to evaluate the following on the following parameters:
- Which product should this answer belong to when the bot answers?
- Is the new question a follow-up or a new topic?
- Should I keep history, or should I drop it?
Think of this as a math set ok.
const selectionSet = [
{
prompt: "How do I create a session with Product A?",
expectProduct: "product_a",
keepHistory: false,
},
{
prompt: "How do I quickstart with Product B?",
expectProduct: "product_b",
expectMode: "topic_shift",
keepHistory: false, // drop Product A chat
},
{
prompt: "Tell me more about the authentication step",
expectProduct: "product_b",
expectMode: "follow_up",
keepHistory: true, // stay on Product B
},
{
prompt: "How do I quickstart with Product C?",
expectProduct: "product_c",
expectMode: "topic_shift",
keepHistory: false,
},
];
And this list became the golden set for me, which I used later to solve the problem.
What I checked in each answer
I kept the checks simple and evaluated the answers based on the following:
- Did the docs bot stay on the right product for a topic shift? Were the right links shared?
- For follow-ups, was the history kept or deleted?
If the user asked about another product and the bot linked other pages, I marked it as a fail. Even if the answer it gave was correct.
In code, it looked like this:
function isWrongCite(expectedProduct, citedPaths) {
if (!expectedProduct || !citedPaths.length) return null;
const matching = citedPaths.filter((path) =>
path.includes(`/${expectedProduct}/`)
);
// Fail if no links match, or most links are from the wrong product
if (matching.length === 0) return true;
return matching.length < citedPaths.length / 2;
}
// Example
isWrongCite("product_b", [
"/product_a/sessions/create",
"/product_a/authentication",
]);
// → true (wrong product links)
When I compared before and after on those key turns, the difference was clear. Product switches started landing on the right docs. Follow-ups stayed on the same product. The bot stopped dragging the old conversation into every new question.
When I compared before and after on those key turns, the difference was clear. Product switches started landing on the right docs. Follow-ups stayed on the same product. The bot stopped dragging the old conversation into every new question.
In those switch turns, the request size also dropped a lot in our runs, as you can see from the image below, from around 11k–17k characters to around 5k. This was because we stopped sending the wrong chat history with the questions.
What this means for the technical writer role
Technical writers never used to evaluate RAG-based systems. But now we do, and this is why I say the role is expanding. This is understandable: RAG is new, and most doc sites never used it. We still do the old work: learn the product, talk to engineers, write the docs, handle feedback.
But now, there is a new question we have to own and answer:
If the docs bot answers from our documentation, is that answer actually right?
That is docs-as-evals for RAG systems in documentation. In my day-to-day work, it looks like this:
I write docs, collect real questions from bot chats and support, then decide what correct looks like for those questions. I test them and use the failed ones to decide what to fix next, and this is something totally new to me. It is just documentation work for a world where AI reads our docs too.
Conclusion
RAG chatbots made it easy to get answers from long documentation, but they also created a new problem for technical writers to solve. Earlier, we mostly focused on information architecture to improve how a reader finds the right page. With RAG systems, that is not enough. We also have to check whether the AI finds the right answer, and that answer must be correct. I personally love this role change because it makes us more valuable in an uncertain world.
Top comments (1)
Carrying about nine earlier turns into a topic-switch question and getting it down to two is the concrete version of something most docs-RAG setups never measure. History gets treated as free context when it's really a bias — and on a multi-product site that bias pulls toward whichever product spoke first.
The golden set is the part worth stealing:
{prompt, expectProduct, expectMode, keepHistory}as a reviewable artefact instead of a vibe check. Marking a wrong-citation answer a fail even when the prose was right is the discipline most eval suites skip, because link correctness is assertable and "helpful enough" isn't. Two questions: how often does that set get refreshed from real bot traffic, and who decides when a question is ambiguous enough that any answer counts as a failure? And do you track the inverse case — a follow-up that reads like a product switch, where dropping history is the wrong call?