Writing Docs for AI

LLMs are trained on data that could be months or even years old. If you shipped a feature today or a few weeks ago, the model won’t know about it. (Of course, these days, LLMs like Claude can go on the web and do a search).

But let’s say your internal docs assistant doesn’t have a web search function.If you ask it about your new feature, rather than say it doesn’t know, it will string together words that are likely to answer the question because that’s just how LLMs work. They predict the next most likely word.

RAG (Retrieval Augmented Generation) is one of the methods used to solve this problem by giving the LLM access to information it hadn’t seen in training (your product docs in this case).

How RAG works

In a very simplified way, when you ask the model a question, rather than answering you immediately, the RAG system first gives it the relevant information it needs to generate a response.

It works in the following steps:

1. **Data chunking: **You chop your massive 100-page product manual into digestible pieces of related topics (e.g., 300-word paragraphs).

This matters because LLMs have a limited context window (the amount of text it can see at a given time) and you want to make sure it’s seeing only the relevant information otherwise you’re just wasting the context space.

1. **Document embedding: **An embedding model then reads each chunk of text and outputs a long list of numbers such as [0.12, -0.43, 0.89, ...]. This list of numbers is called a vector, and it has the meaning of the text baked in.

Vectors of similar concepts like forgot credentials and reset password will point to coordinates close to each other in a multi-dimensional mathematical space.

3*. Document retrieval:* When you ask the LLM a question, the RAG system converts it into a vector using the same model it used on the chunks then uses math to find the chunks vectors closest to it.

It grabs the top matching vectors and returns the corresponding text.

1. **Generating the response: **The system passes the original question plus the retrieved text chunk to the LLM in a single prompt. The LLM reads the prompt and generates an answer the retrieved information.

How you’ve structured your docs will affect how RAG will split it up and the chunks it will retrieve during query time.

How to structure your product docs for RAG

Sections are explicit and self-contained.

A self-contained section provides all the info the reader (or the AI) needs to understand a concept.

So, treat every single subsection as if it could be read entirely by itself. If you introduce a workflow step in paragraph also add the required prerequisites and the expected outcome in that same block of text.

Avoid relying on ambiguous demonstratives like “this,” “the above,” or “as mentioned above.” We can easily trace an explanation back three paragraphs to see what “this” refers to, but RAG can return a chunk completely out of context without previous paragraphs.

Use semantic HTML or markdown

Semantic HTML or markdown preserve the hierarchy of your document. You can define headings, subheadings, paragraphs, lists, etc which your RAG system can use to determine splits. For example, a ### makes it easier to keep an entire subsection together under its parent heading.

I have also seen writers recommend DITA XML as a better alternative to markdown and HTML. If you’re not familiar with DITA, it’s an XML-based standard where you write small, self-contained topics. Each topic has a strict, specific type:

• Concept - Explains an idea (”What is a refund policy”)
• Task - Step-by-step instructions (”How to request a refund”)
• Reference - Lookup material (tables, parameters, specs) DITA-based docs look something like this:

<task id="request-refund">
  <title>Requesting a Refund</title>
  <taskbody>
    <steps>
      <step><cmd>Log in to your account.</cmd></step>
      <step><cmd>Navigate to Order History.</cmd></step>
      <step><cmd>Click "Request Refund."</cmd></step>
    </steps>
  </taskbody>
</task>

These XML tags add so much richer context than a simple bold or italic tag in HTML or Markdown.

That said, its very maximalist compared to Markdown, and I’m hoping to dive deeper into it in a future post to find out if the authoring overhead is even worth it.

Avoid PDFs

PDF discards the structural information (headings, tables, reading order) that RAG pipelines depend on so it’s not really good for retrieval. If the PDF has a table, for instance, you might end up just extracting a word soup that doesn’t mean anything at all.

Complex animations

Any concept that’s critical to understanding your product needs a plain-text explanation somewhere nearby, even if you keep the animation for human readers.

If you only explain a complex feature through a looping GIF or an animated UI diagram, an LLM retrieving that chunk gets zero useful text to embed.

Use descriptive headings and meaningful URLs

It’s wasy to figure out what a section will talk about if it has a heading like “Configuring SSO for Enterprise Accounts”. In that way, it’s easy for the model to know that section could be relevant answering a certain question. However, headings like “Setup” or “Configuration” that have no context give a retrieval system almost nothing to match against.

The same logic applies to URLs. A/docs/page-3 doesn’t really tell you anything but /docs/sso-enterprise-setup is self-describing even out of context.

Text equivalent for visuals

Add an image description describing what the image is displaying. If a screenshot highlights where to click in your dashboard, that information is invisible to text-based retrieval unless you also write it out.

Avoid layouts that convey meanings and minimize tables

Prefer bullet points or simple key-value pairs over tables, and if you must use a table, keep it flat with one row per item.

Merged cells in particular confuse most table-to-text converters since you can’t really tell the relationship between a header and its data cells.

Avoid scattered context, need to keep concepts in proximity

RAG chunking works best when everything you need to understand a piece of information lives close together in the document. Keep a concept and its caveats or related details in the same section, even if it means some repetition across pages.

If you’ve defined your API’s rate limit in one section and the exception for enterprise accounts is three pages later, a retrieval system may pull one chunk without the other and the LLM will answer confidently with only half the info.

Hopefully these guidelines help you the next time, you’re structuring your docs.

Thanks for reading and happy writing!