DEV Community: Virginia Nyambura Mwega

How to Cancel an OpenAI API Request with AbortSignal in Node.js

Virginia Nyambura Mwega — Wed, 28 Oct 2026 12:15:24 +0000

A timeout and a cancellation signal are related, but they are not exactly the same thing.

A timeout is a limit on how long an operation should wait. Cancellation is a way to tell an operation that its result is no longer needed.

For example, an interactive application may abandon a request because a user navigated away, cancelled a task, or no longer needs the generated response. A background process may also have a total deadline that should apply to the whole operation rather than just one network attempt.

The OpenAI JavaScript SDK supports passing an AbortSignal through the request options. Here is a practical example.

1. Create an abort controller

Node.js provides AbortController, which exposes a signal that can be used to cancel compatible asynchronous operations.

The OpenAI JavaScript SDK accepts the signal through the request options object.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const controller = new AbortController();

const deadline = setTimeout(() => {
  controller.abort();
}, 15_000);

try {
  const response = await client.responses.create(
    {
      model: "YOUR_MODEL_ID",
      input: "Explain why request cancellation matters in API clients.",
    },
    {
      signal: controller.signal,
    }
  );

  console.log(response.output_text);
} catch (error) {
  if (controller.signal.aborted) {
    console.error("Request cancelled or overall deadline reached.");
  } else {
    throw error;
  }
} finally {
  clearTimeout(deadline);
}

Replace YOUR_MODEL_ID with a model available to your project. The example also requires a valid API key in OPENAI_API_KEY.

The finally block clears the timer whether the request succeeds, fails, or is cancelled.

SDK reference: https://github.com/openai/openai-node/blob/main/docs/configuration.md

2. Understand what the deadline means

The example starts a 15-second timer that aborts the operation.

This is useful when an application needs a total elapsed-time limit. The SDK also offers its own request timeout setting.

Those settings should not be confused.

A configured SDK timeout governs an individual request attempt. Since the SDK can retry certain failures, the total elapsed time can exceed a single attempt's timeout.

An application-level AbortSignal can provide a cancellation boundary around the operation when the overall deadline matters.

Choose the approach according to your needs. You may want an SDK timeout, an external cancellation signal, or both.

3. Distinguish cancellation from other errors

The catch block checks whether the controller was aborted.

If it was, the application can report that the operation was cancelled or its deadline was reached. Otherwise, the error is propagated.

In a real application, you may want to distinguish user cancellation from deadline expiry by recording which event triggered the abort.

Avoid silently treating every exception as cancellation. Authentication failures, invalid requests, rate limits, and provider errors still need appropriate handling.

4. Use cancellation with streaming too

For a streamed request, pass the signal through the same request-options mechanism:

const controller = new AbortController();

const stream = await client.responses.create(
  {
    model: "YOUR_MODEL_ID",
    input: "Explain asynchronous iteration.",
    stream: true,
  },
  {
    signal: controller.signal,
  }
);

for await (const event of stream) {
  if (event.type === "response.output_text.delta") {
    process.stdout.write(event.delta);
  }
}

If the application needs a deadline or user-cancellation action, it can call controller.abort() while the stream is being consumed. The consuming code should handle the resulting cancellation or error path.

A web application should also connect cancellation to its own lifecycle—for example, stopping work when a result is no longer needed.

5. Common mistakes to avoid

Leaving timers running: Always clean up a manually created timer in a finally block.

Treating every exception as cancellation: Check the cancellation signal and preserve other errors for appropriate handling.

Assuming the timeout covers every retry automatically: Understand the difference between an attempt-level timeout and an overall deadline.

Forgetting the UI or caller: Cancelling the network request is only part of the feature. The application must also stop waiting, update its state, and prevent stale results from replacing a newer result.

Assuming cancellation proves that no work occurred: A request may already have reached the server before the client cancels it. Cancellation should not be treated as proof that no processing or cost occurred.

Conclusion

Cancellation is an important part of an API client's contract. It lets applications stop waiting for work whose result is no longer useful and gives them a practical way to enforce operation deadlines.

Use AbortSignal deliberately, clean up timers and application state, and test the cancellation path separately from ordinary success and failure behavior.

Reference: OpenAI Node.js SDK configuration

Your AI Evaluation Can Pass Every Test and Still Mislead You

Virginia Nyambura Mwega — Fri, 23 Oct 2026 16:27:54 +0000

A passing test suite can tell you that your evaluation code works.

It cannot, by itself, tell you that your AI model produces reliable answers.

That distinction has become important in my work on an OpenAI JavaScript SDK documentation practice project. I have been building an evaluation workflow that includes a rule-based baseline, pairwise comparisons, and an optional live-model judge runner.

The experience has made one thing clear:

Testing the evaluation pipeline is not the same as validating the model.

Here are four lessons I am taking from the process.

1. Separate pipeline correctness from answer quality

An evaluation pipeline has its own software behavior to verify.

You can test whether it processes inputs, applies comparison rules, produces results, and handles failures as expected.

Those tests establish useful evidence about the implementation.

But they do not prove that the underlying evaluation criteria are good measures of AI quality.

For example, a rule that checks whether a response contains a specific phrase might work perfectly in every test while still accepting an incorrect answer or rejecting a correct one written differently.

A passing test means the code met the condition being tested. It does not mean the condition captures every important aspect of answer quality.

2. Know what your evaluation actually measures

A rule-based baseline is useful when the expected behavior is narrow and predictable.

It becomes less informative when answers can be correct in several different ways, or when correctness depends on context.

Before trusting an evaluation result, ask:

What behavior is being measured?
What behavior is not being measured?
Could a bad response pass?
Could a good response fail?
Do the examples represent the cases users will encounter?

These questions help expose the difference between a convenient metric and meaningful evidence.

3. Label simulated and live results accurately

My project includes a pairwise evaluation workflow and an optional live-judge runner.

These serve different purposes.

A simulated comparison can help test whether the evaluation workflow processes comparisons as expected. It is useful for developing and checking the software without relying on live model calls.

However, simulated results do not establish how a real model judge will perform.

If the live evaluation has not been run and validated, the correct report is that live-model validation remains unverified—not that the system has passed a live benchmark.

This distinction makes technical documentation more trustworthy because readers can see what was actually tested.

4. Treat scores as evidence, not verdicts

Pairwise evaluation can identify which of two responses performs better under a given criterion.

But a relative winner is not automatically correct, safe, or production-ready.

The conclusion depends on the criterion, the examples, the comparison method, and the reliability of the judge.

A score without that context can create confidence that the underlying evidence does not justify.

A practical checklist for AI evaluation

When documenting an evaluation workflow, I now find it useful to separate these questions:

Have I tested that the evaluation software behaves correctly?
Have I explained what the evaluation criteria measure?
Have I identified whether results are rule-based, simulated, or generated by a live model?
Have I stated what the results do not prove?
Have I avoided presenting unverified work as completed validation?

This checklist does not make an evaluation perfect. It makes the claims around it more precise.

Final thought

AI evaluation is not just about getting a score. It is about understanding how that score was produced and what conclusions it can reasonably support.

A green test suite is useful evidence about software behavior. A benchmark can provide evidence about model behavior. We should not confuse one for the other.

I am continuing to document the implementation and its limitations in my OpenAI SDK Documentation Practice repository.

What checks do you use to decide whether an AI evaluation result is trustworthy?

How I Tested and Fixed Sensitive Exception Data in OpenTelemetry

Virginia Nyambura Mwega — Wed, 21 Oct 2026 15:24:31 +0000

I'm building an OpenAI Node.js SDK practice project to explore API integration, testing, documentation, and LLM observability.

One issue I investigated was how my observability wrapper recorded exceptions. The original code called:

span.recordException(error);

In my local tracing demonstration, that resulted in an exported exception event containing a synthetic marker embedded in the error message. It wasn't production data, but it gave me a reproducible example of a privacy risk.

Reproduce before fixing

I added a regression test with a fake marker:

const sensitiveMarker = "PRIVATE_PROMPT_DO_NOT_LOG";
const expectedError = new Error(
  `Provider failure; prompt=${sensitiveMarker}`
);

The test executes the failing operation, flushes the test exporter, and checks the exported span attributes and events. It also confirms that the original error object is still rethrown.

Before the fix, the assertion failed because the marker was present in the exported telemetry.

Change the event, not the failure behaviour

I replaced the raw exception recording with a generic event:

if (error instanceof Error) {
  span.addEvent("exception", {
    "exception.type": "Error",
  });
}

The wrapper still sets the span's status to error and rethrows the original exception. This removes the original message and stack trace from the exception event created by this code path while keeping a generic failure signal.

The trade-off is reduced diagnostic detail. In this practice implementation, the error type is deliberately generic. A more mature system could investigate safe, allowlisted error categories, while ensuring that user input and sensitive data are not embedded in those categories.

Verify the change

The privacy regression test passed after the fix, and the full local test suite passed 57 tests with no failures. The project also runs its automated tests, an OpenTelemetry smoke test, and a documentation-link checker in GitHub Actions.

The tracing example uses mocked SDK responses and a console exporter. It does not prove production telemetry safety or live API behaviour. The fix is not general-purpose redaction: other logging paths and exporters need separate review.

Code and evidence: openai-sdk-docs-practice on GitHub.

For background, OpenTelemetry's exception conventions note that exception messages can contain sensitive information: OpenTelemetry exception conventions.

How do you decide which exception details to keep in telemetry by default?

My First OpenAI Streaming Example: The Code, the Tests, and the Edge Cases

Virginia Nyambura Mwega — Thu, 15 Oct 2026 14:45:02 +0000

I wanted to understand AI streaming by building it—not just reading about it.

So I added a streaming command to my OpenAI SDK practice project using JavaScript and the OpenAI Responses API.

The goal was simple: instead of waiting for the entire AI response, display text fragments as they arrive.

Along the way, I got to practise something I care about just as much as the implementation: making an example understandable, testable, and useful to the next developer.

Here is what I built and what I learned.

Project: OpenAI SDK Docs Practice on GitHub

1. The streaming function

The core implementation is a small asynchronous function:

export async function streamText(client, prompt, onText) {
  if (typeof prompt !== "string" || !prompt.trim()) {
    throw new Error("Prompt is required.");
  }

  if (typeof onText !== "function") {
    throw new Error("A text callback is required.");
  }

  const stream = await client.responses.create({
    model: MODEL,
    input: prompt,
    stream: true,
  });

  for await (const event of stream) {
    if (event.type === "response.output_text.delta") {
      onText(event.delta);
    }
  }
}

MODEL is defined elsewhere in the project.

A few things are happening here:

I validate the prompt before making the request.
I validate the callback so the caller has a way to receive text.
I enable streaming in the API request.
I consume the asynchronous event stream.
I forward only text-delta events to the callback.

In the command-line interface, the callback uses process.stdout.write() to display each fragment as it arrives.

To run the example with a valid API key configured locally:

node index.js stream "Explain how an API works"

The project uses OPENAI_API_KEY from a local .env file. Never commit a real API key to your repository.

2. What I decided to test

I didn't want to stop at checking that the function returned without throwing.

Because streaming is event-driven, I wanted to check what happened as different events arrived.

I created a mocked client that yields simulated events, including two text fragments and a completion event.

The test verifies that the received fragments are:

["Hello", ", developer!"]

Joining them produces:

Hello, developer!

The completion event isn't forwarded to the text callback.

I also added coverage for:

Ignoring non-text events
Rejecting an empty prompt
Rejecting a missing callback
Propagating an SDK error when stream startup fails

This makes the tests repeatable without depending on a network connection or consuming API usage.

3. The result—and an important limitation

The latest reported run of the project's automated test suite was:

19 passed, 0 failed.

That is a useful milestone, but I want to be precise about what it means.

The streaming tests use mocked events. I haven't tested a live API stream because I don't currently have an API key.

The tests verify my application's behavior with the events and failures I simulate. They don't prove that a real provider request succeeds or that the implementation handles every possible interruption.

My next improvements would include testing a stream interrupted after some text has already arrived, defining how partial output should be reported, and verifying a live integration when credentials are available.

4. The documentation is part of the feature

I also documented the implementation in STREAMING-COOKBOOK.md.

It covers:

Prerequisites
How the streaming function works
The command to run
How to execute the automated tests
Common setup and troubleshooting issues
What has and hasn't been verified

This exercise reminded me that a code example is much more useful when someone else can follow it without guessing.

A working snippet alone doesn't explain how to configure it, test it, or diagnose a failure.

5. What I learned about Developer Experience

The biggest lesson wasn't simply how to use stream: true.

It was how much of the developer experience lives around the API call:

Validate → Request → Consume events → Handle errors → Test → Document

Every step gives the next developer more confidence and fewer surprises.

I'm building these examples as part of my journey into AI Developer Documentation and Developer Experience. I want to be able to build the integration, understand its limitations, and explain it clearly—not just write a tutorial around code I haven't tested.

If you'd like to explore the project, here's the repository:

OpenAI SDK Docs Practice

I'd love to learn from other developers

When you're implementing a streaming API, which edge case do you make sure to test first?

A failure before the stream starts
A connection dropping halfway through
Cancellation when the user leaves the page
Handling partial output after a failure

I'd be interested to hear how you approach it.

Verification note: Automated tests passed in the latest reported run, using mocked streaming events. Live API streaming has not yet been verified.

Your API Can Work Perfectly and Still Deliver a Bad Developer Experience

Virginia Nyambura Mwega — Thu, 08 Oct 2026 16:31:42 +0000

Your API Can Work Perfectly and Still Deliver a Bad Developer Experience

I thought I was testing an API. I was actually testing everything around it.

An API can work perfectly and still frustrate developers.

The server starts.

The endpoint responds.

The tests pass.

And somehow, someone still gets stuck.

I discovered this while building a small Express API for a Developer Experience experiment.

Instead of stopping when the API worked, I decided to use it like a developer discovering the project for the first time.

I didn't rely on what I knew about the project.

I followed what a new developer would actually see.

I read the README.

I ran the commands.

I copied the examples.

I tested successful requests.

I sent invalid requests.

I read the errors.

And I compared the documentation against the actual application.

That's when I started finding friction.

The API Was Simple

The API only had two endpoints:

```text id="a7u1m2"
GET /hello
POST /hello




A request to `GET /hello` returns:



```json id="h6p9r3"
{
  "message": "Hello, Developer!"
}

The POST endpoint accepts:

```json id="v2k8q4"
{
"name": "Virginia"
}




and returns:



```json id="j5s1d8"
{
  "message": "Hello, Virginia!"
}

For an invalid request, it returns:

```json id="n3x7b5"
{
"error": "Name is required"
}




It's a deliberately small API.

But that made it easier to focus on something I was becoming more interested in:

What does it actually feel like to use this API?

---

 First Problem: The README Told Me to Run a Command That Didn't Exist

The README instructed developers to run:



```bash id="q4w6e2"
npm run dev

Except there was no dev script.

That meant the documented onboarding path failed immediately.

This was interesting because the documentation itself looked reasonable.

The problem only became obvious when I actually followed it.

I fixed it by adding:

```json id="r8t2k6"
{
"scripts": {
"dev": "tsx watch src/server.ts",
"test": "vitest"
}
}




Now the command in the README matched the actual project.

That gave me my first real lesson:

Documentation has to be tested, not just written.**

---

 Second Problem: The API Worked, But the Example Didn't

The original request example used Unix-style `curl` syntax.

I was testing the project in Windows PowerShell.

The API wasn't broken.

The example was.

That's an easy problem to overlook when you're the person who built the API.

You already know how everything works.

A new developer doesn't.

So I replaced the example with a PowerShell-friendly request:



```powershell id="p1m4x7"
Invoke-RestMethod -Uri "http://localhost:3000/hello" -Method Post -ContentType "application/json" -Body '{"name":"Virginia"}'

This was a small change, but it made me think more carefully about developer environments.

A command isn't useful simply because it's technically correct.

The developer has to be able to copy it, run it, and understand what happens next.

Third Problem: The Happy Path Isn't Enough

Documentation often focuses on the successful request.

For example:

```json id="c8v3n1"
{
"name": "Virginia"
}




But what happens when the developer sends:



```json id="k2f7q9"
{}

The API returns:

```json id="d6r1s5"
{
"error": "Name is required"
}




That error is part of the developer experience.

It's not just an implementation detail.

It tells the developer:

* what went wrong
* which input matters
* what the API expects

This made me think about API documentation differently.

Sometimes the best documentation isn't another paragraph.

Sometimes it's a good error message.

---

 Tests Became Documentation Insurance

After manually testing the API, I wanted a repeatable way to verify its behavior.

So I added automated tests using **Vitest** and **Supertest**.

The final suite covered:

* successful GET request
* successful POST request
* invalid `name`
* missing `name`

Result:



```text id="w5n8c2"
Test Files  1 passed
Tests       4 passed

This gave me confidence that the behavior I was documenting was behavior the application actually supported.

It also made me think about documentation drift.

An API can change.

The README might not.

An error message might change.

The documentation might not.

A response format might change.

The example might not.

Tests can help catch some of that gap.

So I started thinking of tests as a kind of:

documentation insurance

Not a replacement for documentation.

A safeguard against documentation silently becoming outdated.

Then I Added OpenAPI

Once the API behavior was stable, I documented it using OpenAPI.

The specification describes:

GET /hello
POST /hello
request body requirements
successful responses
validation errors
operation IDs
the local development server

I also validated the specification with Redocly.

This gave the project another useful layer:

machine-readable API documentation.

Instead of having the implementation and prose documentation as the only sources of information, the API also had a structured specification.

That matters when you're building developer-facing products.

Developers don't all consume documentation the same way.

Some want a QuickStart.

Some want endpoint references.

Some want examples.

Some want machine-readable API definitions.

Good developer documentation needs to support the journey, not just describe the technology.

Docs-as-Code Started Making More Sense

The project eventually included:

QuickStart documentation
API reference information
troubleshooting guidance
a DevEx audit
automated tests
OpenAPI documentation
Redocly validation
GitHub Actions CI
a case study documenting the investigation

I also used Git branches and pull requests to make documentation changes.

That changed how I think about docs-as-code.

It isn't simply:

"Put Markdown in Git."

It's treating developer-facing information as part of the product development process.

Write it.

Review it.

Test it.

Validate it.

Maintain it.

Verify it.

The Developer Experience Loop

The biggest thing I took away from this project wasn't a particular tool.

It was a workflow.

I started thinking about Developer Experience like this:

```text id="s3j7p1"
Build
↓
Document
↓
Use
↓
Find friction
↓
Improve
↓
Test
↓
Verify
↓
Repeat




That changes the question.

Instead of asking:

 "Did I write the documentation?"

I want to ask:

 "Can a developer actually make progress with it?"

That is a much more useful question.

---

 My New Definition of "Done"

Before calling API documentation finished, I'd now check:

 Can a developer get started?

The setup instructions should work from a fresh environment.

 Can they run the examples?

A developer shouldn't have to rewrite your commands before they work.

 Do the examples match reality?

The documentation should reflect the actual API.

 Are errors understandable?

Developers should have enough information to know what to fix.

 Is there a path when something goes wrong?

Troubleshooting shouldn't begin and end with "check your setup."

 Is important behavior tested?

Automated tests can help protect documented behavior.

 Does the API specification match the implementation?

OpenAPI shouldn't become a second, outdated version of the API.

 Has someone actually followed the documentation?

This might be the most important one.

Because documentation can look excellent until someone actually tries to use it.

---

 The Shift

This project changed the question I ask about documentation.

Before:

 "Is this documentation accurate?"**

Now:

 "Does this documentation help the developer succeed?"**

Those aren't the same thing.

A page can be technically accurate and still be difficult to use.

A command can be correct and still fail in someone's shell.

An API can return the right status code and still provide an unhelpful error.

A README can contain everything a developer needs and still leave them thinking:

 "Okay... what do I do now?"

That gap is where Developer Experience lives.

---

 The Best Way to Find Developer Friction?

Become the developer.

Don't just review your README.

Follow it.

Don't just look at your example.

Run it.

Don't just document your errors.

Break the API.

Don't just say the tests pass.

Use them to verify the behavior you promise developers.

That shift in perspective uncovered more issues for me than simply reading through the code.

---

 What This Small Project Taught Me

I started this project thinking I was building an API.

I ended up learning how much of the developer experience exists outside the API itself.

The code has to work.

The documentation has to work.

The examples have to work.

The errors have to help.

The tests have to verify.

And the journey has to make sense.

That's the standard I want to carry into the next things I build and document, especially around APIs, SDKs, AI integrations, and developer tools.

Because developers don't experience your codebase directly.

They experience the path you create around it.

And that path is part of the product.

 Build it. Use it. Break it. Document it. Improve it. Verify it.

---

 One question for the community

What's a small piece of developer friction you've encountered that had a much bigger impact than you expected?

I'd love to hear what you've seen.

What Building an API Taught Me About Developer Experience

Virginia Nyambura Mwega — Tue, 06 Oct 2026 12:18:33 +0000

I thought I was testing an API.

I was actually testing the developer experience.

I built a small Express API with two endpoints:

GET  /hello
POST /hello

The API itself was intentionally simple.

What interested me was what happened when I stopped looking at it as the person who built it and started using it as a developer discovering the project for the first time.

I didn't just ask:

"Does the code work?"

I started asking:

"Can another developer actually use this without getting stuck?"

That changed the way I looked at the project.

I started with the README

The README included the usual setup instructions:

npm install
npm run dev

Looks fine, right?

Except there was a problem.

The project didn't actually have a dev script.

The documentation was telling developers to run a command that the project couldn't execute.

That was my first reminder:

Documentation is a promise.

If I tell a developer to run a command, that command needs to work.

I fixed the issue by adding tsx and configuring the development script:

{
  "scripts": {
    "dev": "tsx watch src/server.ts",
    "test": "vitest"
  }
}

After that:

npm run dev

successfully started the server.

A small fix, but an important lesson:

A documentation review isn't complete until the documented steps have actually been executed.

Then I discovered a portability problem

Next, I tested the POST request example from Windows PowerShell.

My original example used Unix-style curl syntax.

It didn't behave as expected.

The API wasn't broken.

My documentation was.

I had written an example without considering the environment in which another developer might run it.

I replaced it with a PowerShell-compatible example:

Invoke-RestMethod -Uri "http://localhost:3000/hello" -Method Post -ContentType "application/json" -Body '{"name":"Virginia"}'

This worked.

The API returned:

Hello, Virginia!

That led to another important lesson:

A code example isn't finished when it looks correct. It's finished when a developer can copy it, run it, and get the expected result.

That's an important distinction in developer documentation.

I tested the failure path too

Developers don't only follow the happy path.

They make mistakes.

They send incomplete requests.

They misunderstand parameters.

They use the wrong data type.

So I intentionally sent:

{}

The API responded with:

{
  "error": "Name is required"
}

That's useful API behavior.

The developer immediately knows what went wrong and what needs to be fixed.

They don't have to inspect the source code.

They don't have to guess.

They don't necessarily have to leave the terminal and search through documentation.

The API itself provides useful guidance.

And that made me think about something important:

Error messages are part of the developer experience too.

Tests became part of the API contract

I added automated tests covering:

GET /hello
A successful POST /hello
An invalid name
A missing name

The result:

Test Files  1 passed
Tests       4 passed

These tests aren't only there to catch bugs.

They also describe the behavior the API is expected to provide.

In that sense, tests become a form of executable documentation.

If the API behavior changes unexpectedly, the tests can expose that change.

That gives developers and documentation authors something concrete to rely on.

I documented the friction

After going through the developer journey, I created two additional documents.

DEVEX-AUDIT.md

This documents:

The developer journey I tested
The friction points I discovered
Why the problems happened
What I changed
How I verified the fixes

TROUBLESHOOTING.md

This documents the actual problems encountered during setup and API usage, along with their solutions.

This distinction matters.

I wasn't writing troubleshooting content based on hypothetical problems.

I was documenting problems I had actually encountered while using the project.

That made the documentation much more grounded.

The workflow changed

The project eventually became this loop:

Build
  ↓
Document
  ↓
Follow the documentation
  ↓
Find friction
  ↓
Investigate
  ↓
Fix
  ↓
Test
  ↓
Update documentation
  ↓
Verify again

And this is probably the biggest lesson I've taken from the exercise.

Documentation shouldn't be treated as a final step after development.

It should be part of the development and feedback loop.

Developer Experience is bigger than documentation

A developer experience isn't just a documentation website.

It's the entire journey between:

"I found this project."

and:

"I successfully built something with it."

That journey can include:

Installation
Environment setup
Commands
APIs
Code examples
Error messages
Tests
Troubleshooting
Documentation
Tooling

Every one of those can either create momentum or create friction.

A beautiful README doesn't help if the first command fails.

A technically correct example doesn't help if it doesn't work in the developer's environment.

And an API that returns unclear errors creates friction even when the documentation is excellent.

Everything is connected.

The question I'm asking now

Instead of asking:

"Did I document this?"

I'm starting to ask:

"Can a developer successfully use this?"

That's a much better standard.

It pushes documentation beyond simply writing content and into:

building → testing → observing → improving.

And it gives you a way to collect evidence.

You can test the setup.

You can test the examples.

You can test the API behavior.

You can test the failure paths.

You can verify that the documentation reflects reality.

What I'm taking forward

This was a small API.

But the exercise taught me something much bigger.

I found a missing development command.

I found a platform-specific documentation issue.

I tested API validation.

I improved the error experience.

I added automated tests.

I created troubleshooting documentation.

And then I followed the documentation again to verify that the experience actually worked.

That changed how I think about Developer Experience.

Good developer documentation isn't simply about explaining how a system works.

It's about helping developers make progress.

And sometimes the best way to find out whether your documentation does that is to become the developer who has to use it.

Build it. Use it. Break it. Document it. Improve it.

ktx Docs Review: Excellent for AI Agents, One Onboarding Gap

Virginia Nyambura Mwega — Wed, 08 Jul 2026 16:29:42 +0000

A few days ago I announced a personal challenge: reviewing AI startup documentation in public — one thing done well, one place onboarding could be smoother, one rewritten example. This is review #1.

The product: ktx by Kaelio — an open-source, self-improving context layer for data agents. It teaches coding agents like Claude Code, Codex, and Cursor how to query your warehouse accurately, using approved metric definitions and business context stored as reviewable YAML and Markdown in git. You install it with npm install -g @kaelio/ktx (Node.js 22+) and run one guided ktx setup.

I tested the docs rather than just reading them. Everything below was verified live against docs.kaelio.com on July 8, 2026.

What Kaelio does really well: docs written for agents, not just humans

This is the part I want other AI startups to copy.

Most documentation assumes a human reading a browser tab. ktx treats an AI coding agent as a first-class reader:

A dedicated AI Resources route with an Agent Quickstart, Agent Instructions, and Prompt Recipes.
/llms.txt (a curated index of high-value pages) and /llms-full.txt (the full corpus) so an assistant can discover the right pages before diving in.
Per-page Markdown on demand: curl -H "Accept: text/markdown" returns clean source — frontmatter stripped, code blocks preserved, tables preserved.
Missing pages return a plain-text 404 instead of silently falling back to rendered HTML. Small detail, big kindness — it stops an agent from confidently parsing a garbage page.
Copy as Markdown, View MD, and Copy MDX actions sit right on each rendered page.

One of my standing review questions is literally: is the documentation structured so both developers and AI agents can use it effectively? ktx is the clearest "yes" I've come across. If you're building a developer product in 2026, this is the bar.

The human path is strong too: a single guided ktx setup wizard with clearly numbered steps, a live demo warehouse with paste-ready credentials, and a Common issues table that maps symptom → fix.

Where onboarding could be smoother: the MCP step the happy path skips

Here's the one seam.

The quickstart walks you through a clean sequence: install → ktx setup → ktx status (verify) → Connect a coding agent. That final section covers installing project-local agent rules with ktx setup --agents.

What it doesn't mention is that, before your agent client can actually reach ktx, you may need to start the local MCP daemon. That step lives in the README and FAQ — not in the quickstart:

The README notes: if ktx status prints ktx mcp start --project-dir ..., run it before opening your agent client.
The FAQ explains there's no hosted service — the local MCP daemon runs on demand via ktx mcp start when an agent client needs it.

So the failure mode is: a new developer follows the quickstart end-to-end, opens Claude Code or Cursor, and the agent silently can't see ktx — with no breadcrumb in the quickstart pointing at the fix. Making it slightly stickier, the ktx status example output in the Verify section shows a fully-ready project but doesn't include the ktx mcp start line the README says status can emit — so even the example doesn't prepare you for it.

To be fair: ktx is under active development, and this is one missing signpost on an otherwise excellent path. But it sits at the last mile of onboarding — exactly where a new developer's first impression is decided.

The rewritten section

Here's how I'd close the gap — a version of "Connect a coding agent" written in ktx's own voice, folding in the missing step:

Connect a coding agent

The setup wizard installs project-local agent rules in its last step. To install or change targets later:

ktx setup --agents

Claude Code and Codex also support global installs with --global. Agent rules point at the ktx CLI path that created them, so agents don't need a separate ktx binary on PATH. If the CLI path changes, rerun ktx setup --agents.

Start the MCP server if your status asks for it. Some setups serve agents through an on-demand local MCP daemon. If ktx status prints a line like:

ktx mcp start --project-dir /home/user/analytics

run that command before opening your agent client. Otherwise the agent connects to nothing and ktx tools appear empty. You can confirm with ktx status — Agent integration should read ready.

One short subsection, one status breadcrumb. That's the whole fix — and it's the difference between a silent dead-end and a thirty-second recovery.

The pattern worth noticing

The irony here is instructive: ktx has some of the best agent-facing documentation I've reviewed, and its one gap is on the human path — a step the team likely stopped seeing because they already know it's there. That's the most common failure I find across audits: docs that are perfectly correct for someone who already understands the product, which is exactly the one audience that doesn't need them.

If you write docs, the cheapest test available is still the best one: hand your quickstart to someone (or some agent) with zero context, and watch where they stall.

Next review

If you're building an AI product with public documentation, I'd love to include it — drop a link in the comments or reach out. Same format every time: one thing done well, one gap, one rewrite.

I'm Virginia Mwega — a full-stack & AI engineer specializing in documentation engineering and developer experience. Portfolio: virginia-mwega.vercel.app · Writing: virginiamwega-com.vercel.app · Connect: https://www.linkedin.com/in/virginia-mwega-196309313/

I Reviewed 10 AI Startup Documentation Sites. Here Are the 7 Mistakes I Kept Seeing.

Virginia Nyambura Mwega — Mon, 06 Jul 2026 18:12:05 +0000

Documentation is often the first product a developer experiences.

Before they see your architecture, your engineering culture, or your code quality, they interact with your documentation.

If that experience is confusing, incomplete, or frustrating, many developers won't make it to their first successful API request.

Over the past few weeks, I've been reviewing documentation from AI startups to understand what makes onboarding smooth—and where teams unintentionally create friction.

While every company is different, the same patterns kept appearing.

1. Quickstarts assume too much

Many Quickstarts jump straight into code without explaining prerequisites.

Developers are expected to know:

Where to get an API key
Which SDK to install
Required environment variables
Authentication steps

A Quickstart should help someone go from zero to a successful request with as little guesswork as possible.

2. Error messages aren't documented

Developers don't judge documentation by how it works when everything goes right.

They judge it by how quickly it helps them recover when something goes wrong.

Instead of only listing error codes, explain:

Why the error happens
Common causes
How to fix it
What to try next

Good troubleshooting documentation builds confidence.

3. Examples are incomplete

Too many examples leave out important details.

Developers shouldn't have to infer:

Authentication headers
Environment variables
Request payloads
Expected responses

Examples should be copy, paste, run, and understand.

4. There's no clear learning path

Documentation often feels like a collection of pages instead of a guided journey.

A better structure might look like this:

Quickstart
Core Concepts
Tutorials
API Reference
Advanced Guides
Troubleshooting

When developers always know what to read next, they make progress faster.

5. Documentation isn't written for AI-assisted development

Today, developers increasingly rely on AI coding assistants.

That means documentation should also be easy for AI tools to interpret.

This includes:

Consistent headings
Clear terminology
Structured examples
Explicit parameter descriptions
Predictable page organization

Well-structured documentation helps both humans and AI systems retrieve accurate information.

6. Missing "next steps"

A successful API call shouldn't be the end of the journey.

Guide developers toward meaningful progress:

Build a chatbot
Upload files
Authenticate users
Stream responses
Explore advanced features

Momentum matters.

7. Documentation is treated as an afterthought

The strongest engineering teams treat documentation as part of the product—not something that's written after the code ships.

Documentation improves:

Developer experience
Product adoption
Support efficiency
Customer success
Developer trust

It's not just a support resource.

It's a growth asset.

My Challenge

I'm starting a public challenge where I review AI startup documentation and share practical improvements.

Each review includes:

One thing the team did well
One improvement opportunity
A rewritten example or suggestion

The goal isn't to criticize.

It's to learn, contribute, and help create better developer experiences.

If you're building an AI product with public documentation, I'd love to review it.

I'm always looking for examples of thoughtful documentation—and opportunities to make good docs even better.

Thanks for reading!

The Documentation System Every Startup Should Have

Virginia Nyambura Mwega — Wed, 01 Jul 2026 16:43:37 +0000

Key Takeaways

Most documentation advice is written by people whose job is only documentation. This isn't that. This is the system I run as a solo founder shipping production AI, where docs that rot don't get caught by a docs team — they get caught by me, at the worst possible time.
A documentation system isn't a wiki. It's four separable things (Diátaxis: tutorial, how-to, reference, explanation) plus a mechanism that keeps them from drifting out of sync with the code.
The differentiator between docs that survive and docs that rot isn't writing quality. It's whether documentation is coupled to the same PR that changes the behavior, and whether it fails loudly when it goes stale.
Reference docs should be generated from typed code wherever possible, so the one category most likely to drift can't.

Why you should ignore most documentation advice (including some of mine)

Nearly every "how to do documentation" post is written by a technical writer — someone whose entire role is documentation, on a team where documentation is a budgeted function. That advice is often good, but it quietly assumes a luxury most early startups don't have: a person whose job is to notice when the docs stopped being true.

I don't have that person. I'm a solo founder building a production AI system (FamNest, a multi-agent wellness coach). When my documentation drifts, no docs team catches it. I catch it — usually at 11pm, when I'm trying to remember how my own auth boundary works and the note I wrote three months ago is now confidently wrong.

So the system I'm about to describe isn't optimized for a documentation department. It's optimized for the constraint most startups actually have: nobody's full-time job is keeping the docs honest, so the system itself has to do it. That constraint changes the answer, and it's why this isn't a generic "write good docs" post.

The four things a documentation system actually is

The single most useful mental model I've found is Diátaxis — the framework created by Daniele Procida and now used by Cloudflare, Gatsby, and a chunk of the Python ecosystem. Its core claim is deceptively simple: there aren't many kinds of documentation, there are exactly four, they serve different needs, and mixing them is the most common cause of docs that feel wrong even when every individual sentence is correct.

Tutorials — learning-oriented. A beginner, on rails, following exact steps to a known destination. Your quickstart is a tutorial. Its job is confidence, not completeness.
How-to guides — task-oriented. An already-competent user trying to accomplish a specific real-world goal. "How to configure X." Not teaching — helping someone who already knows the basics get a thing done.
Reference — information-oriented. The dry, factual, complete description of the machinery. Your API reference, your schema, your config options. Consulted, not read.
Explanation — understanding-oriented. The "why" behind a design decision, read away from the keyboard. Why the architecture is shaped this way. This is the category startups skip and later regret skipping, because it's the institutional memory of why.

The reason this matters practically: the most common documentation failure isn't a missing page, it's a tutorial that derailed into reference — the getting-started guide that stops to enumerate every flag, until a beginner who wanted to feel capable instead feels buried. Once you can name the four types, you can see the mixing, and most "our docs are confusing" problems turn out to be mixing problems, not writing problems.

The part nobody frames as engineering: keeping it from rotting

Here's the thing the framework alone won't save you from, and where the solo-builder constraint actually produces a better answer than the well-resourced one.

Documentation doesn't fail at the moment of writing. It fails silently, later, every time the code changes and the sentence describing it doesn't. A page that's 80% accurate is arguably worse than a missing page — a missing page sends you to read the source; a subtly-wrong page confidently teaches you the wrong thing and costs you an hour before you stop trusting it.

The teams that beat this don't beat it with discipline. Discipline doesn't scale and it definitely doesn't survive a solo founder's worst week. They beat it structurally, by borrowing the property that makes tests valuable: failing loudly instead of silently. Concretely, the system I run has three rules:

1. Docs live in the same PR as the behavior they describe. If a change to an endpoint doesn't touch the docs, that's a visible gap in the diff, not an invisible one discovered months later. This is the single highest-leverage practice and it costs nothing but a PR-template checkbox.

2. Reference is generated, not hand-written, wherever the code is typed. Reference is the category most prone to drift — it's the most detailed and the most tightly coupled to code. So it's the category you should hand-write the least. My Next.js API routes are typed; the OpenAPI reference is generated from those types, which means the one kind of documentation most likely to lie can't, because it has no independent existence to drift away from. (I've got a full walkthrough of the endpoint-to-OpenAPI generation as its own post — linked at the end.)

3. Some docs should be executable, so staleness breaks a build. A code example that's just text will rot. A code example that actually runs in CI fails the day it stops being true. The closer your documentation sits to something that executes, the less it can quietly lie to you. My agent-boundary contracts are validated with Zod, which means the "shape" documentation and the runtime enforcement are the same artifact — the docs can't disagree with the code because they are the code.

The startup-specific version of this

If you're pre-first-hire, here's the minimum viable documentation system, in priority order — not "write all four types of everything," which is how docs projects die, but the smallest thing that compounds:

One tutorial — a quickstart that actually works if a stranger copy-pastes it. Test it by having someone who isn't you run it. If they get stuck, that's a bug, not their fault.
Generated reference — wire your typed API to auto-generated reference so the most drift-prone category maintains itself.
A running "why" doc — one explanation file where you write down why you made the load-bearing decisions, as you make them. This is the cheapest thing on the list and the one you'll be most grateful for in a year, because it's the context you can't reconstruct later.
How-to guides on demand — write one every time you answer the same question twice. Let real friction, not a content plan, decide what gets written.

Notice what's not on the list: exhaustive coverage. A documentation system isn't measured by how many pages it has. It's measured by whether the pages it has stay true, and whether the structure tells a reader which kind of page they're looking at.

Why I'm the one writing this

Because I live on the wrong side of the constraint that makes this advice real. I don't document my systems because a style guide told me to — I document them because I'm the one who has to reload the entire context of my own architecture at unpredictable moments, alone, and the difference between a system I documented well and one I didn't is whether that reload takes five minutes or two hours.

That's also, not coincidentally, exactly the position a new engineer joining your startup is in. Every argument I just made for why I document my own system for my own future self is the same argument for why your docs determine how fast your next hire becomes useful. The solo constraint just makes the cost legible sooner, because the person paying it is me, tonight.

A documentation system every startup should have, in one sentence: four clearly separated types, reference generated from typed code, docs coupled to the PRs that change behavior, and the "why" written down as you go. Everything else is a variation on those four moves.

I build FamNest and write about the engineering underneath it. The specific pieces of the system above have their own deep-dives — endpoint-to-OpenAPI generation, validating agent boundaries with Zod, and treating Supabase RLS as living auth documentation — linked as I publish them in this series.

Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)

Virginia Nyambura Mwega — Wed, 01 Jul 2026 15:54:18 +0000

Key Takeaways

You can't unit-test a coach agent the way you test a pure function — the output is non-deterministic and "good" is a judgment call, not an assertion.
An LLM-as-judge harness lets you grade a whole test set automatically against a rubric, which is the only way solo-scale eval stays sustainable.
But the judge is itself a fallible model. If you don't design around its known biases — position, verbosity, self-preference, and quiet drift when the judge model updates — you build a green dashboard that means nothing.
The mitigations that actually work are mechanical, not prompt-magic: shuffle order on every pairwise call, pin the judge version, keep a small human-labelled anchor set, and re-check the judge against it.

The problem I actually had

FamNest's coach agent generates responses to parents — check-ins, encouragement, the occasional gentle redirect. I have a growing pile of these interactions, and every time I change a prompt, swap a model, or adjust the pipeline, I need to know one thing: did I just make it better or worse?

For normal code, that's what tests are for. I change something, the suite runs, red or green, done. But there's no assertEqual for "was this an empathetic, useful response to a tired parent." The output changes every run even at temperature zero-ish, and the quality bar is a human judgment, not a fixed string. Two responses can be worded completely differently and both be good. One can match my "expected output" word for word and still be worse than a version that didn't.

So the honest options were: read every response by hand every time I change something (does not scale past about week two), or build a harness where a model grades the outputs against a rubric. I built the harness. Then I spent an uncomfortable amount of time learning all the ways a harness like that can lie to you.

What the harness actually is

At its simplest, it's a loop:

def evaluate(test_cases, coach_agent, judge):
    results = []
    for case in test_cases:
        response = coach_agent.generate(case.input, case.context)
        verdict = judge.score(
            rubric=COACH_RUBRIC,
            user_message=case.input,
            response=response,
        )
        results.append({
            "case_id": case.id,
            "score": verdict.score,
            "reasoning": verdict.reasoning,
        })
    return results

COACH_RUBRIC is the part that matters. It's not "rate this 1–10" — that produces mush. It's specific, scored dimensions: does the response acknowledge the actual thing the user said (not a generic version of it)? Does it avoid giving medical advice? Is it the right length for the moment, or is it a wall of text at someone who's exhausted? Each dimension gets a small integer and a one-line justification, and the harness keeps the justification, not just the number — because when the score drops, the reasoning is what tells me whether the agent regressed or the judge just had an opinion.

That last distinction is the whole game.

The part where I stopped trusting the judge

Here's the failure mode that made me rebuild the whole thing. You score helpfulness at 0.91 all quarter. Then the judge model ships a minor version bump. The mean shifts a few points, the distribution narrows, and your CI gate keeps passing — so you don't look. Weeks later the agent does something genuinely bad and the eval never flagged it, because the judge changed underneath you and the number stopped meaning what it meant the day you set the threshold.

The research here is not subtle, and it's worth internalizing before you trust a single green checkmark. A 2026 RAND study that stress-tested judges across multiple benchmarks concluded that no judge was uniformly reliable, and frontier models exceeded 50% error rates on hard bias benchmarks. Consistency broke on inputs as trivial as formatting changes and paraphrasing. Separately, the classic MT-Bench work found that in pairwise comparisons, the answer in the first slot wins something like 10–15 points more often purely because it's first — position bias, nothing to do with quality.

(Worth noting the field isn't static: some 2026 reproductions find position bias has shrunk to near-negligible on current-gen models under a clean pairwise rubric, while verbosity bias stays small. Which is exactly the point — the biases move as the models move, so you measure them yourself rather than trusting a blog post from last year, including this one.)

The named biases I actually design around:

Position bias — in any A-vs-B comparison, slot order can decide the winner. Mitigation: run every pairwise comparison twice with the order flipped, and only count it if the verdict is stable across both.
Verbosity bias — longer answers tend to score higher even at matched quality. Mitigation: put length appropriateness in the rubric as an explicit dimension so the judge is scoring it on purpose instead of rewarding it by accident.
Self-preference — a judge from the same model family as the candidate tends to over-score it. Mitigation: don't let the judge be the same model as the agent it's grading. (In my case the coach runs on one provider; I judge with a different family entirely.)
Calibration drift — the silent one above. Mitigation below, because it's the most important.

The anchor set is the thing that keeps you honest

The single highest-leverage piece of the harness isn't the judge prompt. It's a small set — a few dozen cases — that I labelled by hand, carefully, once. Good responses, bad responses, and the genuinely ambiguous ones. That's my ground truth.

Every time I run the harness, it grades the anchor set too. If the judge's scores on those known cases still line up with my human labels, I trust its scores on the rest of the run. If the judge drifts on the anchor set — because the model updated, because I tweaked the rubric, because Mercury is in retrograde — I find out immediately, on cases where I already know the right answer, instead of finding out in production on a case where I don't.

This is the same instinct as the deterministic crisis floor I wrote about earlier in this series: the most consequential check should be the one that's simplest and least dependent on a model behaving. For safety, that's regex. For evaluation, it's a few dozen examples I graded with my own eyes and refuse to let a model overrule silently.

What I'd tell someone starting this

Build the harness — reading every output by hand does not scale, and an LLM judge genuinely does correlate with human preference well enough to be useful. But treat the judge as a component that can fail, not an oracle. Pin its version so it doesn't change without you deciding. Shuffle order on comparisons. Keep the reasoning, not just the score. And keep a small hand-labelled anchor set that you re-check every single run, because a green eval dashboard that you never validate is worse than no dashboard — it's the confidence of measurement without the substance, and that's exactly the kind of thing that ships a broken agent with a clean conscience.

The harness didn't remove my judgment from the loop. It moved my judgment to where it's cheap and permanent — a small set of examples I curate once — instead of where it's expensive and forgettable, which is re-reading a hundred responses every time I touch a prompt.

Part of an ongoing series documenting FamNest's architecture. Earlier posts cover the deterministic crisis floor and the multi-agent coach pipeline. Next: how we test a non-deterministic system end to end.

The doc was the spec: building a safety layer for an AI app that talks to tired parents published: false

Virginia Nyambura Mwega — Tue, 30 Jun 2026 10:23:52 +0000

"What happened when I stopped designing my AI coach for flexibility and started designing it to be predictable enough to write down."

Last month I deleted three features from my AI app. Nobody asked me to.
I removed them because I couldn't explain them clearly. And I've started treating that as a hard signal: if I can't describe how a feature behaves in plain language a tired parent would understand, it isn't finished — it's a liability.
This post is about the engineering that came out of that rule. It's specific, it has code, and it's the part of building an AI product that nobody puts in the demo.
The context
I'm building FamNest, an AI wellness tool for parents. The user I design for is the one refreshing a feeding tracker at 3am, googling "is this normal" with one hand while holding a sleeping baby with the other.
That user changes your engineering priorities. Latency matters less than predictability. "Clever" is a risk, not a feature. And the moment your AI says something subtly wrong to someone in that state, you've lost them — and possibly done real harm.
So the question I kept returning to wasn't "what can the model do?" It was "what is the model never allowed to improvise?"
Problem 1: a coach with too many answers
In the first version, my AI coach could respond to the same question a dozen slightly different ways. It felt smart. Flexible, even.
It was actually a nightmare. When I sat down to write the documentation, I couldn't promise a parent what the system would do. And if I can't promise it, why should they trust it?
The fix was to put a second model in front of the user — a safety reviewer — and give it exactly three outcomes. Not a confidence score. Not a freeform critique. Three discrete verdicts.
``type Verdict = "ok" | "revise" | "crisis";

interface ReviewResult {
verdict: Verdict;
reason: string; // for logging, never shown to the user
revisedDraft?: string; // only present when verdict === "revise"
}``
The coach generates a draft. The reviewer judges it. The whole contract fits in four lines, and that's the point — I can document it in one sentence: the reviewer either approves the draft, rewrites it, or escalates to a crisis response.

``async function generateReply(userMessage: string): Promise {
const draft = await coach.respond(userMessage);
const review = await safetyReviewer.review(userMessage, draft);

switch (review.verdict) {
case "ok":
return draft;
case "revise":
return review.revisedDraft ?? draft;
case "crisis":
return CRISIS_RESPONSE; // see below — this is not generated
}
}Problem 2: the crisis floor Here's the decision I'm most sure about. When a message hints at a crisis, the system does not generate a response. It returns fixed, human-reviewed text. Every time. Byte for byte.// This string is reviewed by a human, version-controlled,
// and documented word-for-word in the user-facing docs.
const CRISIS_RESPONSE = It sounds like you're going through something really hard right now. You don't have to handle this alone. If you're in immediate danger, please contact your local emergency number....trim();This is the opposite of how we usually think about generative AI. The whole appeal of an LLM is that it composes something new. But the situation where a parent most needs the response to be right is exactly the situation where I least want the model to be creative. I call this the crisis floor: a deterministic baseline that the system can never generate its way below. The model can make the experience better above the floor. It is never allowed to touch what happens at it. A subtle but important detail: the floor is checked before the clever path, not after.async function handleMessage(userMessage: string): Promise {
// Deterministic guardrail runs first, independent of the LLM.
if (crisisFloor.matches(userMessage)) {
return CRISIS_RESPONSE;
}
return generateReply(userMessage);
}If the generative pipeline is down, malfunctioning, or hallucinating, the floor still holds. It doesn't depend on the part of the system most likely to fail. Problem 3: what happens when the provider falls over LLM APIs go down. Rate limits hit. A region has a bad day. For most apps that's an annoyance. For an app a parent leans on at 3am, a blank screen is a broken promise. So every external dependency has a known fallback, and "the model is unavailable" is a documented state — not an exception that bubbles up to a stack trace.async function coachRespond(message: string): Promise {
try {
return await llm.complete(buildPrompt(message));
} catch (err) {
logger.warn("LLM provider unavailable, degrading gracefully", { err });
// A safe, generic, pre-written reply. Not an error.
return GRACEFUL_FALLBACK;
}
}``
The user gets a calm, honest message instead of a spinner that never resolves. Graceful degradation isn't a nice-to-have here. It's part of the trust contract.
The thing I actually learned
The pattern underneath all three: documentation wasn't something I wrote after the engineering. It was the engineering.
The doc became the spec. When a behavior couldn't be written down cleanly — three verdicts, fixed crisis text, a named fallback state — that was the signal the design was wrong, not the writing.
It turns out "predictable enough to document" is a great design constraint. It pushes you toward small, discrete contracts and away from the kind of open-ended cleverness that demos well and ships badly.
I think a lot of AI products are shipping behavior they can't fully explain and quietly calling that confusion "intelligence." For the users I build for, clarity is the feature.
If you can't document it, you don't understand it yet.

I write about building and documenting production AI systems. If you've shipped a guardrail or fallback you're proud of — or one that bit you — I'd genuinely like to hear about it in .

Writing API docs an AI agent can actually consume

Virginia Nyambura Mwega — Mon, 29 Jun 2026 15:01:58 +0000

Your docs are written for a human who can guess. The agent calling your API can't.

I found the gap the embarrassing way: one of my own agents couldn't call one of my own APIs.

FamNest runs a small agent graph — a router hands a parent's message to a retriever, the retriever grounds an answer in a vetted corpus, a coach agent (Groq, Llama 3.3 70B) drafts a reply, and a safety-reviewer agent signs off before anything reaches a human. The agents call internal endpoints the same way a third-party integrator would. And one afternoon the coach kept constructing malformed calls to the retrieval endpoint — wrong field name, missing a required filter, occasionally inventing a parameter that never existed.

The endpoint wasn't broken. The docs were. They were written for a human who could fill in the blanks, and the agent had no blanks to fill — only the tokens I gave it.

That's the whole lesson, and it's worth more than a trend.

The trend everyone's shipping — and where it stops

If you've touched developer tooling in 2026 you've watched llms.txt go from a September-2024 proposal to a routine piece of infrastructure. It's a Markdown file at your domain root that points AI systems at the content that matters, with a one-line summary of each link. Mintlify, Fern, and GitBook ship one-click toggles for it. IDE agents — Cursor, Windsurf, Claude Code, Copilot — fetch it when you point them at a docs site, then pull only the linked pages they need before writing code. LangChain even shipped an MCP server (mcpdoc) that hands those files to host apps as a fetch_docs tool.

People are calling this the Business-to-Agent web, and the framing is right: just as you once needed a site humans could navigate, you now need surfaces agents can route on. Ship the llms.txt. It's a half-day of work.

But notice what it actually solves: discovery. It answers "which page matters." It says nothing about the harder question that broke my coach agent:

Once the agent has found your endpoint, can it call it correctly on the first try — with no human in the loop to recover when your prose is ambiguous?

That's not a discovery problem. That's a contract problem. And it's where most docs quietly fail.

An agent is a different kind of reader

A human reading your docs brings a lifetime of priors. They infer that userId is probably a UUID. They notice the example uses snake_case and adjust. They hit a 400, shrug, read the error, and try again. If they're really stuck they ask a teammate. Human docs can be good enough because the human closes the gap.

An agent closes nothing. It has your tokens and a probability distribution. It pattern-matches structure: if your example shows one field, it produces one field; if you describe an error in a sentence, it treats the sentence as flavor, not as a branch it has to handle. Ambiguity doesn't make an agent cautious — it makes it confident and wrong.

So the doc stops being documentation and becomes the interface itself. Everything the agent will ever know about your endpoint is in the text. If a fact isn't on the page, it doesn't exist.

That reframes what a good endpoint doc has to contain. Here are the five things mine were missing.

A typed schema, not a prose description

Prose says: "Send the user's question and an optional list of topic tags."

A schema says exactly what's allowed, and an agent can pattern-match it without guessing:

ts// retrieve — request const RetrieveRequest = z.object({ query: z.string().min(1).max(2000), topics: z.array(z.enum(["sleep", "feeding", "behavior", "self_care"])) .max(4) .default([]), topK: z.number().int().min(1).max(10).default(5), });

The difference is the enum, the bounds, the default. "Optional list of topic tags" let my coach invent "toddler_tantrums". z.enum([...]) makes the valid set unguessable-wrong. Publish the schema, not a paragraph about the schema.

Exhaustive examples — including the unhappy paths

Agents copy examples. Whatever you show is what you'll get back. If your only example is the happy path, the happy path is the only thing the model knows how to produce.

So I document the empty result and the rejected request as first-class examples, not footnotes:

``jsonc// 200 — results found
{ "matches": [{ "id": "c_18", "score": 0.82, "text": "..." }], "truncated": false }

// 200 — valid query, nothing relevant (NOT an error)
{ "matches": [], "truncated": false }

// 422 — query failed validation
{ "error": "validation_error", "field": "topics", "detail": "unknown topic 'toddler_tantrums'" }`

The middle case is the one humans leave out and agents desperately need. "No matches" is a normal outcome, not a failure — and if you don't say so, the agent will treat an empty array as a bug and retry forever.

An error taxonomy with recovery semantics

Most docs describe errors. Agents need to be told what to do about them. "Returns 429 when rate-limited" is a description. An agent needs a decision.

So I ship a table where every row ends in an action:

Code	`error`	Cause	What the caller should do
422	`validation_error`	Bad input	Fix the field named in `detail`; do not retry unchanged
429	`rate_limited`	Too many calls	Back off using `Retry-After`; retry the same body
503	`model_unavailable`	Upstream LLM down	Fall back to cached/deterministic path; do not retry tightly
409	`idempotency_conflict`	Key reused, different body	Stop; surface to a human

A human reads that table for reference. An agent reads it as a control-flow graph. The "do not retry" cells are the ones that stop a confused agent from hammering your endpoint at 3am.

An explicit determinism / idempotency contract

The single most useful sentence I added to any endpoint doc was: "Is it safe to retry this?"

Agents retry. Networks are flaky, and a retried call that isn't idempotent is how you double-charge a card or send two replies to one anxious parent. For anything with a side effect, I now state the contract in the doc itself:

Idempotency: required for POST /coach/reply and all payment routes. Send an `Idempotency-Key` header (UUID). Replays with the same key + same body return the original result. Same key + different body → 409. Retrieval (GET /retrieve) is side-effect-free and safe to retry freely.

That paragraph is the difference between a retry loop that heals and one that does damage. It's also the kind of thing humans infer and agents simply won't — there is no prior that tells a model your payment webhook is replay-safe. You have to say it.

Auth and limits as data, not folklore

"Authenticated requests only, please don't spam it" is not a contract. Scopes, the exact header, the rate limit, and the window belong in the doc as structured fields the agent can read and self-regulate against:

`plaintext

Auth: Bearer token inAuthorization. Scopecoach:readfor /retrieve. Limits: 60 req/min/token. On exceed → 429 +Retry-After` (seconds).

`plaintext

Now the agent can pace itself instead of discovering your limit by tripping it.

Keep it honest: one source of truth

All of this rots the moment your docs and your code disagree — and an agent can't smell a stale doc the way a human can. So the contract has to be generated, not hand-maintained.

My chain is boring on purpose: the typed Next.js handler validates with the Zod schema, the schema generates the OpenAPI spec, and my llms.txt links to the generated reference. The schema is the only thing I edit. The doc can't drift, because the doc is downstream of the thing that's actually true.

*Zod schema ──► request validation (runtime)
│
└────────► OpenAPI spec ──► /llms.txt entry ──► agent reads it
*
If the handler changes, every artifact downstream changes with it. The doc lies only if the code lies.

The test that actually proves it

Here's the check I run before I trust an endpoint doc: give a fresh model only the doc — no codebase, no context — and ask it to (a) construct a valid call and (b) handle a seeded error. If it can't, the gap is in the doc, not the model. I keep these as tiny snapshot tests next to the endpoint, so a doc regression fails CI like any other bug.

When my coach agent broke, this test would have caught it in seconds. The retrieval doc, fed to a cold model, produced exactly the malformed call I saw in production — because the ambiguity was right there on the page.

This isn't new. It's just a new reader.

None of this is novel discipline. "Unambiguous, complete, verifiable" is the spine of IEEE 29148 — the requirements standard I write everything against. What's changed is that the consumer of your interface is no longer guaranteed to be a person who can paper over a vague spec. Half your integrators in 2026 are agents, and they read your docs literally, exhaustively, and without charity.

Ship the llms.txt so agents can find you. But the thing that makes them succeed once they arrive is older and less glamorous: a contract precise enough that it can't be misread. The agentic web doesn't need prettier docs. It needs docs that can't be guessed wrong.

I build FamNest, an AI wellness coach for busy parents, and write about production reliability and safety for multi-agent systems. If you're documenting an agent-callable API and want a second pair of eyes on the contract, my notes are open.