We started writing docs for AI agents, not humans — and made it an open standard

Iwasoft — Sun, 30 Aug 2026 07:50:56 +0000

Most people install our tools through an agent now

We build a few developer products — an event-ledger database, an S3-compatible
object store, and others. Over the last few months we noticed a shift: most
people who try them don't start in a browser. They tell a coding agent “add this
to my project,” and the agent does the install and the wiring.

So we started watching agents work against our documentation.

Watching an agent guess

Most of the time it went fine. But every so often the agent would invent a
config flag that didn't exist, or reach for the wrong port, or call an endpoint
that was almost right. It wasn't the model being dumb — our docs were written
to persuade and onboard a human, and the agent was reconstructing facts from
prose three paragraphs apart. A fact an agent has to infer is a fact you failed
to state.

A second document, for a different reader

So per release we started shipping a second file: documentation.ai.md. Not a
terser translation of the human docs — a different document, for a different
reader, held to a stricter standard of precision.

It's English, dense, and self-sufficient: an agent that has read only that file
can install, configure and call the product. Fixed sections, in order:

One-line identity
Install / run — exact commands, exact image names and ports
Configuration — every env var / key, with its default
API quickstart — a real request/response, not pseudocode
Admin surface — how the first credential is obtained, the common operations
Architecture facts that affect integration
Links

No marketing language. Absolute honesty about status — if something is early or
unaudited, the file says so, because an agent that recommends an unstable feature
to a user because the doc oversold it is a real failure.

“Why not just llms.txt?”

Fair question, and we get it a lot. We're not trying to replace
llms.txt. It's a great site-level index of your content.
documentation.ai.md is a per-product, per-release operational doc — closer in
spirit to llms-full.txt: complete, not an index. They compose cleanly.

We made it a standard

It worked well enough across our products that we wrote it up as an open standard
(CC BY 4.0), so anyone can adopt it:

GitHub: https://github.com/iwasoftcom/ai-docs-standard
Write-up: https://iwasoft.com/blog/ai-docs-standard

If you build developer tools, your users' agents are already reading your docs.
This is just writing the version they actually need.

We didn't build a code-review agent. Here's why.

Iwasoft — Thu, 27 Aug 2026 09:13:35 +0000

We didn't build a code-review agent. Here's why.

We just shipped a Code Review API for Consensus Room, and the most interesting part of building it wasn't the AI — it was deciding what not to build.

The tempting wrong answer

The obvious pitch is: "point our agent at your repo, it reviews everything, done." That's what a lot of AI code review tools do. We almost built it too.

Then we looked at what it would actually require:

Tool-calling/function-calling wired up per LLM provider (we had zero of this — our stack is pure text completion via Claude, GPT, Gemini, and a dozen OpenRouter models)
A sandboxed execution environment to let models poke around an arbitrary user-submitted repo — a real security surface on a multi-tenant SaaS
Open-ended cost. If an agent decides how many files to read and how many tool calls to make, you can't tell a user "this will cost $0.03" before you start. Our whole product's trust model is a pre-flight cost estimate before anything runs.
Much higher latency — a multi-step tool loop per review vs. a couple of LLM calls.

None of that is impossible. It's just a different, much bigger project than "let multiple models review this code and merge their findings."

What we built instead

A single, boring, fast endpoint: POST /api/v1/code-review. You send one code unit — a file, a function, a diff — and get back a structured report from multiple models plus a merged verdict. That's it. No repo access, no tool calls, no crawling.

The trick is who calls it. If your own coding agent (Claude Code, Cursor, a CI script, whatever) already knows how to walk a repo and decide what's worth reviewing, it can just call our endpoint once per file as part of its own loop. We don't need to reinvent repo traversal — every coding agent already does that. We just need to be a really good "get a second, third, and fourth opinion on this specific piece of code" primitive that's cheap and fast to call repeatedly.

This reframing cut the actual engineering scope by an order of magnitude, and it's honestly a better fit for how people already work with coding agents.

The part that was still a real problem: structured output across providers

We wanted findings back as structured data — severity, category, line number — not just a paragraph of prose. Obvious answer: JSON mode.

Except we support Claude, GPT, Gemini (via an OpenAI-compatible endpoint), and whatever OpenRouter has that week. Checking the actual Spring AI option classes: OpenAiChatOptions has a responseFormat field. AnthropicChatOptions does not. There's no single JSON-mode contract that works uniformly across all of them.

So we didn't fight it. Each reviewer is instructed to emit one strict line format instead:

- [SEVERITY:HIGH] [CATEGORY:security] [LINE:42] SQL injection — user input is concatenated directly into the query.

The server regex-parses that into structured JSON for the API response. If a model ever drifts from the format, the raw text is still returned in full alongside the (possibly incomplete) parsed findings — a parsing miss never silently drops information, it just means one field is less structured than usual.

The moderator step takes every reviewer's raw findings, deduplicates the ones multiple models independently flagged ([CONFIRMED BY:2/3]), and closes with a single VERDICT: APPROVE|COMMENT|REQUEST_CHANGES line.

Making it resumable, because money is involved

Every LLM call costs real money the moment it completes, and it's charged immediately — not batched at the end. That means a review that fails halfway through (a provider hiccup, balance running out mid-flight) must not throw away work you already paid for.

So the run loop is checkpointed: every reviewer's finding is persisted the instant it comes back. If a review fails or pauses, calling it again reloads what's already stored and only retries what's missing — nothing gets recharged, nothing gets lost. This wasn't a nice-to-have we added later; it came directly from asking "what happens when this fails halfway, with a customer's money on the line."

Trying it

curl -s "https://consensusroom.com/api/v1/code-review?wait=90" \
  -H "Authorization: Bearer cr_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "code": "def get_user(username):\n    query = \"SELECT * FROM users WHERE username = \" + username\n    return db.execute(query)",
    "filename": "users.py",
    "panelists": [
      {"key": "claude", "model": "claude-haiku-4-5", "role": "security"},
      {"key": "gpt",    "model": "gpt-4o-mini",       "role": "bugs"}
    ]
  }'

Full reference (fields, limits, error codes, the exact parsing regex) is in documentation.ai.md — written to be pasted straight into an agent's context.

Would genuinely like feedback from anyone who's solved the "structured output across heterogeneous LLM providers" problem differently — curious if there's a cleaner approach we're missing.

DEV Community: Iwasoft

We started writing docs for AI agents, not humans — and made it an open standard

Most people install our tools through an agent now

Watching an agent guess

A second document, for a different reader

“Why not just llms.txt?”

We made it a standard

We didn't build a code-review agent. Here's why.

We didn't build a code-review agent. Here's why.

The tempting wrong answer

What we built instead

The part that was still a real problem: structured output across providers

Making it resumable, because money is involved

Trying it