DEV Community: Nazar Boyko

AI Code Review: Helpful Assistant Or False Confidence Machine?

Nazar Boyko — Tue, 02 Jun 2026 05:03:21 +0000

You open a pull request. Thirty seconds later, an AI reviewer drops a comment: "Looks good to me. No issues found."

You feel a tiny chemical reward. Approval. Speed. You're one click closer to merging. The cognitive cost of waiting for a human reviewer just got compressed into half a minute, and the diff you spent two hours wrestling with is now blessed by a system that has read more code than any single engineer alive.

Then a week later, the same code ships a subtle authorization bug to production, and you're staring at an incident channel wondering how that "Looks good to me" survived a real review.

This is the question that follows every AI code reviewer around like a shadow. Is it a helpful assistant - a faster, calmer, more patient version of the senior engineer who used to leave you twelve comments before lunch? Or is it a false confidence machine - a tool that says reassuring things about code it doesn't fully understand, and convinces you to merge anyway?

The honest answer is both, and which one you get depends almost entirely on how you set up the workflow around it. Let's break down the four angles that actually matter - what these reviewers catch well, where they fail on correctness, where they fail on security, where they make things up, and how to design a review process that gets the upside without buying the downside.

What AI Code Review Is Actually Good At

Before we pile on the failure modes, it's worth being fair about what these tools do well - because most teams hire them for the right reasons and then forget what those reasons were.

AI reviewers are excellent at the surface layer of code review. The kind of comments that good engineers stop making out loud because they got tired of writing them but still wish someone would catch:

Inconsistent naming inside the same diff. You called it userId in one function and user_id in the next.
Style drift. A trailing comma here, a missing one there, a callback where the rest of the file uses async/await.
Obvious nullability holes. You destructured response.data.user two lines after a try block that could return undefined.
Forgotten error handling. The new endpoint catches the database error but swallows the auth error two lines below.
Dead code, unused variables, imports that no longer point at anything.
Documentation that contradicts the function it's documenting because the signature changed and the JSDoc didn't.

These comments aren't glamorous, but they used to consume real review bandwidth from senior engineers. Pushing them onto a tool that doesn't get tired, doesn't get cranky, and doesn't write "as I've mentioned in the last four reviews..." in the comment thread is a clear win.

The second thing AI reviewers are good at is mechanical pattern matching against well-known antipatterns. The same patterns that have been on the OWASP Top 10 for fifteen years. The same patterns that have been in every senior interview question for ten years. The same patterns that show up in a thousand training-set blog posts.

If you write a SQL query by concatenating a string with req.query.user, an AI reviewer is going to spot it. If you skip CSRF protection on a state-changing endpoint, it'll spot it. If you log a raw password, it'll spot it. If you commit a JWT secret, it'll spot it.

What both of these categories have in common is that the code reveals the problem. The function looks suspicious. The diff is the evidence. The reviewer doesn't need to know what your system does - it only needs to read the lines you handed it.

This is the half of code review that AI is genuinely changing. And if your team treats AI review as exactly this much and no more, you'll get a real productivity win with very little risk.

The problem starts the moment you start expecting it to do the other half.

The Other Half: Correctness, Intent, And Where Bugs Actually Live

Real bugs almost never live in the code you can see in the diff. They live in the relationship between the code in the diff and the rest of the system.

Think about the last few real bugs your team shipped. Walk through them. How many of them were caught by:

A reviewer noticing that the new endpoint duplicated logic from another endpoint, and now both versions disagreed?
A reviewer remembering that the field account.status could be pending_review only in tenants migrated before 2022, and the new code didn't handle that case?
A reviewer realizing that the new background job ran every five minutes, but the table it queried got partitioned by date last quarter, and the query was about to start scanning the entire archive partition?
A reviewer noticing that the new logic was correct in isolation, but the caller already did the same check three frames up, and now the count was off by one?

None of those bugs are visible in the diff. The diff looks fine. The diff is fine, locally. The bug lives in the conversation between the diff and everything outside it - the other twenty thousand lines of your codebase, the migration that ran two years ago, the operational reality of how the system runs in production.

This is where AI code review hits its hardest ceiling, and it's worth being precise about why. It's not that the model is bad. It's that the information needed to catch the bug isn't in the prompt. The reviewer is reading the diff. It might be reading the surrounding file. It might even be reading some retrieved chunks of your codebase. But it isn't reading:

The Slack thread from two years ago where the team decided that all auth checks happen at the middleware layer, not the controller.
The runbook that says this service runs in three regions and the new code's assumption of a single timezone breaks two of them.
The product manager's verbal decision that "soft delete" means hidden-from-list-but-still-billable, not gone.
The unwritten rule that any for await in this codebase needs a concurrency limit because the database connection pool is sized for ten.

A senior reviewer who's worked on your team for two years catches these because the system lives in their head. The AI catches them only if some surface artifact of those decisions made it into the code or into the context window. Most of the time, none of it did.

The failure mode here isn't wrong feedback. It's confident absent feedback. The AI reviewer doesn't flag the bug because it doesn't see the bug. And worse, it tells you "no issues found", which sounds like an affirmative review, not an admission of partial coverage. A human who didn't know the codebase well would at least hesitate. A model doesn't hesitate. That's the false confidence.

The fix isn't to expect the model to do more. It's to stop treating its silence as evidence.

Security Review Is The Sharpest Edge Of This Problem

Security is where the gap between "what looks suspicious in the diff" and "what's actually exploitable" gets the widest, the fastest.

Consider three diffs:

diff-1.py

@app.route("/users/<user_id>")
def get_user(user_id):
    query = f"SELECT * FROM users WHERE id = {user_id}"
    return db.execute(query).fetchone()

This is a SQL injection. Every AI reviewer alive will catch it. It's in the training data of every model from every era. It's also unlikely to ship in any team that has any review at all.

diff-2.py

@app.route("/users/<user_id>")
@authenticated
def get_user(user_id):
    user = db.users.find_one({"id": user_id})
    return user.to_dict()

This looks clean. There's an @authenticated decorator. The query is parameterized. The return is structured. An AI reviewer reads this and says "looks good" - and on the surface, it does.

But this might be an Insecure Direct Object Reference (IDOR) vulnerability. There's no check that the authenticated user is allowed to fetch this particular user_id. Anyone who's logged in can read anyone else's profile.

The model can't catch this without knowing the intent of your authorization model. Does your system let authenticated users read all other users? Maybe - it's a social app. Or maybe not - it's a healthcare app and each user can only read themselves and their dependents. The diff doesn't say which. The reviewer doesn't know which. So it defaults to a comfortable answer, which is silence.

diff-3.py

@app.route("/teams/<team_id>/invite", methods=["POST"])
@authenticated
def invite_member(team_id):
    email = request.json["email"]
    team = db.teams.find_one({"id": team_id})
    team.invite(email)
    return {"ok": True}

Same shape, harder to spot. Even if the model checks for the user's authorization to act on team_id, it might miss that request.json["email"] is not validated, that the same endpoint can be used to enumerate which emails are already members based on the response shape, that the rate limit on this endpoint is missing because the team's existing pattern is to put rate limits on the gateway and this new endpoint is registered through a different path. None of that is visible in the eight lines you can see.

This pattern repeats across every security category. Authorization bugs depend on the rules of your domain. Race conditions depend on the concurrency model your runtime actually uses in production. Cryptographic mistakes depend on which threat model you've accepted. Secrets handling depends on which systems the data flows through after this code returns. None of it is in the diff. All of it is in your team's head.

The AI reviewer is genuinely useful as a first pass on security - it'll catch the patterns that match the training data, the OWASP-shaped problems, the obvious dumb stuff. The danger is treating that first pass as a security review. It isn't. Real security review needs someone who knows what your application is for, what data it holds, who's allowed to do what, and what an attacker would actually want.

Warning
When the AI reviewer says "no security issues found", read it as "no patterns from my training set matched the diff". That is not the same sentence.

The Hallucination Problem In Review Specifically

Hallucination in code generation is a known and well-discussed failure mode. Models invent function names, package versions, config flags, API endpoints. You've probably watched a model confidently import a module that doesn't exist.

What's less discussed is what hallucination looks like in code review. The shape is different - and arguably more dangerous - because the model isn't writing code you'll run. It's making assertions you'll act on.

A reviewing model can hallucinate in three distinct ways, and each one bites a different way.

Phantom references. The reviewer comments: "This is similar to the pattern in src/auth/middleware.py where we already handle this case." You open src/auth/middleware.py. There is no such pattern. There might not even be a file at that path. The model invented a plausible reference because that's what its training distribution rewards.

You can spot this one if you check, but the dangerous version is when the reference is almost real - the file exists, the function name is close but slightly wrong, the pattern is similar but not identical. You scan it, decide "yeah, that looks consistent", and merge. The reviewer has just convinced you of a fact that isn't quite true.

Phantom guarantees. The reviewer comments: "The framework handles input validation for you here, so you don't need to add it explicitly." Sometimes this is true. Sometimes the framework has some validation but not the kind that matters for this endpoint. Sometimes the framework had this feature in v3 and you're on v5 and they removed it. Sometimes the framework never had it and the model is mixing it up with a different framework's docs.

You're now relying on an assertion about a system you didn't verify. The diff didn't change. The vulnerability is now live, and the reasoning that justified shipping it is a sentence that sounded confident in a review thread.

Phantom approvals. The reviewer reads the diff, doesn't understand a chunk of it, and chooses safety: it says nothing about that chunk, and says positive things about the parts it does understand. The overall summary lands as "looks good". You read the summary. You don't notice that the most complex part of the diff was never actually addressed.

This is the quietest failure mode and the easiest to miss. The reviewer didn't say anything wrong. It just didn't say anything about the part that mattered. And because there's no visual indicator of "I read this and have no comment" vs "I didn't really understand this and have no comment", you can't tell the difference.

The defence against all three is the same and it's old-fashioned: don't trust assertions. When the AI reviewer references a file, open the file. When it claims the framework does X, find the docs. When the diff has a tricky section and the AI didn't comment on it, that's a signal, not a relief.

This is the discipline that turns AI review from a confidence machine into a useful one. The model is allowed to say things you'd never have noticed. It's not allowed to be the last layer of trust on anything that matters.

Designing A Review Workflow That Actually Works

Once you accept that AI review is great at one half of the job and structurally incapable of the other half, the workflow design becomes obvious. You arrange the two halves in series, not in parallel, and you give each one the job it can actually do.

Here's the shape of a workflow that gets the upside.

Step 1 - AI reviewer runs first, as a lint pass. The moment a pull request opens, the AI reviewer sweeps it. It flags style issues, naming inconsistencies, obvious antipatterns, classic security patterns, missing error handling, contradictory documentation. It posts these as inline comments on the diff, the same way a linter would.

The author addresses these before any human reviewer is paged. This is the whole productivity win - the senior engineer never has to write "please use the existing util for this" again. The boring layer is done. The author shows up to human review with a diff that's already past the obvious objections.

Step 2 - Author writes the description that the AI didn't. This is the step that gets cut and shouldn't. The AI is reviewing the code. The human reviewer needs to review the change. A change is the code plus the intent plus the constraints plus the trade-offs the author made.

The author writes - in plain English - what this change does, why now, what alternatives they considered, what the rollout plan is, what they're explicitly not trying to solve. If you've never read Tigran Sloyan's or Will Larson's writing on this, the short version is: a PR description is for the next person who reads the commit in three years, not for you today.

A good description here is also the thing that catches the author's own hallucinations. If you can't explain in two paragraphs why this change is correct, you've found the actual problem.

Step 3 - Human reviewer focuses on what the AI can't see. With the lint layer handled and the description written, the human reviewer's attention can land on the real questions:

Does this change match what we agreed to build?
Does it interact correctly with the rest of the system?
Are the security invariants of this product preserved, given who the users are and what data they hold?
Is this the right unit of change to ship, or should it be split, sequenced, or guarded behind a feature flag?
What are we not going to be able to undo after this merges?

These are the questions that need a person who knows the codebase and the product. They're the questions you wanted the AI to handle and it can't.

Step 4 - Critical paths get an explicit "AI didn't review this" gate. Any change to authentication, authorization, payments, data deletion, migrations, or anything else where the worst-case outcome is "the company gets fined or sued" carries a flag in your review template: "This change touches a critical path. AI review is not sufficient. Require N human approvals with one being from ."

This isn't because the AI's review is wrong on critical paths. It might be perfectly fine. It's because the cost of being wrong is asymmetric, and you don't let asymmetric risks ride on confidence-shaped reassurance.

Step 5 - Treat AI review output as evidence, not verdict. When the AI says "no issues found", that is not a green light. It's an absence of red flags from a system that can only see a fraction of them. Your team's mental model of an AI review needs to be: "the obvious problems are probably caught; the subtle ones are not addressed either way". The decision to merge still belongs to a human and a passing test suite.

This shape works. It doesn't deliver the "agents review your PRs and you ship faster forever" narrative - but it delivers something better, which is fewer comments your senior engineers hate writing and more attention on the parts of review that actually prevent incidents.

What This Means For The Senior Engineer

If you're a senior engineer reading this, the practical shift is small and worth doing deliberately.

You're not going to stop using AI reviewers - they're too useful at the lint layer to give up. But you're going to stop deferring to them on anything that matters. When the AI says "looks good", you're going to read it as "the cheap problems aren't here, what about the expensive ones". When the AI references something in the codebase, you're going to confirm it. When the AI gives you a confident-sounding explanation, you're going to ask the question "how would I know this was wrong?" before you accept it.

You're also going to write better PR descriptions, because once the AI handles the lint layer, the bottleneck in review moves to the human reviewer's ability to understand the change. That's a writing problem, not a coding problem, and it's the one part of code review that has gotten more important since AI showed up, not less.

And you're going to push back when leadership says "we have AI review now, we can have fewer humans on it". The answer is no, and the reason isn't sentiment - it's that the half of the job AI doesn't do is the half where the bugs that cost real money live. You'd rather have one AI reviewer plus a careful senior than two AI reviewers and a fast merge button.

The question in the title of this piece - assistant or false confidence machine - isn't really about the tool. It's about the team. The same model, in the same codebase, on the same diff, is one or the other depending on whether a human is treating its output as evidence or as a verdict.

If you remember anything from this, remember that distinction. AI review is evidence. Humans make the verdict. The day you flip those two roles is the day your incident channel gets a lot louder.

Originally published at nazarboyko.com.

Observability For AI Features In Production

Nazar Boyko — Tue, 02 Jun 2026 04:48:40 +0000

AI features are different from normal application features. A normal API endpoint usually has clear behavior: you send input, you get output, and you can test exact values. You can monitor latency, errors, CPU, memory, database queries, and queue failures. The contract is tight, and when it breaks, the failure has a shape you can recognize.

AI features are messier. The same user question can produce slightly different answers. The model may call a tool, retrieval may return weak documents, the prompt may grow too large, and token cost can spike without warning. A small prompt change may quietly reduce quality, and a model upgrade may improve one workflow while breaking another.

So observability for AI is not only "did the request fail?" You also need to ask: did the model receive the right context, did retrieval find the right documents, did the model call the right tools? Was the answer useful, was it safe, was it too slow, was it too expensive? Could we debug this later? If you cannot answer these questions, your AI feature is not production-ready yet.

Start with the AI request lifecycle

Before adding dashboards, define the lifecycle of one AI request. A common AI feature looks like this:

User input
  -> validation
  -> prompt construction
  -> optional retrieval
  -> model call
  -> optional tool call
  -> response validation
  -> response shown to user
  -> user feedback

Each step can fail in a different way. Input validation can fail because the user asks for unsupported behavior. Prompt construction can fail because the template is missing variables. Retrieval can fail because documents are missing, stale, or irrelevant. The model call can fail because of latency, provider errors, rate limits, or poor output. Tool calls can fail because external APIs fail. Response validation can fail because the answer does not match the required schema. And user feedback can reveal that the answer was technically valid but not useful.

That means your logs should not only say:

{
  "status": "success"
}

They should tell the story of the request.

What to log for prompts

Prompt logs are useful, but they must be handled carefully. Prompts can contain personal data, customer data, secrets, internal documents, or sensitive business information. Do not blindly log everything forever. A safe approach is to log structured metadata by default and store full prompts only in controlled environments or with redaction.

Example prompt log:

{
  "request_id": "ai_req_01HX9",
  "feature": "support_reply_assistant",
  "prompt_template": "support_reply_v4",
  "template_version": 4,
  "system_prompt_hash": "sha256:8f91...",
  "user_prompt_length": 1840,
  "final_prompt_tokens": 3120,
  "redaction_applied": true,
  "created_at": "2026-05-03T18:40:12Z"
}

Notice the hash. You do not always need to store the entire system prompt in every log; a hash plus a version is often enough to connect a production request to the exact prompt that produced it.

For debugging, you may also store redacted prompt snapshots:

function redactPrompt(prompt: string): string {
  return prompt
    .replace(/[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi, '[EMAIL]')
    .replace(/\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g, '[CARD_NUMBER]')
    .replace(/sk-[A-Za-z0-9_-]+/g, '[API_KEY]');
}

This is simple, not perfect. In real systems, redaction should be layered and tested. But the principle is important: observability should not become a data leak.

What to log for retrieval

RAG features need retrieval observability. If an AI answer is bad, the model may not be the problem. The retrieved context may be weak, the index may be stale, or the user's question may sit outside the knowledge base entirely. Log what retrieval returned:

{
  "request_id": "ai_req_01HX9",
  "retrieval": {
    "index": "support_docs_prod",
    "query": "refund after annual renewal",
    "top_k": 5,
    "documents": [
      {
        "doc_id": "refund-policy-2026",
        "title": "Refund Policy",
        "score": 0.82,
        "version": "2026-04-12"
      },
      {
        "doc_id": "billing-faq",
        "title": "Billing FAQ",
        "score": 0.64,
        "version": "2026-02-01"
      }
    ]
  }
}

This helps you answer practical questions. Did we retrieve the correct policy? Was the document stale? Did the top result have a low score? Did the user ask a question outside the knowledge base? You can also track retrieval quality over time:

type RetrievalMetric = {
  query: string;
  topScore: number;
  resultCount: number;
  clickedDocumentId?: string;
  userFeedback?: 'helpful' | 'not_helpful';
};

function detectWeakRetrieval(metric: RetrievalMetric): boolean {
  return metric.resultCount === 0 || metric.topScore < 0.65;
}

Weak retrieval should be visible. Otherwise, teams blame the model when the actual issue is missing content.

What to log for tool calls

AI agents often call tools: database lookups, internal APIs, search services, code runners, ticket systems, or deployment systems. Tool calls need serious observability because they can change real systems.

Log the tool name, the input schema version, the sanitized arguments, the result status, the duration, the retry count, the authorization context, and whether the tool was read-only or write-enabled. Example:

{
  "request_id": "ai_req_01HX9",
  "tool_call": {
    "tool": "get_customer_subscription",
    "tool_version": "v2",
    "mode": "read_only",
    "duration_ms": 182,
    "status": "success",
    "arguments_redacted": {
      "customer_id": "cus_[REDACTED]"
    }
  }
}

For write tools, add extra guardrails:

{
  "tool": "cancel_subscription",
  "mode": "write",
  "requires_human_confirmation": true,
  "confirmation_id": "confirm_7831",
  "executed": false
}

A production AI assistant should not casually execute dangerous actions because the model "thought it was right." Read-only tools are safer. Write tools need approvals, audit logs, permissions, and rollback plans.

Latency: measure the full path, not only the model

AI latency is often multi-part. A slow response may include prompt building, retrieval, model generation, tool calls, response validation, streaming delay, and frontend rendering. Track each part separately:

type AiTiming = {
  requestId: string;
  promptBuildMs: number;
  retrievalMs: number;
  modelMs: number;
  toolMs: number;
  validationMs: number;
  totalMs: number;
};

function logTiming(timing: AiTiming): void {
  console.log(JSON.stringify({
    event: 'ai_timing',
    ...timing,
  }));
}

A dashboard should show p50, p95, and p99 latency. Average latency hides pain; if most requests finish in two seconds but 5% take thirty, users will notice the long tail before any chart does.

For streaming responses, also track time to first token:

{
  "request_id": "ai_req_01HX9",
  "time_to_first_token_ms": 740,
  "time_to_complete_ms": 8420
}

Time to first token matters because users feel the product is alive when streaming starts quickly.

Token usage and cost

Token usage is not just a billing detail. It is a product health signal. A feature can become expensive because prompts include too much irrelevant context, retrieval returns too many long chunks, conversation history is not summarized, the model is too powerful for a simple task, agents call each other repeatedly, or retries happen silently. Log token usage per feature:

{
  "feature": "pr_summary_assistant",
  "model": "example-large-model",
  "input_tokens": 18420,
  "output_tokens": 1620,
  "cached_input_tokens": 9000,
  "estimated_cost_usd": 0.084,
  "request_id": "ai_req_45KQ"
}

Then create cost dashboards by feature, customer, team, or workflow. A useful metric is cost per successful outcome, not only cost per request. For example:

support_reply_assistant
- 10,000 requests
- $420 model cost
- 6,800 helpful responses
- cost per helpful response = $0.061

That is much more useful than saying "we spent $420."

Failed generations and schema validation

AI output can fail even when the API request succeeds. Maybe the response is not valid JSON, maybe it misses required fields, maybe it includes text when your application expects structured data. Use schema validation:

import { z } from 'zod';

const PrSummarySchema = z.object({
  summary: z.string().min(20),
  changedBehavior: z.array(z.string()),
  riskyFiles: z.array(z.string()),
  testsRun: z.array(z.string()),
  missingTests: z.array(z.string()),
});

function parsePrSummary(raw: unknown) {
  const result = PrSummarySchema.safeParse(raw);

  if (!result.success) {
    throw new Error(`Invalid AI response schema: ${result.error.message}`);
  }

  return result.data;
}

Then log validation failures:

{
  "event": "ai_response_validation_failed",
  "feature": "pr_summary_assistant",
  "template_version": 3,
  "model": "example-large-model",
  "error": "missing required field: testsRun"
}

This lets you detect prompt regressions quickly.

User feedback is part of observability

AI quality is not only technical. Users can tell you when an answer was helpful, wrong, too long, unsafe, or irrelevant. Do not collect only thumbs up/down; add lightweight reason categories:

type AiFeedback = {
  requestId: string;
  rating: 'positive' | 'negative';
  reason?:
    | 'incorrect'
    | 'missing_context'
    | 'too_verbose'
    | 'unsafe'
    | 'not_actionable'
    | 'other';
  comment?: string;
};

This feedback can feed evaluation datasets. If users repeatedly mark answers as missing_context, the problem may be retrieval. If they mark answers as too_verbose, the prompt may need tighter formatting rules. If they mark answers as incorrect, you need deeper analysis: bad prompt, bad context, weak model, ambiguous user input, or missing business rule.

Evaluation and regression testing

Production observability tells you what happened. Evaluations help you prevent known failures from coming back. Create a small dataset of realistic cases:

[
  {
    "id": "refund_annual_plan_001",
    "input": "Can I get a refund if my annual plan renewed yesterday?",
    "expected_traits": [
      "mentions refund window",
      "does not promise refund automatically",
      "asks for account details if needed"
    ],
    "forbidden_traits": [
      "invented policy",
      "asks for full credit card number"
    ]
  }
]

You can run this dataset whenever you change the prompt template, the retrieval index, the model, the tool definitions, the system instructions, or the response schema. The goal is not perfect testing. The goal is catching obvious regressions before users do.

A simple AI observability schema

Here is a practical event model:

type AiEvent =
  | {
      type: 'ai.request.started';
      requestId: string;
      feature: string;
      userId?: string;
      promptTemplate: string;
    }
  | {
      type: 'ai.retrieval.completed';
      requestId: string;
      topScore: number;
      resultCount: number;
      documentIds: string[];
    }
  | {
      type: 'ai.model.completed';
      requestId: string;
      model: string;
      inputTokens: number;
      outputTokens: number;
      durationMs: number;
    }
  | {
      type: 'ai.tool.completed';
      requestId: string;
      toolName: string;
      status: 'success' | 'failure';
      durationMs: number;
    }
  | {
      type: 'ai.response.validation_failed';
      requestId: string;
      error: string;
    }
  | {
      type: 'ai.feedback.received';
      requestId: string;
      rating: 'positive' | 'negative';
      reason?: string;
    };

You can send these events to your normal observability stack. The exact vendor matters less than consistency.

Privacy and retention

AI observability can collect sensitive information if you are not careful. Set clear rules: redact secrets before logging, avoid storing raw prompts by default, set retention limits, separate debugging access from general analytics access, record prompt template versions, store document IDs instead of full documents when possible, and audit access to AI traces. This is especially important for internal assistants that can read customer support tickets, invoices, medical records, legal documents, or private engineering docs.

Final thoughts

AI features need observability because they are probabilistic, context-sensitive, and expensive. You need more than HTTP 200 and p95 latency. You need to see prompts, retrieval, tool calls, tokens, cost, validation failures, user feedback, and evaluation results.

The best AI observability systems do not only help you debug failures. They help you improve the product. They show where retrieval is weak, where prompts waste context, where model upgrades changed behavior, and where users do not trust the answer. That is the difference between an AI demo and an AI product. A demo only needs to work once. A product needs to keep working tomorrow.

AI for Documentation That Developers Actually Maintain

Nazar Boyko — Tue, 02 Jun 2026 04:41:03 +0000

Roughly 45% of the traffic to your developer documentation in 2026 isn't human.

That's not a metaphor. Mintlify, which hosts docs for a few thousand engineering teams, published a traffic study in March 2026 showing that AI coding agents (Claude Code, Cursor, OpenCode, ChatGPT, and friends) now account for 45.3% of all requests to their hosted docs sites. Human browsers are at 45.8%. Two of those agents, Claude Code and Cursor, make up 95.6% of all the bot traffic.

So when your docs are wrong, you're not just misleading new hires. You're feeding wrong answers into the tool the rest of your team uses to ship features. The asymmetry is brutal: a single stale POST /v1/orders page can leak into thousands of Cursor autocompletions before anyone notices.

Which makes the old documentation problem suddenly load-bearing.

And the old documentation problem isn't writing docs. AI has solved writing. You can dictate three bullet points to Claude and get a polished wiki entry in twelve seconds. The hard part, the part nobody has actually fixed yet, is keeping the docs true the day after they're written, the week after, the quarter after.

This piece is about what that takes. Specifically: how to use AI in a way that produces wikis, API references, and architecture decision records that survive contact with a moving codebase. The headline trick is that AI's most useful role here isn't generation. It's verification. And once you accept that, three workflows fall out naturally.

The Maintenance Problem That Generation Doesn't Fix

Let's name the bug before we fix it.

Documentation rots because nothing in the normal development loop forces it to stay aligned with the code. You merge a refactor. Tests pass. CI is green. The wiki page about that subsystem? Nothing pinged it. Nothing knows it exists. It is now subtly, expensively wrong, and it will stay wrong until somebody hits a bug six weeks from now and goes looking.

Stripe's Developer Coefficient study found that developers spend an average of 17.3 hours per week fixing the past instead of building the future: debugging, refactoring, and servicing technical debt, out of a roughly 41-hour week. That's about 42% of the week spent going backwards. A meaningful chunk of it is "I trusted the doc, the doc was wrong, I rewrote the doc after burning two hours on a bad assumption." It's a tax everyone pays and nobody itemises.

The root cause is structural, not motivational. Developers don't avoid documentation because they're lazy. They avoid it because the system never makes them. Three failure modes show up over and over:

No ownership. Code has a CODEOWNERS file. Docs don't. When the auth team renames a flag, nobody pings the doc that mentions the old name, because no script knows the doc exists.
No verification. Tests fail when code is wrong. Nothing fails when docs are wrong. Drift is invisible until a human notices.
No update prompt. The PR template asks if you updated tests. It rarely asks if you updated the wiki. Even when it does, devs say "yes" reflexively and move on.

You can dump infinite AI-generated prose into Confluence and not fix a single one of these. The docs will still rot. They'll just rot faster because there's more surface area to rot.

This is why the well-meaning "let's have AI write a wiki entry for every PR" projects keep dying after three months. They produce content, not maintenance. And content that nobody promised to maintain is content that decays at exactly the same rate as content nobody wrote.

What AI Is Bad At (And What It's Actually Good At)

AI is bad at deciding what's true. That's not pessimism, that's mechanics. Frontier models in 2026 hallucinate at rates benchmarked between roughly 3% and 19%, depending on the model, the task, and the reasoning configuration. Three percent is the floor. Most production deployments aren't at the floor.

So if you ask Claude "what does the /v2/checkout endpoint return on partial inventory?" with no context, you will get an answer. It may even be plausible. It may also be entirely invented, indistinguishable from the real one, and confidently formatted as documentation. That's not a flaw of any single model. It's what a language model does when you don't give it ground truth to pull from.

The good news, also from the 2026 benchmark research, is that retrieval grounding cuts hallucinations by up to ~70% (some reports put it at 70-80% on enterprise knowledge-base tasks). The pattern is consistent: when you anchor the model to a verified source (the actual route handler, the actual schema, the actual diff), quality jumps. When you don't, you get plausible fiction.

This is the lever. The job of AI in your docs pipeline isn't "write the doc." It's:

Summarise, given a primary source.
Compare, given a doc and the code it claims to describe.
Flag, when the two have drifted.
Draft, in a structured shape, when you've decided what to say.

In other words: AI is a junior tech writer with a photographic memory and zero judgement. Pair it with a senior reviewer and a verification harness, and it's transformative. Cut it loose with a Confluence write key and a friendly prompt, and you've automated the production of wrongness.

The three workflows below are all variations on this idea.

Wiki Updates: Bind Them to the PR

The wiki failure is the most common one and the easiest to fix.

The pattern that works: every PR is required to produce a wiki delta, and AI writes the first draft of that delta from the diff. Not the whole wiki, just what changed. The human reviewer accepts, edits, or rejects. The reviewer is the same person who's already reviewing the code. The wiki update lives in the same PR, gated by the same merge.

This solves all three failure modes from earlier:

Ownership: the PR author owns the wiki delta the same way they own the code change.
Verification: the reviewer is looking at the code and the doc side by side. Drift is impossible to ship without someone noticing.
Update prompt: it's not a polite request in the PR template, it's a required field. Empty box, no merge.

In practice this looks like a CI step that runs against every PR:

.github/workflows/wiki-delta.yml

name: Wiki Delta
on: [pull_request]
jobs:
  draft-wiki-update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Generate wiki delta draft
        run: |
          npx wiki-delta-bot \
            --diff "${{ github.event.pull_request.base.sha }}..HEAD" \
            --wiki ./docs/wiki \
            --out wiki-delta.md
      - name: Comment delta on PR
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: wiki-delta.md

(The wiki-delta-bot here is a stand-in: substitute whatever internal tool you use, or Mintlify's PR preview, or a custom Claude call with the diff and the relevant wiki page as context.)

The crucial detail isn't the tool. It's that the AI gets both inputs: the diff and the existing wiki page. It writes the smallest change that reconciles the two. Not a full rewrite. Not a fresh page. A delta. The reviewer is reading thirty changed lines, not three hundred regenerated ones, and that's the difference between a process that survives and one that gets disabled after a sprint.

Tip
Start with the wiki pages that get the most agent traffic. If Mintlify-style analytics are available, sort by AI-agent reads, descending. Those are the pages that, if stale, will mislead the most downstream tools.

The "docs updated?" gate as a PR requirement has been a docs-as-code talking point for years, well before AI. The reason it never quite caught on was friction: writing the doc update by hand in every PR is enough of a tax that teams quietly stop enforcing the gate after a few sprints. AI changes the friction math. When the bot has already written 80% of the delta, even the laziest reviewer will tidy it instead of deleting it.

API Docs: Anchor First, Prose Second

API documentation is the place where AI generation has the worst track record. Ask a model to "write API docs for this service" and you'll get a beautiful page where POST /orders accepts a quantity: int field. That field does not exist in your code. Nobody can find where it came from. It's been there for two months. Three internal teams are now writing client code that sends it.

The fix is to invert the writing order: start from the code, end with the prose.

OpenAPI is the lever. Modern frameworks (FastAPI, Spring Boot, NestJS, Hono, Go's huma, Laravel's Scribe) can emit an OpenAPI specification straight from the actual route handlers, decorators, and types. The spec is mechanically true by construction. Routes that don't exist can't appear. Parameters that aren't in the handler signature can't appear. The shape is grounded.

Then you let AI write prose around that ground truth. Endpoint descriptions, examples, error explanations, "when to use this vs that", those are all genuine writing tasks where a model has nothing to fabricate, because the schema is already pinned down. Mintlify's own analysis of why hallucinations happen lands on the same point: structured input is the prevention.

api/routes/orders.ts

// FastAPI-style example, but the pattern is framework-agnostic
app.post(
  '/v2/orders',
  {
    schema: {
      body: OrderCreateSchema,        // ← single source of truth
      response: { 201: OrderSchema }, // ← AI cannot invent a "quantity" field
    },
    summary: 'Create a new order',
    description: '<<<AI_FILLS_THIS>>>', // ← prose, generated against the real schema
  },
  createOrderHandler,
)

The schema is the anchor. Everything AI writes downstream is bounded by it. If a developer adds a new field, the spec changes, the AI's next pass at the description sees the new field, and the docs update with it. If a developer renames a field, the old name vanishes from the spec, and the AI's prose stops referring to it, because it never had license to.

This is also why "generate docs from code" tools used to feel weak and now feel powerful. The tools didn't change much. The grounding layer did. A 2025 spec generator like huma plus a 2026 model reading that spec produces output that an earlier model would have hallucinated past.

The footgun in this section: don't let AI invent example values. A model asked for "an example order body" will happily make up a credit card number, a UUID, and a customer email, none of which correspond to anything in your system. If you can't seed examples from real fixture data or factory output, mark them clearly as illustrative ("customer_id": "<example-uuid>") so the human reader, and the next AI consuming this page, can tell.

ADRs: AI as the Scribe, You as the Decider

Architecture Decision Records are the documentation form that has the worst capture rate.

Not because they're hard to write. They're trivial. The standard ADR format is a handful of short sections: context, decision, consequences, alternatives considered. You can fill it out in fifteen minutes.

The reason ADRs go uncaptured is that the moment you should write one, the moment you make the decision, you're in flow, you're shipping, you're on a Slack call, and an internal voice says "I'll write it down later." Later never comes. Six months later the decision is still in effect, but nobody on the team remembers why, and a new engineer is about to undo it because the original constraint isn't in any document.

AI is shockingly good at fixing exactly this gap. Not by deciding architecture, it shouldn't, and it can't, because the tradeoffs are tied to your business and your team. But by being the scribe at the moment of decision.

The pattern that works: when you make a non-trivial architectural call, you say it out loud (in chat, in a PR, in a meeting transcript). An AI agent (Claude Code skill, Cursor agent, whatever your team standardises on) picks that up, interrogates you for two minutes about context and alternatives, and emits a draft ADR. You read it for thirty seconds, fix the parts it got wrong, commit it under docs/adrs/NNN-thing.md. Done.

This works because the format is structured enough that the AI knows what's missing. If you say "we're switching from polling to webhooks," the ADR scribe knows it needs to ask: why now, what was tried, what's the rollback, what does this lock you out of? The interrogation is the value. You'd have skipped those questions on your own.

Here's roughly what one looks like, drafted from a five-minute conversation:

docs/adrs/0042-switch-fulfilment-queue-to-redis-streams.md

# ADR 0042: Switch Fulfilment Queue from SQS to Redis Streams

**Status:** Accepted
**Date:** 2026-05-12
**Deciders:** @nazar, @ana, @jp

## Context

Fulfilment workers process ~12k orders/day with bursts up to 800/min during
flash sales. We're hitting SQS message size limits (256KB) for ~3% of payloads
after we added line-item metadata in Q1, forcing an S3-pointer workaround that
adds two round trips per message.

## Decision

Replace SQS with Redis Streams (already deployed for session state) as the
primary fulfilment queue. Keep SQS as a dead-letter sink for messages that
fail Redis-side validation.

## Consequences

- Removes the S3-pointer hop; expected p99 latency drop from ~340ms to ~90ms.
- Adds operational responsibility: Redis Streams retention and consumer-group
  lag now matter to ops on-call.
- Loses SQS's native at-least-once + visibility-timeout semantics; we
  reimplement those via Stream ack + idempotency keys at the worker.

## Alternatives Considered

- **Stay on SQS, increase message size via S3 pointers.** Already in place;
  the two-round-trip cost is what's prompting this change.
- **Kafka.** Closer match semantically, but we have no Kafka operational
  experience and Redis is already in the stack.
- **NATS JetStream.** Strong candidate; deferred, not enough team familiarity
  and we don't want two new pieces of infrastructure.

Note what the AI did not do here: it didn't decide. It collected what was already decided and put it in a shape that future-you can scan. The decision happened in the heads of three people on a call. The ADR captured it before it evaporated.

This is, in my view, the highest-leverage use of AI in documentation right now. ADRs are uniquely valuable (they explain why, which code itself can't), uniquely under-captured (because of the timing problem), and uniquely well-suited to AI scribing (because the format is rigid and the input is conversational).

If you only do one thing from this article, set up an ADR-scribe workflow.

The Verification Layer Behind All of This

The three workflows above share a hidden ingredient: they all depend on something checking whether the doc still matches the code. Without that check, you're just generating prose on a treadmill.

The verification layer has three components, and they're not interchangeable.

1. Anchors that bind doc to code. Swimm's whole product is built on this idea: the documentation includes "smart tokens" and "smart paths" that reference specific functions, types, or line ranges in the codebase. When the code changes, Swimm flags the affected doc with a "Review required" status. The mechanism is straightforward: it's structured references plus a CI job that compares the current code against what the doc expected. The same idea works without Swimm. You can roll your own by linking docs to git refs and running a diff check in CI.

2. Prose linting and link checking. Vale for style (tone, terminology, forbidden phrasing) and any link checker (Lychee, markdown-link-check, Fern's built-in) for broken references. These are 2010s tech, but they're still the cheapest way to catch a category of rot (typos, dead URLs, renamed pages) that AI-generated content is especially prone to.

3. AI-driven drift checks. This is the new piece. On a schedule (nightly, weekly), an agent walks the docs tree, for each page pulls the code it references, and asks the model: "does this page accurately describe this code?" Output is a drift report. Humans triage. This is the verification arm of the same workflow you use to generate wiki deltas: same anchor, same comparison, just run on a cadence instead of on every PR.

None of these three works alone. Anchors without lint checks miss broken external links. Lint checks without anchors don't catch the case where the doc still parses cleanly but describes a function that's been renamed. AI drift checks without anchors get expensive fast: you're rescanning the whole repo every night instead of the bits the doc actually references.

Stack all three and you have docs that fail loudly, like tests. Which is the only kind that gets maintained.

The llms.txt Detour Nobody Saw Coming

A small piece of context that's reshaping all of the above: llms.txt.

In September 2024, Jeremy Howard of Answer.AI proposed a standard for a /llms.txt file at the root of any docs site, structured to be cheap for a language model to ingest. Within months, Mintlify rolled out automatic support across every site they host. Mintlify built the llms-full.txt variant with Anthropic (a single concatenated dump of all docs), because parsing fragmented HTML was killing Anthropic's indexing pipelines. Cloudflare, Vercel, and a thousand others followed in months.

It's a small file. It just lists your docs and gives a short description of each. But it changes what "documentation" means: it's the AI-readable interface to your knowledge, separate from the human-readable one. And the same maintenance problem applies, only worse, because nobody is looking at llms.txt with human eyes. A page that drops out of the file is silently invisible to every Claude Code session for the next quarter.

If you're already running the three-layer verification stack above, llms.txt falls out for free; it's just a generated index over docs you're already keeping true. If you're not, llms.txt accelerates the problem. AI agents read what's in the index, generate code based on it, and your stale docs get amplified into stale autocompletions.

The Failure Mode That Bites Senior Teams

Worth naming directly, because it's the one I see senior teams stumble on, not junior ones.

The temptation: "AI is good enough now. Let's have it generate the docs in bulk from the codebase, ship the result, and call the project done." A weekend hackathon, three thousand new wiki pages, victory.

This is how you build a much larger pile of plausibly-wrong content than you had before.

Three things go wrong. First, the bulk-generated pass has a real hallucination rate. At 5-10% per page (a reasonable midpoint of the 2026 benchmarks), three thousand pages contain hundreds of wrong claims. Nobody knows which ones. Second, none of those pages have a maintainer or a verification hook, so they start rotting on day one. Third, and this is the new failure mode, those pages are now read by your team's AI tools. Cursor pulls from your wiki. Claude Code pulls from your wiki. The wrong content compounds into wrong code suggestions, and the loop closes.

Generation without maintenance was a productivity drain in 2022. In 2026, with AI agents accounting for nearly half your doc traffic, it's a self-reinforcing source of bugs.

The fix isn't subtle: never generate a doc page that nobody has agreed to maintain. If no human's name is on it and no anchor binds it to the code, don't write it. AI can produce infinite content; the bottleneck is and always will be the verification layer attached to it.

What Actually Changes in Your Workflow

If you've read this far, the practical shape is probably clear, but here it is compactly.

Your wiki gets a CI job that drafts a delta on every PR, anchored to the diff and the existing page. Reviewer approves or edits, in the same PR that ships the code. No delta, no merge.

Your API docs get generated from a typed schema (OpenAPI, GraphQL SDL, gRPC proto, whatever), and AI fills in prose against that schema, never against thin air. Example values come from real fixtures, not from the model's imagination.

Your ADRs get a scribe agent. The moment you make a non-trivial decision in chat or in a PR comment, the agent prompts you for two minutes and produces a draft. You commit it before the day ends.

A verification stack runs in the background: anchors for code-to-doc binding, prose linting and link checking for hygiene, and a scheduled AI drift check for the cases the first two miss.

And llms.txt (or whatever standard your team settles on) gets generated from the same verified source, so the half of your doc traffic that's AI agents stays as accurate as the half that's humans.

None of this is novel in isolation. Anchors existed before AI. Linting existed before AI. ADRs predate Claude by a decade. What's new is that AI, cheap enough to run on every PR against the actual diff in seconds, finally makes the verification side affordable. The 2010s answer to documentation rot was "be more disciplined." The 2026 answer is "make the discipline tractable." That's the actual win.

Docs that developers maintain aren't a writing problem. They're an integration problem. AI is most valuable not as the writer but as the integration glue: the thing that reads the diff, compares it to the doc, drafts the delta, runs the check, and surfaces the drift before it ships.

Build that, and your wiki stops lying. Build the alternative, and the lies just get faster.